av9ash/GitBugs huggingface dataset:
Bug: Heavy external accumulators may lead driver full GC problem
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Sep/20 03:45
Affects Version/s: 2.4.7

We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (very heavy) when users submit a MERGE INTO query. The driver held over 100GB memory (depends on how much the max heap size set) and can not be GC forever. By making a heap dump we found the root cause.
 !Screen Shot 2020-09-25 at 11.32.51 AM.png|width=70%! 
 !Screen Shot 2020-09-25 at 11.35.01 AM.png|width=100%! 
 !Screen Shot 2020-09-25 at 11.36.48 AM.png|width=100%! 

From above heap dump, Delta uses a SetAccumulator to records touched files names
{code}
    // Accumulator to collect all the distinct touched files
    val touchedFilesAccum = new SetAccumulator[String]()
    spark.sparkContext.register(touchedFilesAccum, TOUCHED_FILES_ACCUM_NAME)

    // UDFs to records touched files names and add them to the accumulator
    val recordTouchedFileName = udf { (fileName: String) => {
      touchedFilesAccum.add(fileName)
      1
    }}.asNondeterministic()
{code}

In a big query, each task may hold thousands of file names, and if a stage contains dozens of thousands of tasks, DAGscheduler may hold millions of `CompletionEvent`. And each `CompletionEvent` holds the thousands of file names in its `accumUpdates`. All accumulator objects will use Spark listener event to deliver to the event loop and even a full GC can not release memory.

A PR will be submitted. With the patch, the memory problem was gone.
Before the patch: A full GC doesn't help.
 !Screen Shot 2020-09-24 at 5.19.58 PM.png|width=70%! 
After the patch: No full GC and memory is not ramp up.
 !Screen Shot 2020-09-24 at 5.19.26 PM.png|width=70%!
Bug: orc table column name doesn't support special characters.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/20 12:31
Affects Version/s: 3.1.0

when execute 

"CREATE TABLE tbl(`$` INT, b INT) using orc";

error occurs like below. but in it's ok in hive.

Column name "$" contains invalid character(s). Please use alias to rename it.;Column name "$" contains invalid character(s). Please use alias to rename it.;org.apache.spark.sql.AnalysisException: Column name "$" contains invalid character(s). Please use alias to rename it.; at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$.checkFieldName(OrcFileFormat.scala:51) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$.$anonfun$checkFieldNames$1(OrcFileFormat.scala:59) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$.$anonfun$checkFieldNames$1$adapted(OrcFileFormat.scala:59) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$.checkFieldNames(OrcFileFormat.scala:59) at org.apache.spark.sql.execution.command.DDLUtils$.$anonfun$checkDataColNames$1(ddl.scala:924) at org.apache.spark.sql.execution.command.DDLUtils$.$anonfun$checkDataColNames$1$adapted(ddl.scala:913) at scala.Option.foreach(Option.scala:407) at org.apache.spark.sql.execution.command.DDLUtils$.checkDataColNames(ddl.scala:913) at org.apache.spark.sql.execution.command.DDLUtils$.checkDataColNames(ddl.scala:908) at org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:231) at org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:80) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$2(AnalysisHelper.scala:108) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:108) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:207) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:73) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:72) at
Bug: Spark SQL INTERSECT ALL does not keep all rows.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 24/Sep/20 03:04
Affects Version/s: 2.4.6

Spark SQL INTERSECT ALL should keep all rows. But, it actually remove replicated like Spark SQL INTERSECT.

with base as (

select 1 as id union all select 2 as id

), a as (

select 1 as id union all select 3 as id)

select * from a INTERSECT ALL select * from base;

 

with base as (

select 1 as id union all select 2 as id

), a as (

select 1 as id union all select 3 as id)

select * from a INTERSECT select * from base;

 

Both the above queries return one record that is 1.

I think the 1st query should return

1

1
Bug: [SQL] JavaDoc on Default Save mode Incorrect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 15:47
Affects Version/s: 3.0.1

The JavaDoc says that the default save mode is dependent on DataSource version which is incorrect. It is always ErrorOnExists.

http://apache-spark-developers-list.1001551.n3.nabble.com/DatasourceV2-Default-Mode-for-DataFrameWriter-not-Dependent-on-DataSource-Version-td29434.html
Bug: Spark| PartitionBy is taking long time to process
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Sep/20 16:49
Affects Version/s: 2.4.5

# When I do a write without any partition it takes 8 min

df2_merge.write.mode('overwrite').parquet(dest_path)

 

       2. I have added conf - spark.sql.sources.partitionOverwriteMode=dynamic ; it took a longer time (more than 50 min before I force terminated the EMR cluster). But I have observed the partitions have been created and data files are present. But in EMR cluster the process is still showing as running, where as in spark history server it is showing no running or pending process.

df2_merge.write.mode('overwrite').partitionBy("posted_on").parquet(dest_path_latest)

 

      3. I have modified with new conf - spark.sql.shuffle.partitions=3; it took 24 min

df2_merge.coalesce(3).write.mode('overwrite').partitionBy("posted_on").parquet(dest_path_latest)

 

     4. Again I disabled the conf and run plain write with partition. It took 30 min.

df2_merge.coalesce(3).write.mode('overwrite').partitionBy("posted_on").parquet(dest_path_latest)

 

Only one conf is common in the above scenarios is spark.sql.adaptive.coalescePartitions.initialPartitionNum=100

My point is to reduce the time of writing with partitionBy. Is there anything I am missing
Bug: totalExecutorRunTimeMs is too big
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 05:18
Affects Version/s: 2.4.7

This might be because of incorrectly calculating executorRunTimeMs in Executor.scala
 The function collectAccumulatorsAndResetStatusOnFailure(taskStartTimeNs) can be called when taskStartTimeNs is not set yet (it is 0).

As of now in master branch, here is the problematic code: 

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L470]

 

There is a throw exception before this line. The catch branch still updates the metric.
 However the query shows as SUCCESSful. Maybe this task is speculative. Not sure.

 

submissionTime in LiveExecutionData may also have similar problem.

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala#L449]
Bug: CostBasedJoinReorder should produce same result in Scala 2.12 and 2.13 with same input
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Sep/20 16:08
Affects Version/s: 3.0.0

The optimization result of {{CostBasedJoinReorder}} maybe different with same input in Scala 2.12 and Scala 2.13 if there are more than one same cost candidate plans.

The test case named "Test 4: Star with several branches" in StarJoinCostBasedReorderSuite is a typical case.

 

If the input is 
{code:java}
d1.join(t3).join(t4).join(f1).join(d2).join(t5).join(t6).join(d3).join(t1).join(t2)
  .where((nameToAttr("d1_c2") === nameToAttr("t3_c1")) &&
    (nameToAttr("t3_c2") === nameToAttr("t4_c2")) &&
    (nameToAttr("d1_pk") === nameToAttr("f1_fk1")) &&
    (nameToAttr("f1_fk2") === nameToAttr("d2_pk")) &&
    (nameToAttr("d2_c2") === nameToAttr("t5_c1")) &&
    (nameToAttr("t5_c2") === nameToAttr("t6_c2")) &&
    (nameToAttr("f1_fk3") === nameToAttr("d3_pk")) &&
    (nameToAttr("d3_c2") === nameToAttr("t1_c1")) &&
    (nameToAttr("t1_c2") === nameToAttr("t2_c2")))

{code}
the optimization result  in Scala 2.12 is 
{code:java}
  f1.join(d3, Inner, Some(nameToAttr("f1_fk3") === nameToAttr("d3_pk")))
    .join(d1, Inner, Some(nameToAttr("f1_fk1") === nameToAttr("d1_pk")))
    .join(d2, Inner, Some(nameToAttr("f1_fk2") === nameToAttr("d2_pk")))
    .
    .
    .{code}
and the optimization result  in Scala 2.13 is 
{code:java}
f1.join(d3, Inner, Some(nameToAttr("f1_fk3") === nameToAttr("d3_pk")))
    .join(d2, Inner, Some(nameToAttr("f1_fk2") === nameToAttr("d2_pk")))
    .join(d1, Inner, Some(nameToAttr("f1_fk1") === nameToAttr("d1_pk")))
    .
    .
    .
{code}
Bug: File source V2: support bucketing
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/20 08:07
Affects Version/s: 3.1.0

Datasource V2 does not currently support bucketed reads or writes similar to Datasource V1 does.  See DatasourceScanExec and config 

spark.sql.sources.bucketing.enabled.  We need to add support to V2 as well.

 

Support writing file data source with bucketing looks like:
{code:java}
 
fileDf.write.bucketBy(...).sortBy(..)... 
{code}
Bug: ApplicationMaster fails to receive UpdateDelegationTokens message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 02:22
Affects Version/s: 3.0.1


{code:java}
20-09-15 18:53:01 INFO yarn.YarnAllocator: Received 22 containers from YARN, launching executors on 0 of them.
20-09-16 12:52:28 ERROR netty.Inbox: Ignoring error
org.apache.spark.SparkException: NettyRpcEndpointRef(spark-client://YarnAM) does not implement 'receive'
	at org.apache.spark.rpc.RpcEndpoint$$anonfun$receive$1.applyOrElse(RpcEndpoint.scala:70)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:203)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20-09-17 06:52:28 ERROR netty.Inbox: Ignoring error
org.apache.spark.SparkException: NettyRpcEndpointRef(spark-client://YarnAM) does not implement 'receive'
	at org.apache.spark.rpc.RpcEndpoint$$anonfun$receive$1.applyOrElse(RpcEndpoint.scala:70)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:203)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}


With a long-running application in kerberized mode, the AMEndpiont handles the token updating message wrong.
Bug: Pass all `sql/hive-thriftserver` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 08:34
Affects Version/s: 3.1.0

There are 1 failed and 1 aborted test case of  sql hive-thriftserver module in Scala 2.13
 * HiveThriftServer2Suites(1 FAILED)
 * HiveThriftServer2ListenerSuite(ABORTED)
Bug: Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Sep/20 14:22
Affects Version/s: 3.1.0

Basically, there are 3 decommission use cases: k8s, standalone, dynamic allocation. And they are all using the DecommissionInfo to represent their own case. But, DecommissionInfo now is not enough to tell whether the decommission is triggered at executor or not after SPARK-32850. So, it's the time to improve both DecommissionInfo and DecommissionState.
Bug: GeneratePredicate should be able to eliminate common sub-expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 20:29
Affects Version/s: 3.1.0

Both {{GenerateMutableProjection}} and {{GenerateUnsafeProjection}}, such codegen objects can eliminate common sub-expressions. But {{GeneratePredicate}} currently doesn't do it.

We encounter a customer issue that a Filter pushed down through a Project causes performance issue, compared with not pushed down case. The issue is one expression used in Filter predicates are run many times. Due to the complex schema, the query nodes are not wholestage codegen, so it runs {{Filter.doExecute}} and then call {{GeneratePredicate}}. The common expression was run many time and became performance bottleneck. {{GeneratePredicate}} should be able to eliminate common sub-expressions for such case.
Bug: Pass all `sql/hive` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/20 13:00
Affects Version/s: 3.1.0

There are only 4 test cases failed in sql hive module with cmd 

 
{code:java}
mvn clean install -DskipTests -pl sql/hive -am -Pscala-2.13 -Phive
mvn clean test -pl sql/hive -Pscala-2.13 -Phive{code}
 

The failed cases as follow:
 * HiveSchemaInferenceSuite (1 FAILED)
 * HiveSparkSubmitSuite (1 FAILED)
 * StatisticsSuite (1 FAILED)
 * HiveDDLSuite (1 FAILED)
Bug: DataSourceV2 allow ACCEPT_ANY_SCHEMA in write path
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/20 20:56
Affects Version/s: 3.0.0

During the development of a Spark-Collibra-Connector using the DataSourceV2 framework, I found some blocking limitation in the current version.

The connector should accept DataFrames of arbitrary schemas and send them to the Import API of Collibra. The problem is the method {{inferSchema}} of the {{TableProvider}}. Although, my {{Table}} implementation has the capability to {{ACCEPT_ANY_SCHEMA}}. I need to infer the schema without knowing the actual schema of the data frame. This is impossible. The behaviour is maybe intended, if you are writing to an existing table with fix schema, but not if you accept any schema. Such cases cannot be implemented, right now. I found in [{{DataFrameWriter.scala}}|https://github.com/apache/spark/blob/4fac6d501a5d97530edb712ff3450890ac10e413/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L333] that for data sources inherited from {{FileDataSourceV2}} there is an exception and {{inferSchema}} is not called on the write path and {{getTable}} is called with the schema of the actual data frame. This is the reason why it works for these data source derived from {{FileDataSourceV2}}. I would expect a similar behaviour for my data source which has the capability to accept any schema. The problem is that the capabilities are retrieved by the {{Table}} implementation, but to get a table via {{getTable}} you need a schema. I guess the interface should be designed differently:
* two different methods to infer the schema: 
** one for the read path like the current implementation
** one for the write path getting the actual schema of the data frame as parameter, this allows the implementation to decide:
*** Do I accept all schemas and just the return the schema of the data frame?
*** Do I know the schema of the target and ignore the schema of the actual data frame?
*** Can the schema of the target be evolved and I check the schema of the data frame to be a valid evolution of the target schema?

If you agree, I'm willing to make a PR.
Bug: BytesToBytesMap at MAX_CAPACITY exceeds growth threshold
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/20 20:51
Affects Version/s: 1.6.3

When BytesToBytesMap is at {{MAX_CAPACITY}} and reaches the growth threshold, {{numKeys >= growthThreshold}} is true but {{longArray.size() / 2 < MAX_CAPACITY}} is false. This correctly prevents the map from growing, but {{canGrowArray}} incorrectly remains true. Therefore the map keeps accepting new keys and exceeds its growth threshold. If we attempt to spill the map in this state, the UnsafeKVExternalSorter will not be able to reuse the long array for sorting, causing grouping aggregations to fail with the following error:

{{2020-09-13 18:33:48,765 ERROR Executor - Exception in task 0.0 in stage 7.0 (TID 69)
org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 12982025696 bytes of memory, got 0
	at org.apache.spark.memory.MemoryConsumer.throwOom(MemoryConsumer.java:160)
	at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:100)
	at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:118)
	at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:253)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithKeys_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithoutKey_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:733)
	at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:80)
	at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$1(Collector.scala:187)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:144)
	at org.apache.spark.scheduler.Task.run(Task.scala:117)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$9(Executor.scala:660)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1581)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:663)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)}}
Bug: from_avro is giving empty result
Status: Open | Priority: Major | Resolution: 
Created: 09/Sep/20 17:52
Affects Version/s: 3.0.0

I am trying to read a Kafka topic with Spark readStream but getting problem while applying avro schema

 

Code:

{code}
df = spark\
  .readStream\
  .format("kafka")\
  .option("kafka.bootstrap.servers", "host:6667")\
  .option("subscribe", "utopic1")\
  .option("failOnDataLoss", "false")\
  .option("startingOffsets", "earliest")\
  .option("checkpointLocation", "/home/abc/wspace/spark_test/data/")\
  .load()
 
outputDF = df\
        .select(from_avro("value", jsonFormatSchema, options={"mode":"FASTFAIL"}).alias("user"))

outputDF.printSchema()

query = outputDF.writeStream.format("console").start()
time.sleep(10)
{code}

Input:

avro schema file: [user.avsc|https://github.com/apache/spark/raw/4ad9bfd53b84a6d2497668c73af6899bae14c187/examples/src/main/resources/user.avsc]

Kafka topic: \{'favorite_color': 'Red', 'name': 'Alyssa'}

Expected Output:

It should print values. 

Actual Output:

{code}
+----+
|user|
+----+
|[,]|
+----+
{code}

Additional information:
 - Searched in the internet and found that other peson faced same issue. [https://stackoverflow.com/questions/59222774/spark-from-avro-function-returning-null-values]
 - I am able to print values to console if I cast to String using below line df.selectExpr("CAST(value AS STRING)")
Bug: Improve and refactor parallel listing in HadoopFSUtils
Status: Open | Priority: Major | Resolution: 
Created: 14/Sep/20 17:52
Affects Version/s: 3.1.0

We have a few follow-ups after refactoring the parallel listing feature into Spark Core:
 # Expose metrics as a callback
 # Enhance listing support for S3A/other cloud `FileSystem` impls
 # Other cleanups and refactorings on the code.
Bug: Mark more aggregate functions as order irrelevant
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/20 10:30
Affects Version/s: 3.1.0

In the EliminateSorts rule there is a list of aggregate functions, that are order irrelevant. The list is incomplete - for example HyperLogLogPlusPlus and BitAggregate are also order irrelevant.
Bug: EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions == false
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 01/Sep/20 11:55
Affects Version/s: 3.0.0

Currently, EliminateJoinToEmptyRelation Rule will convert Join into EmptyRelation in some cases with AQE on. But if either sub plan of Join contains a ShuffleQueryStage(canChangeNumPartitions == false), which means the Exchange produced by repartition Or singlePartition, in this case, if we were to convert it into an EmptyRelation, it will lost user specified number partition information for downstream operator, it's not right. 

So in the Patch, try not to do the conversion if either sub plan of Join contains ShuffleQueryStage(canChangeNumPartitions == false)
Bug: Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/20 22:21
Affects Version/s: 2.3.4

Structured Streaming micro-batch engine has the contract with V1 data sources that, after a restart, it will call `source.getBatch()` on the last batch attempted before the restart. However, a very rare combination of sequences violates this contract. It occurs only when 
- The streaming query has specific types of stateful operations with watermarks (e.g., aggregation in append, mapGroupsWithState with timeouts). 
    - These queries can execute a batch even without new data when the previous updates the watermark and the stateful ops are such that the new watermark can cause new output/cleanup. Such batches are called no-data-batches.
- The last batch before termination was an incomplete no-data-batch. Upon restart, the micro-batch engine fails to call `source.getBatch` when attempting to re-execute the incomplete no-data-batch.

This occurs because no-data-batches has the same and end offsets, and when a batch is executed, if the start and end offset is same then calling `source.getBatch` is skipped as it is assumed the generated plan will be empty. This only affects V1 data sources which rely on this invariant to initialize differently when the query is being started from scratch or restarted. How will a source misbehave is very source-specific.