av9ash/GitBugs huggingface dataset:
Bug: NoSuchElementException: None.get for needsUnsafeRowConversion
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 11/Aug/20 14:12
Affects Version/s: 3.0.0

I have run into an error somewhat non-deterministically where a query fails with 

{{NoSuchElementException: None.get}}

which happens at [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L182]

getActiveSession apparently is returning None. I only use Pyspark, and I think this is a threading issue, since the active session comes from an InheritableThreadLocal. I encounter this both when I manually use threading to run multiple jobs at the same time, as well as occasionally when I have multiple streams active at the same time. I tried using the PYSPARK_PIN_THREAD flag but it didn't seem to help. For the former case I hacked around it in my manual threading code by doing

{{spark._jvm.SparkSession.setActiveSession(spark._jvm.SparkSession.builder().getOrCreate())}}

at the start of each new thread, and this sometimes doesn't work reliably either.

I see this was mentioned in this [issue|https://issues.apache.org/jira/browse/SPARK-21418?focusedCommentId=16174642&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16174642]

I'm not sure if the problem/solution is something to do with Python threads, or adding a default value or some other way of updating this function. One other note is that I started encountering this when using Delta Lake OSS, which reads parquet files as part of the transaction log, which is when this error always happens. It doesn't seem like anything specific to that library though that would be doing something incorrectly that would cause this issue.
Bug: path option is treated differently for 'format("parquet").load(path)' vs. 'parquet(path)'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/20 20:59
Affects Version/s: 2.4.6

When data is read, "path" option is treated differently depending on how dataframe is created:
{code:java}
scala> Seq(1).toDF.write.mode("overwrite").parquet("/tmp/test")
                                                                                
scala> spark.read.option("path", "/tmp/test").format("parquet").load("/tmp/test").show
+-----+
|value|
+-----+
|    1|
+-----+


scala> spark.read.option("path", "/tmp/test").parquet("/tmp/test").show
+-----+
|value|
+-----+
|    1|
|    1|
+-----+
{code}
Bug: CostBasedJoinReorder should have deterministic Optimization result
Status: In Progress | Priority: Minor | Resolution: 
Created: 22/Aug/20 08:34
Affects Version/s: 3.0.0

In [SPARK-32526|https://github.com/apache/spark/pull/29434], we found the optimization result of CostBasedJoinReorder is non-deterministic now, it affected by the input order if there more than one same cost candidate plans.

The test case named "Test 4: Star with several branches" in StarJoinCostBasedReorderSuite is a typical case.

If we use {{permutations of 10 tables(}}{{d1, t3, t4, f1, d2, t5, t6, d3, t1, t2}}{{) as input set, there will be total of 3628000 candidate input plan with different orders.}}

 

We define original expected optimization result as *A:*

 
{code:java}
f1.join(d3, Inner, Some(nameToAttr("f1_fk3") === nameToAttr("d3_pk")))
 .join(d1, Inner, Some(nameToAttr("f1_fk1") === nameToAttr("d1_pk")))
 .join(d2, Inner, Some(nameToAttr("f1_fk2") === nameToAttr("d2_pk")))
 .join(t4.join(t3, Inner, Some(nameToAttr("t3_c2") === nameToAttr("t4_c2"))), Inner,
 Some(nameToAttr("d1_c2") === nameToAttr("t3_c1")))
 .join(t2.join(t1, Inner, Some(nameToAttr("t1_c2") === nameToAttr("t2_c2"))), Inner,
 Some(nameToAttr("d3_c2") === nameToAttr("t1_c1")))
 .join(t5.join(t6, Inner, Some(nameToAttr("t5_c2") === nameToAttr("t6_c2"))), Inner,
 Some(nameToAttr("d2_c2") === nameToAttr("t5_c1")))
{code}
 

{{and define the other one optimization result as }}{{*B*}}
{code:java}
f1.join(d3, Inner, Some(nameToAttr("f1_fk3") === nameToAttr("d3_pk")))
   .join(d2, Inner, Some(nameToAttr("f1_fk2") === nameToAttr("d2_pk")))
   .join(d1, Inner, Some(nameToAttr("f1_fk1") === nameToAttr("d1_pk")))
   .join(t4.join(t3, Inner, Some(nameToAttr("t3_c2") === nameToAttr("t4_c2"))), Inner,
     Some(nameToAttr("d1_c2") === nameToAttr("t3_c1")))
   .join(t2.join(t1, Inner, Some(nameToAttr("t1_c2") === nameToAttr("t2_c2"))), Inner,
     Some(nameToAttr("d3_c2") === nameToAttr("t1_c1")))
   .join(t5.join(t6, Inner, Some(nameToAttr("t5_c2") === nameToAttr("t6_c2"))), Inner,
     Some(nameToAttr("d2_c2") === nameToAttr("t5_c1")))
{code}
 

{{Then we use "Test 4: Star with several branches" in StarJoinCostBasedReorderSuite to test optimization results of 3628000 inputs as described above. We found that 1813600 results were candidate *A* and 1814400 results were candidate *B* because f1.join(d3).join(d1).join(d2) has the same cost with }}{{f1.join(d3).join(d2).join(d1), both Cost(200, 9200), and the candidate plan generated first will be choice.}}

Now the result of CostBasedJoinReorder rule is non-deterministic, we need to find a way to make it deterministic even if the input order is different.
Bug: Unify to `assertEqualJoinPlans ` for join reorder suites
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Aug/20 09:14
Affects Version/s: 3.0.0

Now three join reorder suites(`JoinReorderSuite`, `StarJoinReorderSuite`, `StarJoinCostBasedReorderSuite`) all contain an `assertEqualPlans` method and the logic is almost the same. We can extract the method to a single place for code simplicity.
Bug: Using groupby with rand creates different values when joining table with itself
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 28/Aug/20 13:24
Affects Version/s: 2.4.5

When running following query in a python3 notebook on a cluster with *multiple workers (>1)*, the result is not 0.0, even though I would expect it to be.
{code:java}
import pyspark.sql.functions as F
sdf = spark.range(100)
sdf = (
    sdf.withColumn("a", F.col("id") + 1)
    .withColumn("b", F.col("id") + 2)
    .withColumn("c", F.col("id") + 3)
    .withColumn("d", F.col("id") + 4)
    .withColumn("e", F.col("id") + 5)
)
sdf = sdf.groupby(["a", "b", "c", "d"]).agg(F.sum("e").alias("e"))
sdf = sdf.withColumn("x", F.rand() * F.col("e"))
sdf2 = sdf.join(sdf.withColumnRenamed("x", "xx"), ["a", "b", "c", "d"])
sdf2 = sdf2.withColumn("delta_x", F.abs(F.col('x') - F.col("xx"))).agg(F.sum("delta_x"))
sum_delta_x = sdf2.head()[0]
print(f"{sum_delta_x} should be 0.0")
assert abs(sum_delta_x) < 0.001
{code}
If the groupby statement is commented out, the code is working as expected.
Bug: Make `DataFrameReader.table` take the specified options
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/20 18:36
Affects Version/s: 3.1.0

Currently, `DataFrameReader.table` ignores the specified options. The options specified like the following are lost.

{code:java}
    val df = spark.read
      .option("partitionColumn", "id")
      .option("lowerBound", "0")
      .option("upperBound", "3")
      .option("numPartitions", "2")
      .table("h2.test.people")
{code}

We need to make `DataFrameReader.table` take the specified options.
Bug: Write Hive ORC/Parquet bucketed table with hive murmur3hash (for Hive 3)
Status: Open | Priority: Minor | Resolution: 
Created: 27/Aug/20 06:09
Affects Version/s: 3.1.0

Hive ORC/Parquet write code path is same as data source v1 code path (FileFormatWriter). This JIRA is to add the support to write Hive ORC/Parquet bucketed table with hive murmur3hash. The change is to custom `bucketIdExpression` to use hive murmur3hash when the table is Hive bucketed table, and the Hive version is 3.x.y.

 

This will allow us write Hive/Presto-compatible bucketed table for Hive 3.
Bug: [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/20 00:29
Affects Version/s: 3.1.0

The decommissioning state is a bit fragment across two places in the TaskSchedulerImpl:
 * [https://github.com/apache/spark/pull/29014/|https://github.com/apache/spark/pull/29014/files] stored the incoming decommission info messages in _TaskSchedulerImpl.executorsPendingDecommission._

 * While [https://github.com/apache/spark/pull/28619/|https://github.com/apache/spark/pull/28619/files] was storing just the executor end time in the map _TaskSetManager.tidToExecutorKillTimeMapping_ (which in turn is contained in TaskSchedulerImpl).

While the two states are not really overlapping, its a bit of a code hygiene concern to save this state in two places. 

With [https://github.com/apache/spark/pull/29422], TaskSchedulerImpl is emerging as the place where all decommissioning book keeping is kept within the driver. So consolidate the information in _tidToExecutorKillTimeMapping_ into _ExecutorDecommissionInfo._
Bug: select from table TABLESAMPLE gives wrong resultset.
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 25/Aug/20 14:46
Affects Version/s: 3.0.0

create table test(id int,name string) stored as parquet;
insert into test values (5,'Alex'),(8,'Lucy'),(2,'Mary'),(4,'Fred'),(1,'Lisa'),(9,'Eric'),(10,'Adam'),(6,'Mark'),(7,'Lily'),(3,'Evan');
SELECT * FROM test TABLESAMPLE (50 PERCENT); --> output is giving only 3 rows.
spark-sql> SELECT * FROM test TABLESAMPLE (50 PERCENT);
5 Alex
10 Adam
4 Fred

 

Expected as per the link is 5 rows -->[https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-sampling.html]

 

Also the bucket parameter for select from table TABLESAMPLE gives wrong resultset.
spark-sql> SELECT * FROM test TABLESAMPLE (BUCKET 4 OUT OF 10);
5 Alex
8 Lucy
9 Eric
1 Lisa
3 Evan

Expected is 4 records.
Bug: Un-deprecate inferring DataFrame schema from list of dictionaries
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Aug/20 17:42
Affects Version/s: 3.1.0

Inferring the schema of a DataFrame from a list of dictionaries feels natural for PySpark users, and also mirrors [basic functionality in Pandas|https://stackoverflow.com/a/20638258/877069].

This is currently possible in PySpark but comes with a deprecation warning. We should un-deprecate this behavior if there are no deeper reasons to discourage users from this API, beyond wanting to push them to use {{Row}}.
Bug: SizeEstimator class not always properly initialized in tests.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/20 11:59
Affects Version/s: 3.0.0

The SizeEstimator class reads some system properties when it is initialized. Some tests override these system properties in order to force the SizeEstimator class to produce the same results on all platforms. However there are multiple places where the SizeEstimator class is either not re-initialized after system properties have been changed or system properties are not correctly set.

This causes test failures when using OpenJ9 compiled in large heap mode. This build of OpenJ9 does not use compressed references which affects the settings that SizeEstimator is originally initialized with.

I have a PR that fixes this and will submit it shortly.
Bug: Rename EmptyHashedRelationWithAllNullKeys and simplify NAAJ generated code
Status: Resolved | Priority: Minor | Resolution: Resolved
Created: 21/Aug/20 07:16
Affects Version/s: 3.1.0

EmptyHashedRelationWithAllNullKeys is a bit of confusing for its naming, and this minor change also simplify the generated code for BHJ NAAj.
Bug: CountVectorizerModel: Remove requirement for minimum vocabulary size
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Aug/20 16:47
Affects Version/s: 3.0.0

Currently `CountVectorizer.scala` has the following requirement:
{code:java}
require(vocab.length > 0, "The vocabulary size should be > 0. Lower minDF as necessary."){code}
But this is not a necessary constraint. It should be able to function even for empty vocabulary case.

This also gives the ability to run the model over empty datasets. HashingTF works fine in such scenarios. CountVectorizer doesn't.

 

spark-user discussion reference: [http://apache-spark-user-list.1001560.n3.nabble.com/Ability-to-have-CountVectorizerModel-vocab-as-empty-td38396.html]
Bug: "path" option is added again to input paths during infer()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Aug/20 01:27
Affects Version/s: 2.4.6

When "path" option is used when creating a DataFrame, it can cause issues during infer.
{code:java}
class TestFileFilter extends PathFilter {
  override def accept(path: Path): Boolean = path.getParent.getName != "p=2"
}

val path = "/tmp"
val df = spark.range(2)
df.write.json(path + "/p=1")
df.write.json(path + "/p=2")

val extraOptions = Map(
  "mapred.input.pathFilter.class" -> classOf[TestFileFilter].getName,
  "mapreduce.input.pathFilter.class" -> classOf[TestFileFilter].getName
)

// This works fine.
assert(spark.read.options(extraOptions).json(path).count == 2)

// The following with "path" option fails with the following:
// assertion failed: Conflicting directory structures detected. Suspicious paths
//	file:/tmp
//	file:/tmp/p=1
assert(spark.read.options(extraOptions).format("json").option("path", path).load.count() === 2)
{code}
Bug: Deletes orphan directories under a warehouse dir in SQLQueryTestSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Aug/20 08:12
Affects Version/s: 3.1.0

In case that a last SQLQueryTestSuite test run is killed, it will fail in a next run because of a following reason:
{code}
[info] org.apache.spark.sql.SQLQueryTestSuite *** ABORTED *** (17 seconds, 483 milliseconds)
[info]   org.apache.spark.sql.AnalysisException: Can not create the managed table('`testdata`'). The associated location('file:/Users/maropu/Repositories/spark/spark-master/sql/core/spark-warehouse/org.apache.spark.sql.SQLQueryTestSuite/testdata') already exists.;
[info]   at org.apache.spark.sql.catalyst.catalog.SessionCatalog.validateTableLocation(SessionCatalog.scala:355)
[info]   at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:170)
[info]   at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
{code}
This ticket targets at adding code to deletes orphan directories under a warehouse dir in SQLQueryTestSuite before creating test tables.
Bug: Repartition(split) bucketed tables for join if applicable
Status: In Progress | Priority: Minor | Resolution: 
Created: 19/Aug/20 00:49
Affects Version/s: 3.1.0

[SPARK-31350|https://issues.apache.org/jira/browse/SPARK-31350] introduced coalescing one side of bucketed tables for join. Instead of coalescing, one should be able to repartition the bucketed files to increase parallelism based on the workloads.
Bug: Fix AQE aggregateMetrics java.util.NoSuchElementException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Aug/20 06:53
Affects Version/s: 3.0.0

{code:java}
// Reproduce Step
sql/test-only org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z "SPARK-32573: Eliminate NAAJ when BuildSide is EmptyHashedRelationWithAllNullKeys"
{code}
{code:java}
// Error Message
14:40:44.089 ERROR org.apache.spark.util.Utils: Uncaught exception in thread element-tracking-store-worker
14:40:44.089 ERROR org.apache.spark.util.Utils: Uncaught exception in thread element-tracking-store-worker java.util.NoSuchElementException: key not found: 12 
at scala.collection.immutable.Map$Map1.apply(Map.scala:114) 
at org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$aggregateMetrics$11(SQLAppStatusListener.scala:257) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.apache.spark.sql.execution.ui.SQLAppStatusListener.aggregateMetrics(SQLAppStatusListener.scala:256) at org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$onExecutionEnd$2(SQLAppStatusListener.scala:365) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.tryLog(Utils.scala:1971) at org.apache.spark.status.ElementTrackingStore$$anon$1.run(ElementTrackingStore.scala:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)[info] - SPARK-32573: Eliminate NAAJ when BuildSide is EmptyHashedRelationWithAllNullKeys (2 seconds, 14 milliseconds)
{code}
This issue is mainly because during AQE, while sub-plan changed, the metrics update is overwrite. for example, in this UT, change from BroadcastHashJoinExec into a LocalTableScanExec, and in the onExecutionEnd action it will try aggregate all metrics including old ones during the execution, which will cause NoSuchElementException, since the metricsType is already updated with plan rewritten. So we need to filter out those outdated metrics.
Bug: Fix the link to metrics.dropwizard.io in monitoring.md to refer the proper version
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Aug/20 17:24
Affects Version/s: 3.0.0

There are links to metrics.dropwizard.io in monitoring.md but the link targets refer the version 3.1.0, while we use 4.1.1.

Now that users can create their own metrics using the dropwizard library, it's better to fix the links to refer the proper version.
Bug: Remove fullOutput from RowDataSourceScanExec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/20 18:17
Affects Version/s: 3.1.0

`RowDataSourceScanExec` requires the full output instead of the scan output after column pruning. However, in v2 code path, we don't have the full output anymore so we just pass the pruned output. `RowDataSourceScanExec.fullOutput` is actually meaningless so we should remove it.
Bug: Tune Event Drop in Async Event Queue
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Aug/20 06:02
Affects Version/s: 3.0.0

There are scenarios where we have seen the event drop in spark, resulting in the inconsistent state for the spark Application(some time application is hung state).

For example - This can be due to the large number of parallel task processing. Producer thread keeps on adding the events to the Queue and Consumer thread is consuming the events at the slower rate compare to the Producer adding in the queue. Resulting the Queue to reach max size and events get dropped from that.

There are times if Queue Size would be little bit higher like 10 percent (or 20 percent) extra of the existing Queue size, the events can be processed preventing the event drop at that point of time.

As per the current architecture size of event Queue can be configured at start of the application by setting spark.scheduler.listenerbus.eventqueue.capacity. Once this is set there is fixed size event Queue(LinkedBlockingQueue), which cannot be changed at run time to accommodate some extra events before dropping event from the Queue

This Jira for adding a support of VariableLinkedBlockingQueue to tune the dropping of the events.

VariableLinkedBlockingQueue -> [https://www.rabbitmq.com/releases/rabbitmq-java-client/v3.5.4/rabbitmq-java-client-javadoc-3.5.4/com/rabbitmq/client/impl/VariableLinkedBlockingQueue.html]
Bug: Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Aug/20 07:56
Affects Version/s: 3.0.0

In SPARK-32290, we introduced several new types of HashedRelation
 * EmptyHashedRelation
 * EmptyHashedRelationWithAllNullKeys

They were all limited to used only in NAAJ scenario. These new HashedRelation could be applied to other scenario for performance improvements.
 * EmptyHashedRelation could also be used in Normal AntiJoin for fast stop
 * While AQE is on and buildSide is EmptyHashedRelationWithAllNullKeys, can convert NAAJ to a Empty LocalRelation to skip meaningless data iteration since in Single-Key NAAJ, if null key exists in BuildSide, will drop all records in streamedSide.

This Patch including two changes.
 * using EmptyHashedRelation to do fast stop for common anti join as well
 * In AQE, eliminate BroadcastHashJoin(NAAJ) if buildSide is a EmptyHashedRelationWithAllNullKeys
Bug: Support PostgreSQL `bpchar` type and array of char type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/20 01:50
Affects Version/s: 3.0.0

Attempting to read the following table:
{code:java}
CREATE TABLE test_table (
  test_column char(64)[]
)
{code}
results in the following exception:
{code:java}
java.sql.SQLException: Unsupported type ARRAY
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:256)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:321)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:321)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:339)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279)
	at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:203)
{code}
non-array and varchar equivalents are fine.

 

I've tracked it down to an internal function of the postgres dialect, that accounts for the special 1-byte char, but doesn't deal with different length ones, which postgres represents as bpchar: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala#L60-L61].

Relevant driver code can be found here: [https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/TypeInfoCache.java#L85-L87]

 

I'll submit a fix shortly
Bug: Add unique ID on QueryExecution to enable listeners to deduplicate 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Aug/20 05:05
Affects Version/s: 3.1.0

Somehow there's a change Spark calls QueryExecutionListener multiple times on same QueryExecution instance (even same funcName for onSuccess).

There's no unique ID on QueryExecution, hence it's a bit tricky if the listener would like to deal with same query execution only once.

This issue tracks the effort on adding unique ID on QueryExecution, so that listener can leverage the ID to deduplicate callbacks.
Bug: Use local time zone for the timestamp logged in unit-tests.log
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/20 07:46
Affects Version/s: 3.0.0

SparkFunSuite fixes the default time zone to America/Los_Angeles so the timestamp logged in unit-tests.log is also based on the fixed time zone.

It's confusable for developers whose time zone is not America/Los_Angeles.