av9ash/GitBugs huggingface dataset:
Bug: Support to manual close/release entries in MemoryStore and InMemoryRelation instead of replying on GC
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/21 07:35
Affects Version/s: 3.0.3

This PR is proposing a add-on to support to manual close entries in MemoryStore and InMemoryRelation
h3. What changes were proposed in this pull request?

Currently:
MemoryStore uses a LinkedHashMap[BlockId, MemoryEntry[_]] to store all OnHeap or OffHeap entries.
And when memoryStore.remove(blockId) is called, codes will simply remove one entry from LinkedHashMap and leverage Java GC to do release work.

This PR:
We are proposing a add-on to manually close any object stored in MemoryStore and InMemoryRelation if this object is extended from AutoCloseable.

Veifiication:
In our own use case, we implemented a user-defined off-heap-hashRelation for BHJ, and we verified that by adding this manual close, we can make sure our defined off-heap-hashRelation can be released when evict is called.
Also, we implemented user-defined cachedBatch and will be release when InMemoryRelation.clearCache() is called by this PR
h3. Why are the changes needed?

This changes can help to clean some off-heap user-defined object may be cached in InMemoryRelation or MemoryStore
h3. Does this PR introduce _any_ user-facing change?

NO
h3. How was this patch tested?

WIP

Signed-off-by: Chendi Xue [chendi.xue@intel.com|mailto:chendi.xue@intel.com]
Bug: Explain cost is not showing statistics for all the nodes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/May/21 23:33
Affects Version/s: 3.0.0

Explain cost command doesn't show statistics for all the nodes in most of the TPCDS queries

For eg: Query1

!image-2021-05-31-05-09-09-637.png!
Bug: Essential information missing in TreeNode json string
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/May/21 15:01
Affects Version/s: 3.1.1

TreeNode can be serialized to json string with the method toJSON() or prettyJson(). To avoid OOM issues, [SPARK-17426|https://issues.apache.org/jira/browse/SPARK-17426] only keep part of Seq data that can be written out to result json string.
 Essential data like [cteRelations|https://github.com/apache/spark/blob/v3.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala#L497] in node With, [branches|https://github.com/apache/spark/blob/v3.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L123] in CaseWhen will be skipped and written out as null.
Bug: Fix places about Yarn and Kubernetes in Cluster Overview and  Submitting Applications pages
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/May/21 06:19
Affects Version/s: 3.1.2

Currently there are some places in the "Cluster Overview" doc page that are not mentioning Kubernetes, where Kubernetes is actually supported. Also there is a place in the "Submitting Applications" doc page, where it gives the user a sample spark-submit command for Yarn cluster manager, but that command is not runnable in bash.
Bug: add hostNetwork feature to executor pod
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/May/21 06:00
Affects Version/s: 3.1.2

Add new feature hostNetwork to executor pod 

Allow spark users to perceive use k8s hostNetwork

In the process of the company's business promotion, the function of k8s hostNetwork is used. But it is found that spark does not support enable hostNetwork in the configuration.
Bug: Make query stage materialized more readable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/May/21 04:16
Affects Version/s: 3.2.0

Currently, we use `resultOption().get.isDefined` to check if a query stage has materialized. The code is not readable at a glance. It's better to use a new method like `isMaterialized` to define it.
Bug: TimestampType: OverflowError: mktime argument out of range 
Status: Open | Priority: Minor | Resolution: 
Created: 25/May/21 11:54
Affects Version/s: 3.1.1

This issue occurs, for example, when trying to create a data frame from Python {{datetime}} objects that are "out of range" where "out of range" is platform-dependent due to the use of [{{time.mktime}}|https://docs.python.org/3/library/time.html#time.mktime] in {{TimestampType.toInternal}}:

{code}
import datetime
spark_session.createDataFrame([(datetime.datetime(9999, 12, 31, 0, 0),)])
{code}

A more direct way to reproduce the issue is by invoking {{TimestampType.toInternal}} directly:
{code}
import datetime
from pyspark.sql.types import TimestampType
dt = datetime.datetime(9999, 12, 31, 0, 0)
TimestampType().toInternal(dt)
{code}

The suggested improvement is to avoid using {{time.mktime}} to increase the range of {{datetime}} values. A possible implementation may look as follows:

{code}
import datetime
import pytz

EPOCH_UTC = datetime.datetime(1970, 1, 1).replace(tzinfo=pytz.utc)
LOCAL_TZ = datetime.datetime.now().astimezone().tzinfo

def toInternal(dt):
	if dt is not None:
		dt = dt if dt.tzinfo else dt.replace(tzinfo=LOCAL_TZ)
		dt_utc = dt.astimezone(pytz.utc)
		td = dt_utc - EPOCH_UTC
		return (td.days * 86400 + td.seconds) * 10 ** 6 + td.microseconds
{code}

This relies on the ability to derive the local timezone. Other mechanisms may be used to what is suggested above.

Test cases include:
{code}
dt1 = datetime.datetime(2021, 5, 25, 12, 23)
dt2 = dt1.replace(tzinfo=pytz.timezone('Europe/Zurich'))
dt3 = datetime.datetime(9999, 12, 31, 0, 0)
dt4 = dt3.replace(tzinfo=pytz.timezone('Europe/Zurich'))

toInternal(dt1) == TimestampType().toInternal(dt1)
toInternal(dt2) == TimestampType().toInternal(dt2)
toInternal(dt3) # TimestampType().toInternal(dt3) fails
toInternal(dt4) == TimestampType().toInternal(dt4)
{code}
Bug: GenerateSafeProjection.generate will generate SpecificSafeProjection class, but if column is array type or map type, the code cannot be reused which impact the query performance
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 24/May/21 08:15
Affects Version/s: 2.4.5

Reproduce steps:
 # create a new table with array type: create table test_code_gen(a array<int>);
 # Add log4j.logger.org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator = DEBUG to log4j.properties;
 # Enter spark-shell, fire a query: spark.sql("select * from test_code_gen").collect
 # Everytime, Dataset.collect is called, SpecificSafeProjection class is generated, but the code for the class cannot be reused because everytime the id for two variables in the generated class is changed: MapObjects_loopValue and MapObjects_loopIsNull. So even the class generated before has been cached, new code cannot match the cache key so that new code need to be compiled again which cost some time.  The time cost for compile is increasing with the growth of column number, for wide table, this cost can more than 2s. 
{code:java}
object MapObjects {
  private val curId = new java.util.concurrent.atomic.AtomicInteger()
 val id = curId.getAndIncrement()
 val loopValue = s"MapObjects_loopValue$id"
 val loopIsNull = if (elementNullable) {
   s"MapObjects_loopIsNull$id"
 } else {
   "false"
 }
{code}
First time run: 
{code:java}
class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
 private int MapObjects_loopValue1;
 private boolean MapObjects_loopIsNull1;
 private UTF8String MapObjects_loopValue2;
 private boolean MapObjects_loopIsNull2;
}
{code}
Second time run:
{code:java}
class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
 private int MapObjects_loopValue3;
 private boolean MapObjects_loopIsNull3;
 private UTF8String MapObjects_loopValue4;
 private boolean MapObjects_loopIsNull4;
}{code}

Expectation:
The code generated by GenerateSafeProjection can be reused if the query is same.
Bug: Upgrade dropwizard metrics to 4.2.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/May/21 03:18
Affects Version/s: 3.2.0

Dropwizard metrics 4.2.0 was released last week and it contains some improvements.
https://github.com/dropwizard/metrics/releases/tag/v4.2.0
Bug: Verify checksums of downloaded artifacts in build/mvn
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/21 16:11
Affects Version/s: 2.4.7

build/mvn is a convenience script that will automatically download Maven (and Scala) if not already present. While it downloads from official ASF mirrors, it does not check the checksum of the artifact, which is available as a .sha512 file from ASF servers.

The risk of a supply chain attack is a bit less theoretical here than usual, because artifacts are downloaded from any of several mirrors worldwide, and injecting a malicious copy of Maven in any one of them might be simpler and less noticeable than injecting it into ASF servers.

(Note, Scala's download site does not seem to provide a checksum. They do all come from Lightbend, at least, not N mirrors. Not much we can do there.)
Bug: ARM CI failed: failed to validate maven sha512
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/21 06:56
Affects Version/s: 3.2.0

Log:

 

Veryfing checksum from /home/jenkins/workspace/spark-master-test-maven-arm/build/apache-maven-3.6.3-bin.tar.gz.sha512
 *Unknown option: q*
 *Type shasum -h for help*
 Bad checksum from [https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz.sha512]

 

Looks like shasum validation had some wrong change in:

[https://github.com/apache/spark/commit/6c5fcac6b787d01ebf3d9f53410db2c894ab9abd#diff-590845f9441f6be1f05f517fd1caf31d64d0b5126ea9a2a13d79c74f761417ce]

 

[1] [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/]

[2] [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/]
Bug: Mark K8s secrets and config maps as immutable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/May/21 11:22
Affects Version/s: 3.2.0

Kubernetes supports marking secrets and config maps as immutable to gain performance. 

[https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable]

[https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable]

For K8s clusters that run many thousands of Spark applications, this can yield significant reduction in load on the kube-apiserver.

From the K8s docs:
{quote}For clusters that extensively use Secrets (at least tens of thousands of unique Secret to Pod mounts), preventing changes to their data has the following advantages:
 * protects you from accidental (or unwanted) updates that could cause applications outages
 * improves performance of your cluster by significantly reducing load on kube-apiserver, by closing watches for secrets marked as immutable.{quote}
 

For any secrets and config maps we create in Spark that are immutable, we could mark them as immutable by including the following when building the secret/config map
{code:java}
.withImmutable(true)
{code}
This feature has been supported in K8s as beta since K8s 1.19 and as GA since K8s 1.21
Bug: Upgrade scalatestplus artifacts to 3.2.9.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/May/21 12:56
Affects Version/s: 3.2.0

scalatestplus artifacts seem to be renamed and their latest release supports dotty.

* https://github.com/scalatest/scalatestplus-scalacheck
* https://github.com/scalatest/scalatestplus-mockito
* https://github.com/scalatest/scalatestplus-selenium
Bug: Use the SHA of the latest commit when checking out databricks/tpcds-kit
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/May/21 10:24
Affects Version/s: 3.0.2

This ticket proposes to use the SHA of the latest commit ([2a5078a782192ddb6efbcead8de9973d6ab4f069](https://github.com/databricks/tpcds-kit/commit/2a5078a782192ddb6efbcead8de9973d6ab4f069)) when checking out `databricks/tpcds-kit`. This can prevent the test workflow from breaking accidentally if the repository changes drastically.
Bug: Refactor sort merge join code-gen be agnostic to join type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/May/21 22:38
Affects Version/s: 3.2.0

This is a pre-requisite of [https://github.com/apache/spark/pull/32476,] in discussion of [https://github.com/apache/spark/pull/32476#issuecomment-836469779] . This is to refactor sort merge join code-gen to depend on streamed/buffered terminology, which makes the code-gen agnostic to different join types and can be extended to support other join types than inner join.
Bug: Split generated switch code into pieces in ExpandExec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/21 15:12
Affects Version/s: 3.2.0

This ticket aim at splitting generated switch code into smaller ones in `ExpandExec`. In the current master, even a simple query like the one below generates a large method whose size (`maxMethodCodeSize:7448`) is close to `8000` (`CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT`);
{code:java}
scala> val df = Seq(("2016-03-27 19:39:34", 1, "a"), ("2016-03-27 19:39:56", 2, "a"), ("2016-03-27 19:39:27", 4, "b")).toDF("time", "value", "id")
scala> val rdf = df.select(window($"time", "10 seconds", "3 seconds", "0 second"), $"value").orderBy($"window.start".asc, $"value".desc).select("value")
scala> sql("SET spark.sql.adaptive.enabled=false")
scala> import org.apache.spark.sql.execution.debug._ 
scala> rdf.debugCodegen

Found 2 WholeStageCodegen subtrees.
== Subtree 1 / 2 (maxMethodCodeSize:7448; maxConstantPoolSize:189(0.29% used); numInnerClasses:0) ==
                                    ^^^^
*(1) Project [window#34.start AS _gen_alias_39#39, value#11]
+- *(1) Filter ((isnotnull(window#34) AND (cast(time#10 as timestamp) >= window#34.start)) AND (cast(time#10 as timestamp) < window#34.end))
   +- *(1) Expand [List(named_struct(start, precisetimestampcon...
{code}
Bug: Increase the stack size of JVM for Java 11 build test
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/May/21 17:13
Affects Version/s: 3.2.0

After merging https://github.com/apache/spark/pull/32439, there is flaky error from the Github action job "Java 11 build with Maven":

{code:java}
Error:  ## Exception when compiling 473 sources to /home/runner/work/spark/spark/sql/catalyst/target/scala-2.12/classes
java.lang.StackOverflowError
scala.reflect.internal.Trees.itransform(Trees.scala:1376)
scala.reflect.internal.Trees.itransform$(Trees.scala:1374)
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:51)
{code}

We can resolve it by increasing the stack size of JVM.
Bug: Allow to turn off the normalization applied by static PageRank utilities
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/21 17:46
Affects Version/s: 3.1.1

Since SPARK-18847, static PageRank computations available in `PageRank.scala` are normalizing the sum of the ranks after the fixed number of iterations has completed, and *there is no way for a developer to access the raw non normalized ranks values*.

Since SPARK-29877 one can run a fixed number of PageRank iterations starting from previous `preRankGraph`'s ranks.
 This nice feature open the door for interesting *incremental algorithms*, for example:
 "Run some initial pagerank iterations using `PageRank.runWithOptions` and then update the graph's edges and update the ranks with a call to `PageRank.runWithOptionsWithPreviousPageRank`, and so on...".

This kind of algorithms would highly benefit (precision gain) from being allowed to manipulate directly the raw ranks (and not the normalized ones) in the case where the graph has a substantial proportion of sinks (vertices without outgoing edges).

It would be nice to add a method's signature having a boolean that allows to turn off the automatic normalization run at the end of `PageRank.runWithOptions` and `PageRank.runWithOptionsWithPreviousPageRank`, making the developers free to apply the normalization only when they really need it.
Bug: Fallback config should override defaultValue
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/May/21 01:28
Affects Version/s: 3.2.0

Some config use `.fallbackConf` to reference another config but lost the defaultValue.

 

A negative case is `SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.defaultValue` would return `None`. It could mislead user.
Bug: Fix issue of the createTable when externalCatalog is InMemoryCatalog
Status: In Progress | Priority: Minor | Resolution: 
Created: 09/May/21 14:32
Affects Version/s: 3.0.0

In the case of externalCatalog is InMemoryCatalog, Using a different SparkSession will report the following error when create the same table , as follow
{code:java}
sparkSession.sql("create table if not exists t1 (id Int) using orc")
sparkSession.sql("insert into t1 values(1)")

// will report error as follow
sparkSession2.sql("create table if not exists t1 (id Int) using orc") {code}
{code:java}
Can not create the managed table('`default`.`t1`'). The associated location('file:***/t1') already exists.
{code}
Bug: Add string-to-number conversion support to JacksonParser
Status: In Progress | Priority: Minor | Resolution: 
Created: 11/May/21 16:46
Affects Version/s: 2.4.7

In the current implementation, spark.read.json doesn't convert numbers represented as string even though types are specified by schema.
Here is anexample.
{code}
$ cat test.json
{ "value": "foo" }                                                                                                                                
{ "value": "100" }                                                                                                                                
{ "value": "257" }                                                                                                                                
{ "value": "32768" }                                                                                                                              
{ "value": "2147483648" }                                                                                                                         
{ "value": "2.71f" }                                                                                                                              
{ "value": "1.0E100" }  

$ bin/spark-shell
import org.apache.spark.sql.types._
val df = spark.read.
  schema(StructType(StructField("value", DoubleType)::Nil)).json("test.json")
df.show
+-----+
|value|
+-----+
| null|
| null|
| null|
| null|
| null|
| null|
| null|
+-----+
{code}
Bug: Set maximum Java heap used for release build
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/21 21:39
Affects Version/s: 3.0.2

When I was cutting RCs for 2.4.8, I frequently encountered OOM during building using mvn. It happens many times until I increased the heap memory setting.

I am not sure if other release managers encounter the same issue. I will try to increase the heap memory setting.
Bug: Use the newer dsdgen for TPCDSQueryTestSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/May/21 03:54
Affects Version/s: 3.2.0

This PR intends to replace `maropu/spark-tpcds-datagen` with `databricks/tpcds-kit` for using a newer dsdgen and update the golden files in `tpcds-query-results`.
Bug: Upgrade Jersey to 2.34
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/21 13:05
Affects Version/s: 3.0.2

CVE-2021-28168, a local information disclosure vulnerability, is reported.
Spark 3.1.1, 3.0.2 and 3.2.0 use an affected version 2.30.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-28168
Bug: Keep benchmark result consistent between spark-submit and SBT
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/May/21 23:03
Affects Version/s: 3.2.0

Currently benchmark can be done in two ways: {{spark-submit}}, or SBT command. However in the former Spark will miss some properties such as {{IS_TESTING}}, which is useful to turn on/off some behavior like codegen. Therefore, the result could differ with the two methods. In addition, the benchmark GitHub workflow is using the {{spark-submit}} approach.

This propose to set {{IS_TESTING}} to true in {{BenchmarkBase}} so that it is always on.
Bug: Simplify the way to get classes from ClassBodyEvaluator in CodeGenerator.updateAndGetCompilationStats method
Status: In Progress | Priority: Trivial | Resolution: 
Created: 13/May/21 09:33
Affects Version/s: 3.2.0

SPARK-35253 upgraded janino from 3.0.16 to 3.1.4, {{ClassBodyEvaluator}} provides the {{getBytecodes}} method to get
the mapping from {{ClassFile.getThisClassName}} to {{ClassFile.toByteArray}} directly in this version and we don't need to get this variable by reflection api anymore.
Bug: Storage UI tab Storage Level tool tip correction
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/May/21 14:09
Affects Version/s: 3.1.1

Storage UI tab Storage Level tool tip correction required.
|</span>|
| |</th><th width="" class="">|
| |<span data-toggle="tooltip" title="StorageLevel displays where the persisted RDD is stored, format of the persisted RDD (serialized or de-serialized) andreplication factor of the persisted RDD">|
| |Storage Level|
| |</span>|

please change *andreplication * to *and replication*
Bug: Re-cleanup `procedure syntax is deprecated` compilation warning in Scala 2.13
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/May/21 04:16
Affects Version/s: 3.2.0

Similar to SPARK-29291 and SPARK-33352, just to track Spark 3.2.0

 

There are still some compilation warnings about `procedure syntax is deprecated`:

 
{code:java}
[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/MapOutputTracker.scala:723: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `registerMergeResult`'s return type
[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/MapOutputTracker.scala:748: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `unregisterMergeResult`'s return type
[WARNING] [Warn] /spark/core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala:223: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `testSimpleSpillingForAllCodecs`'s return type
[WARNING] [Warn] /spark/mllib-local/src/test/scala/org/apache/spark/ml/linalg/BLASBenchmark.scala:53: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `runBLASBenchmark`'s return type
[WARNING] [Warn] /spark/sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala:110: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `assertEmptyRootPath`'s return type
[WARNING] [Warn] /spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala:602: [deprecation @  | origin= | version=2.13.0] procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `executeCTASWithNonEmptyLocation`'s return type
{code}