av9ash/GitBugs huggingface dataset:
Bug: PySpark Context should have getCheckpointDir() method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/20 14:56
Affects Version/s: 3.1.0

To match the Scala API, PySpark should offer a direct way to get the checkpoint dir.
{code:scala}
scala> spark.sparkContext.setCheckpointDir("/tmp/spark/checkpoint")
scala> spark.sparkContext.getCheckpointDir
res3: Option[String] = Some(file:/tmp/spark/checkpoint/34ebe699-bc83-4c5d-bfa2-50451296cf87)
{code}
Currently, the only was to do that from PySpark is via the underlying Java context:
{code:python}
>>> spark.sparkContext.setCheckpointDir('/tmp/spark/checkpoint/')
>>> sc._jsc.sc().getCheckpointDir().get()
'file:/tmp/spark/checkpoint/ebf0fab5-edbc-42c2-938f-65d5e599cf54'
{code}
Bug: The example of expressions.Aggregator in Javadoc / Scaladoc is wrong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/20 19:25
Affects Version/s: 3.0.0

There is an example of expressions.Aggregator in Javadoc and Scaladoc like as follows.
{code:java}
val customSummer =  new Aggregator[Data, Int, Int] {
  def zero: Int = 0
  def reduce(b: Int, a: Data): Int = b + a.i
  def merge(b1: Int, b2: Int): Int = b1 + b2
  def finish(r: Int): Int = r
}.toColumn(){code}
But this example doesn't work because it doesn't define bufferEncoder and outputEncoder.
Bug: Change default fallback versions in HiveExternalCatalogVersionsSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Sep/20 10:32
Affects Version/s: 2.4.7

Temporarily we set the fallback versions at https://issues.apache.org/jira/browse/SPARK-31716.  Spark 2.3 is EOL and 2.4.7. is released. 

We should better match it.
Bug: Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1
Status: Open | Priority: Minor | Resolution: 
Created: 21/Sep/20 12:44
Affects Version/s: 3.0.1

I'm seeing the test 'CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1' fail on IBM Z with the wrong number of partitions (1 instead of 2). It's strange because none of the other tests fail.

I'd be grateful for any hints as to how the number of partitions is calculated. Could that calculation be affected by incorrect unsafe code? Is there a way to trace the calculation?

{noformat}
CoalesceShufflePartitionsSuite:
- determining the number of reducers: aggregate operator(minNumPostShufflePartitions: 5)
- determining the number of reducers: join operator(minNumPostShufflePartitions: 5)
- determining the number of reducers: complex query 1(minNumPostShufflePartitions: 5)
- determining the number of reducers: complex query 2(minNumPostShufflePartitions: 5)
- determining the number of reducers: plan already partitioned(minNumPostShufflePartitions: 5)
- determining the number of reducers: aggregate operator
- determining the number of reducers: join operator
- determining the number of reducers: complex query 1 *** FAILED ***
 1 did not equal 2 (CoalesceShufflePartitionsSuite.scala:221)
- determining the number of reducers: complex query 2
- determining the number of reducers: plan already partitioned
- SPARK-24705 adaptive query execution works correctly when exchange reuse enabled
- Do not reduce the number of shuffle partition for repartition
- Union two datasets with different pre-shuffle partition number{noformat}
Bug: python section in quickstart page doesn't display SPARK_VERSION correctly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/20 22:16
Affects Version/s: 2.2.3

[https://github.com/apache/spark/blame/master/docs/quick-start.md#L402]

It should be {{{{site.SPARK_VERSION}}}} rather than {site.SPARK_VERSION}
Bug: Add Parquet Timestamp output configuration to docs
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 01/Sep/20 16:52
Affects Version/s: 2.3.0

{{Spark 2.3.0 added the spark.sql.parquet.outputTimestampType configuration option for controlling the underlying datatype used when writing Timestamp column types into parquet files. This option is helpful for compatibility with external systems that need to read the output from Spark.}}

{{This was never exposed in the documentation. Fix should be applied to docs for both the next 3.x release and 2.4.x release.}}
Bug: cleanCheckpoints config does not clean all checkpointed RDDs on shutdown
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 25/Sep/20 21:40
Affects Version/s: 2.4.6

Maybe it's just that the documentation needs to be updated, but I found this surprising:
{code:python}
$ pyspark
...
>>> spark.conf.set('spark.cleaner.referenceTracking.cleanCheckpoints', 'true')
>>> spark.sparkContext.setCheckpointDir('/tmp/spark/checkpoint/')
>>> a = spark.range(10)
>>> a.checkpoint()
DataFrame[id: bigint]                                                           
>>> exit(){code}
The checkpoint data is left behind in {{/tmp/spark/checkpoint/}}. I expected Spark to clean it up on shutdown.

The documentation for {{spark.cleaner.referenceTracking.cleanCheckpoints}} says:
{quote}Controls whether to clean checkpoint files if the reference is out of scope.
{quote}
When Spark shuts down, everything goes out of scope, so I'd expect all checkpointed RDDs to be cleaned up.

For the record, I see the same behavior in both the Scala and Python REPLs.

Evidence the current behavior is confusing:
 * [https://stackoverflow.com/q/52630858/877069]
 * [https://stackoverflow.com/q/60009856/877069]
 * [https://stackoverflow.com/q/61454740/877069]
Bug: Decouple bucket filter pruning and bucket table scan
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Sep/20 07:04
Affects Version/s: 3.1.0

As a followup from discussion in [https://github.com/apache/spark/pull/29804#discussion_r493100510] . Currently in data source v1 file scan `FileSourceScanExec`, bucket filter pruning will only take effect with bucket table scan - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L542] . However this is unnecessary, as bucket filter pruning can also happen if we disable bucketed table scan. This help query leverage the benefit from bucket filter pruning to save CPU/IO to not read unnecessary bucket files, and do not bound by bucket table scan when the parallelism of tasks is a concern.
Bug: setting spark.network.timeout no longer sets spark.storage.blockManagerSlaveTimeoutMs
Status: Open | Priority: Minor | Resolution: 
Created: 01/Sep/20 18:18
Affects Version/s: 3.0.0

Setting spark.network.timeout no longer sets spark.storage.blockManagerSlaveTimeoutMs

 

Reproducible by setting spark.network.timeout=3600s and spark.executor.heartbeatInterval=1800s at startup. Workaround by setting spark.storage.blockManagerSlaveTimeoutMs=3600s at startup.
Bug: GenerateExec should require output column ordering
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/20 00:25
Affects Version/s: 3.0.1

{{GenerateExec}} should require column ordering since it binds its input rows directly with its {{requiredChildOutput}} without using the child's output schema. In this case, changing input column ordering will result in {{GenerateExec}} binding the wrong schema to the input columns.
Bug: Fix ScalaReflection issue for Case Classes with complex type aliased data
Status: Open | Priority: Minor | Resolution: 
Created: 29/Sep/20 17:14
Affects Version/s: 3.0.1

THIS BUG WAS DISCOVERED BY AHMED METWALLY AND WAS FIXED BY GAURAV AHUJA WHILE BOTH WERE WORKING AT UBER INC.

 

BELOW IS THE CONTEXT FROM GAURAV AHUJA.

 

Context:
 * Creating a dataframe from a RDD[T] fails with scala.matchError. T is a class with complex type aliased param eg: type TypeAliasedArray = Array[_].
 * After debugging I found that the code path fails to create a serializer for class T.
 ** In order to create the serializer, spark first finds all the params for this class. Then uses reflection to find dataType for each param.
 ** The private function dataTypeFor is called for each parameter fieldType.
 ** The private function dataTypeFor throws an error if the filedType is a Type Aliased array.
 ** This is because it fails to find the element type for this type aliased array.

Fix:
 * Dealias the type before extracting the element type of the type aliased array

 

 

BELOW IS THE FIX FROM GAURAV AHUJA IN org/apache/spark/sql/catalyst/ScalaReflection.scala. ONLY ONE LINE NEEDS TO BE CHANGED, LINE 104.

 

` val TypeRef(_, _, Seq(elementType)) = tpe`

SHOULD BE CHANGED TO

` val TypeRef(_, _, Seq(elementType)) = tpe.dealias`
Bug: Add ability to override default remote repos with internal repos only
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Sep/20 19:02
Affects Version/s: 3.0.1

Building spark internally in orgs where access to outside internet is not allowed takes a long time because unsuccessful attempts are made to download artifacts from repositories which are not accessible. The unsuccessful attempts unnecessarily add significant amount of time to the build. I have seen a difference of up-to 1hr for some runs.
Bug: Potential SQLMetrics missed which might cause WEB UI display issue while AQE is on.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/20 10:37
Affects Version/s: 3.0.0

In current AQE execution, there might be a following scenario which might cause SQLMetrics being incorrectly override.
 # Stage A and B are created, and UI updated thru event onAdaptiveExecutionUpdate.
 # Stage A and B are running. Subquery in stage A keep updating metrics thru event onAdaptiveSQLMetricUpdate.
 # Stage B completes, while stage A's subquery is still running, updating metrics.
 # Completion of stage B triggers new stage creation and UI update thru event onAdaptiveExecutionUpdate again (just like step 1).

 

But it's very hard to re-produce this issue, since it was only happened with high concurrency. For the fix, I suggested that we might be able to keep all duplicated metrics instead of updating it every time.
Bug: Provide better exception on temporary view against DataFrameWriterV2
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 21/Sep/20 23:39
Affects Version/s: 3.1.0

DataFrameWriterV2 doesn't handle fail-back if catalog.loadTable doesn't provide any Table instance. This ends up leading temp view to NoSuchTableException.

It's OK to fail for such case unless we want to resolve it later like DataFrameWriter.insertInto, but throwing NoSuchTableException is probably confusing, as view is loaded via catalog.loadTable and fails with capability check, not NoSuchTableException.

We could check in prior whether the table identifier refers temp view, and provide better exception.
Bug: Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Sep/20 13:04
Affects Version/s: 3.1.0

This PR aims at refactoring code in `RewriteCorrelatedScalarSubquery` for replacing `ExprId`s in a bottom-up manner instead of doing in a top-down one.

This PR comes from the talk with @cloud-fan in https://github.com/apache/spark/pull/29585#discussion_r490371252.
Bug: Migrate DESCRIBE COLUMN to new resolution framework
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Sep/20 23:10
Affects Version/s: 3.1.0

Migrate "DESCRIBE tbl colname" to new resolution framework so that it has a consistent resolution rule for v1/v2 commands.

Currently, the following resolves to a table instead of a temp view:

{code:java}
sql("CREATE TABLE testcat.ns.t (id bigint) USING foo")
sql("CREATE TEMPORARY VIEW t AS SELECT 2 as id")
sql("USE testcat.ns")
sql("DESCRIBE t id") // 't' is resolved to testcat.ns.t
{code}
Bug: AQE plan string should only display one plan when the initial and the current plan are the same
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Sep/20 18:28
Affects Version/s: 3.1.0

Currently, the AQE plan string displays both the initial plan and the current or the final plan. This can be redundant when the initial plan and the current physical plan are exactly the same. For instance, the `EXPLAIN` command will not actually execute the query, and thus the plan string will never change, but currently, the plan string still shows both the current and the initial plan:

 
{code:java}
AdaptiveSparkPlan (8)
+- == Current Plan ==
   Sort (7)
   +- Exchange (6)
      +- HashAggregate (5)
         +- Exchange (4)
            +- HashAggregate (3)
               +- Filter (2)
                  +- Scan parquet default.explain_temp1 (1)
+- == Initial Plan ==
   Sort (7)
   +- Exchange (6)
      +- HashAggregate (5)
         +- Exchange (4)
            +- HashAggregate (3)
               +- Filter (2)
                  +- Scan parquet default.explain_temp1 (1)
{code}
When the initial and the current plan are the same, there should be only one plan string displayed. For example
{code:java}
AdaptiveSparkPlan (8)
+- Sort (7)
   +- Exchange (6)
      +- HashAggregate (5)
         +- Exchange (4)
            +- HashAggregate (3)
               +- Filter (2)
                  +- Scan parquet default.explain_temp1 (1){code}
Bug: Introduce SQL physical plan rule to decide enable/disable bucketing 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Sep/20 18:35
Affects Version/s: 3.1.0

Discussed with [~cloud_fan] offline, it would be better that we can decide enable/disable SQL bucketing automatically according to query plan. Currently bucketing is enabled by default (`spark.sql.sources.bucketing.enabled`=true), so for all bucketed tables in the query plan, we will use bucket table scan (all input files per the bucket will be read by same task). This has the drawback that if the bucket table scan is not benefitting at all (no join/groupby/etc in the query), we don't need to use bucket table scan as it would restrict the # of tasks to be # of buckets and might hurt parallelism.

 

The proposed change is to introduce a physical plan rule (right before `ensureRequirements`):

(1).transformUp() physical plan, matching SparkPlan operator which is FileSourceScanExec, if optionalBucketSet is set, enabling bucket scan (bucket filter in this case).

(2).transformUp() physical plan, matching SparkPlan operator which is SparkPlanWithInterestingPartitioning.

SparkPlanWithInterestingPartitioning: the plan is in \{SortMergeJoinExec, ShuffledHashJoinExec, HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec, etc, which has HashClusteredDistribution/ClusteredDistribution in requiredChildDistribution}, and its requiredChildDistribution HashClusteredDistribution/ClusteredDistribution on its underlying FileSourceScanExec's bucketed columns.

(3).for any child of SparkPlanWithInterestingPartitioning, which does not satisfy the plan's requiredChildDistribution: go though the child's sub query plan tree.
 if (3.1).all node's outputPartitioning is same as child, and all node's requiredChildDistribution is UnspecifiedDistribution.
 and (3.2).the leaf node is FileSourceScanExec on bucketed table and
 and (3.3).if enabling bucket scan for this FileSourceScanExec, the outputPartitioning of FileSourceScanExec satisfies requiredChildDistribution of SparkPlanWithInterestingPartitioning.
 If (3.1),(3.2),(3.3) are all true, enabling bucket scan for this FileSourceScanExec. And double check the new child of SparkPlanWithInterestingPartitioning satisfies requiredChildDistribution.

 

The idea of SparkPlanWithInterestingPartitioning, is inspired from "interesting order" in "Access Path Selection in a Relational Database Management System"([http://www.inf.ed.ac.uk/teaching/courses/adbs/AccessPath.pdf]).
Bug: Empty file for the first partition
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 29/Sep/20 15:36
Affects Version/s: 3.0.1

If I create Dataframe with 1 row, Spark will create empty file for the first partition.

 

Example:

val df = Seq(1).toDF("col1").repartition(8)

df1.write.csv("/csv")

 

I got 2 files. The first contains the first partition and the second contains single row from another partition. It is valid also for parquet, text and etc.
Bug: In OracleDialect, "RowID" SQL type should be converted into "String" Catalyst type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Sep/20 22:28
Affects Version/s: 2.4.7

Most JDBC drivers use long SQL type for dataset row ID:

 

(in org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils)
{code:java}
private def getCatalystType(
 sqlType: Int,
 precision: Int,
 scale: Int,
 signed: Boolean): DataType = {
 val answer = sqlType match {
 // scalastyle:off
 ...
 case java.sql.Types.ROWID => LongType
...
 case _ =>
 throw new SQLException("Unrecognized SQL type " + sqlType)
 // scalastyle:on
 }
if (answer == null)
{ throw new SQLException("Unsupported type " + JDBCType.valueOf(sqlType).getName) }
answer
{code}
 

Oracle JDBC drivers (of all versions) are rare exception, only String value can be extracted:

 

(in oracle.jdbc.driver.RowidAccessor, decompiled bytecode)
{code:java}
...
String getString(int var1) throws SQLException
{ return this.isNull(var1) ? null : this.rowData.getString(this.getOffset(var1), this.getLength(var1), this.statement.connection.conversion.getCharacterSet((short)1)); }
Object getObject(int var1) throws SQLException
{ return this.getROWID(var1); }
...
{code}
 

This caused an exception to be thrown when importing datasets from an Oracle DB, as reported in [https://stackoverflow.com/questions/52244492/spark-jdbc-dataframereader-fails-to-read-oracle-table-with-datatype-as-rowid:]
{code:java}
 
 {{18/09/08 11:38:17 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 5.0 (TID 23, gbrdsr000002985.intranet.barcapint.com, executor 21): java.sql.SQLException: Invalid column type: getLong not implemented for class oracle.jdbc.driver.T4CRowidAccessor at oracle.jdbc.driver.GeneratedAccessor.getLong(GeneratedAccessor.java:440)
 at oracle.jdbc.driver.GeneratedStatement.getLong(GeneratedStatement.java:228)
 at oracle.jdbc.driver.GeneratedScrollableResultSet.getLong(GeneratedScrollableResultSet.java:620)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$8.apply(JdbcUtils.scala:365)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$8.apply(JdbcUtils.scala:364)}}
 
{code}
 

Therefore, the default SQL type => Catalyst type conversion rule should be overriden in OracleDialect. Specifically, the following rule should be added:
{code:java}
case Types.ROWID => Some(StringType)
{code}
Bug: Upgrade fabric8 to 4.10.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/20 07:22
Affects Version/s: 3.1.0

According to [fabric8's compatibility matrix|https://github.com/fabric8io/kubernetes-client#compatibility-matrix], the current version (4.9.2) is not compatible with k8s 1.18.0.


In practice, we have not encountered any issues running spark against k8s 1.18.0, but it seems reasonable to track fabric8's declared compatibility.
Bug: Display time series view for task metrics in history server
Status: In Progress | Priority: Minor | Resolution: 
Created: 30/Sep/20 07:43
Affects Version/s: 3.1.0

Event log contains all tasks' metrics data, which are useful for performance debugging. By now spark UI only displays final aggregation results, much information is hidden by this way. If spark UI could provide time series data view, it would be more helpful to performance debugging problems. We would like to build application statistics page in history server based on task metrics to provide more straight forward insight for spark application.
Bug: CostBasedJoinReorder optimizer rule should be idempotent
Status: In Progress | Priority: Minor | Resolution: 
Created: 25/Sep/20 16:47
Affects Version/s: 3.1.0

The CostBasedJoinReorder has currently the following comment:
{code}
    // Since join costs in AQP can change between multiple runs, there is no reason that we have an
    // idempotence enforcement on this batch. We thus make it FixedPoint(1) instead of Once.
{code} 

This is incorrect reasoning.
In the optimizer, we mean that a batch is idempotent, when applying it for the second time in a row (without changing anything) it does not change the plan at all.
Bug: Fix Hive UDF not support decimal type in complex type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Sep/20 10:34
Affects Version/s: 3.1.0

This pr aims to support Hive UDF when input complex type contains decimal type.

Before this pr, we failed in this code.
{code:java}
class ArraySumUDF extends UDF {
 import scala.collection.JavaConverters._
 def evaluate(values: java.util.List[java.lang.Double]): java.lang.Double = {
 var r = 0d
 for (v <- values.asScala) {
 r += v
 }
 r
 }
}

sql(s"CREATE FUNCTION testArraySum AS '${classOf[ArraySumUDF].getName}'")
sql("SELECT testArraySum(array(1, 1.1, 1.2))")
-- failed msg
Error in query: No handler for UDF/UDAF/UDTF 'ArraySumUDF': org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method for class ArraySumUDF with (array<decimal(11,1)>). Possible choices: _FUNC_(array<double>) ; line 1 pos 7
{code}
Bug: Migrate REFRESH TABLE to new resolution framework
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Sep/20 20:08
Affects Version/s: 3.1.0

Migrate "REFRESH TABLE" to new resolution framework so that it has a consistent resolution rule for v1/v2 commands.

Currently, the following resolves to a table instead of a temp view:
{code:java}
sql("CREATE TABLE testcat.ns.t (id bigint) USING foo")
sql("CREATE TEMPORARY VIEW t AS SELECT 2")
sql("USE testcat.ns")
sql("REFRESH TABLE t") // 't' is resolved to testcat.ns.t
{code}
Bug: pyspark on yarn  executor hang after some time
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 25/Sep/20 00:44
Affects Version/s: 3.0.0

same pyspark program,  previously it works ok, 
but these few days it always keep hang after running some time.  
the task is simple, fetch images from url and use PIL image library to process images 
finally write to tensorflow tfrecords file
Bug: Add jakarta.servlet-api test dependency to yarn module to avoid classpath badcase of UTs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Sep/20 14:45
Affects Version/s: 3.1.0

When I tried to verify that the resource-managers yarn module passed all UTs in Scala 2.13 , I found that there are a classpath dependent badcase blocking the test because there are more than one `servlet-api` dependency in spark now:
 * one is `javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml,
 * the other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test`  cascaded by `org.glassfish.jersey.test-framework.providers`, we can use `mvn dependency:tree` to check it .

So when we execute resource-managers yarn module test use

mvn clean test -pl resource-managers/yarn -Pyarn 

or 

mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13

 

if the position of `javax.servlet-api` in the  in classpath is before `jakarta.servlet-api`, there are some cases failed in YarnClusterSuite, YarnShuffleIntegrationSuite  and YarnShuffleAuthSuite.

 

The failed reason as follow:

 
{code:java}
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Exception in thread "main" java.lang.ExceptionInInitializerError
...
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Caused by: java.lang.SecurityException: class "javax.servlet.http.HttpSessionIdListener"'s signer information does not match signer information of other classes in the same package
...{code}