av9ash/GitBugs huggingface dataset:
Bug: Spark marks intentionally killed speculative tasks as pending leads to holding idle executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/20 19:32
Affects Version/s: 2.3.0

*TL;DR*
 When speculative tasks fail/get killed, they are still considered as pending and count towards the calculation of number of needed executors.
h3. Symptom

In one of our production job (where it's running 4 tasks per executor), we found that it was holding 6 executors at the end with only 2 tasks running (1 speculative). With more logging enabled, we found the job printed:
{code:java}
pendingTasks is 0 pendingSpeculativeTasks is 17 totalRunningTasks is 2
{code}
 while the job only had 1 speculative task running and 16 speculative tasks intentionally killed because of corresponding original tasks had finished.

An easy repro of the issue (`--conf spark.speculation=true --conf spark.executor.cores=4 --conf spark.dynamicAllocation.maxExecutors=1000` in cluster mode):
{code:java}
val n = 4000
val someRDD = sc.parallelize(1 to n, n)
someRDD.mapPartitionsWithIndex( (index: Int, it: Iterator[Int]) => {
if (index < 300 && index >= 150) {
    Thread.sleep(index * 1000) // Fake running tasks
} else if (index == 300) {
    Thread.sleep(1000 * 1000) // Fake long running tasks
}
it.toList.map(x => index + ", " + x).iterator
}).collect
{code}
You will see when running the last task, we would be hold 38 executors (see attachment), which is exactly (152 + 3) / 4 = 38.
h3. The Bug

Upon examining the code of _pendingSpeculativeTasks_: 
{code:java}
stageAttemptToNumSpeculativeTasks.map { case (stageAttempt, numTasks) =>
  numTasks - stageAttemptToSpeculativeTaskIndices.get(stageAttempt).map(_.size).getOrElse(0)
}.sum
{code}
where _stageAttemptToNumSpeculativeTasks(stageAttempt)_ is incremented on _onSpeculativeTaskSubmitted_, but never decremented.  _stageAttemptToNumSpeculativeTasks -= stageAttempt_ is performed on stage completion. *This means Spark is marking ended speculative tasks as pending, which leads to Spark to hold more executors that it actually needs!*

I will have a PR ready to fix this issue, along with SPARK-28403 too
Bug: Reliable single-stage accumulators
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Jan/20 19:22
Affects Version/s: 3.1.0

This proposes a pragmatic improvement to allow for reliable single-stage accumulators. Under the assumption that a given stage / partition / rdd produces identical results, non-deterministic code produces identical accumulator increments on success. Rerunning partitions for any reason should always produce the same increments per partition on success.

With this pragmatic approach, increments from individual partitions / tasks are only merged into the accumulator on driver side for the first time per partition. This is useful for accumulators registered with {{countFailedValues == false}}. Hence, the accumulator aggregates all successful partitions only once.

The implementations require extra memory that scales with the number of partitions.
Bug: Dynamic Pruning thread propagates the localProperties to task
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Jan/20 13:04
Affects Version/s: 3.0.0

Local properties set via sparkContext are not available as TaskContext properties when executing parallel jobs and threadpools have idle threads

Explanation:
When executing parallel jobs via SubqueryBroadcastExec, the {{relationFuture}} is evaluated via a separate thread. The threads inherit the {{localProperties}} from sparkContext as they are the child threads.
These threads are controlled via the executionContext (thread pools). Each Thread pool has a default {{keepAliveSeconds}} of 60 seconds for idle threads.
Scenarios where the thread pool has threads which are idle and reused for a subsequent new query, the thread local properties will not be inherited from spark context (thread properties are inherited only on thread creation) hence end up having old or no properties set. This will cause taskset properties to be missing when properties are transferred by child thread
Bug: Collapse back to back aggregations into a single aggregate to reduce the number of shuffles
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jan/20 22:39
Affects Version/s: 3.1.0

Queries containing nested aggregate operations can in some cases be computable with a single phase of aggregation. This Jira seeks to introduce a new optimizer rule to identify some of those cases and rewrite plans to avoid needlessly re-shuffling and generating the aggregation hash table data twice.

Some examples of collapsible aggregates:
{code:java}
SELECT sum(sumAgg) as a, year from (
      select sum(1) as sumAgg, course, year FROM courseSales GROUP BY course, year
) group by year

// can be collapsed to
SELECT sum(1) as `a`, year from courseSales group by year
{code}
{code}
SELECT sum(agg), min(a), b from (
     select sum(1) as agg, a, b FROM testData2 GROUP BY a, b
     ) group by b

// can be collapsed to
SELECT sum(1) as `sum(agg)`, min(a) as `min(a)`, b from testData2 group by b
{code}
Bug: statistic estimation of FileScan should take partitionFilters and partition number into account
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jan/20 06:39
Affects Version/s: 3.1.0

Currently, FileScan.estimateStatistics does not take partitionFilters and partition number into account, which may lead to bigger sizeInBytes. It should be reasonable to change it to involve partitionFilters and partition number when estimating the statistics.
Bug: Support SHOW TABLES EXTENDED
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jan/20 07:58
Affects Version/s: 3.1.0

{{Intention is to support show tables with a additional column 'type' where type can be MANAGED,EXTERNAL,VIEW using which user can query only tables of required types, like listing only views or only external tables (using a 'where' clause over 'type' column).}}

{{Usecase example:}}
{{Currently its not possible to list all the VIEWS, but other technologies like hive support it using 'SHOW VIEWS', mysql supports it using a more complex query 'SHOW FULL TABLES WHERE table_type = 'VIEW';'}}

Decide to take mysql approach as it provides more flexibility for querying.
Bug: Add config item for limiting partition number when calculating statistics through File System
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jan/20 05:46
Affects Version/s: 3.1.0

Currently, when spark need to calculate the statistics (eg. sizeInBytes) of table partition through file system (eg. HDFS), it does not consider the number of partitions. Then if the the number of partitions is huge, it will cost much time to calculate the statistics which may be not be that useful.

It should be reasonable to add a config item to control the limit of partition number allowable to calculate statistics through file system.
Bug: Partition columns should not be involved when calculating sizeInBytes of Project logical plan
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Jan/20 09:14
Affects Version/s: 3.1.0

When getting the statistics of a Project logical plan, if CBO not enabled, Spark will call SizeInBytesOnlyStatsPlanVisitor.visitUnaryNode to calculate the size in bytes, which will compute the ratio of the row size of the project plan and its child plan.

And the row size is computed based on the output attributes (columns). Currently, SizeInBytesOnlyStatsPlanVisitor.visitUnaryNode involve partition columns of hive table as well, which is not reasonable, because partition columns actually does not account for sizeInBytes.

This may make the sizeInBytes not accurate.
Bug: reduce degree recomputation in StronglyConnectedComponents
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jan/20 09:18
Affects Version/s: 3.1.0

 

So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true.

I have done a small code proposal, because there is a problem when the pregel executions have done,  the degree no need to be recomputed.

 

for example，the Email-EuAll  data set：[http://snap.stanford.edu/data/email-EuAll.html]

do-while loop execute 10 times，and the reduce logic happend 2 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation.

 

I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]

 

I hope you can consider this small code proposal.

Thank you very much,

Best regards,

xs-li
Bug: Improve the memory usage in StronglyConnectedComponents
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jan/20 16:22
Affects Version/s: 3.1.0

This is very consume memory when It use StronglyConnectedComponents(see figure1.png). Because there is no mark the Graph/RDD as non-persistent in the iterative process timely. And it is maybe lead to fail in the big graph.

In order to improve the memory usage, it is verty important to mark the Graph/RDD as non-persistent timely. In the current code, only make the Graph/RDD as non-persistent for 'sccGraph' but not for 'sccWorkGraph' in degree's step and pregel's step.

I have done a optimized code proposal(see my fork:[https://github.com/jmzhoulab/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala])

The storage after optimization see figure2.png
Bug: On Federal Information Processing Standard (FIPS) enabled cluster, Spark Workers are not able to connect to Remote Master.
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 09/Jan/20 06:52
Affects Version/s: 2.3.3

On _*Federal Information Processing Standard*_ (FIPS) enabled clusters, If we configured *spark.network.crypto.enabled true* , Spark Workers are not able to create Spark Context because of communication between Spark Worker and Spark Master is failing.

Default Algorithm ( *_spark.network.crypto.keyFactoryAlgorithm_* ) is set to *_PBKDF2WithHmacSHA1_* and that is one of the Non Approved Cryptographic Algorithm. We had tried so many values from FIPS Approved Cryptographic Algorithm but those values are also not working.

*Error logs :*

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
*fips.c(145): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE*
JVMDUMP039I Processing dump event "abort", detail "" at 2020/01/09 06:41:50 - please wait.
JVMDUMP032I JVM requested System dump using '<SPARK_HOME>/bin/core.20200109.064150.283.0001.dmp' in response to an event
JVMDUMP030W Cannot write dump to file<SPARK_HOME>/bin/core.20200109.064150.283.0001.dmp: Permission denied
JVMDUMP012E Error in System dump: The core file created by child process with pid = 375 was not found. Expected to find core file with name "/var/cores/core-netty-rpc-conne-sig11-user1000320999-group0-pid375-time*"
JVMDUMP030W Cannot write dump to file <SPARK_HOME>/bin/javacore.20200109.064150.283.0002.txt: Permission denied
JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20200109.064150.283.0002.txt' in response to an event
JVMDUMP010I Java dump written to /tmp/javacore.20200109.064150.283.0002.txt
JVMDUMP032I JVM requested Snap dump using '<SPARK_HOME>/bin/Snap.20200109.064150.283.0003.trc' in response to an event
JVMDUMP030W Cannot write dump to file <SPARK_HOME>/bin/Snap.20200109.064150.283.0003.trc: Permission denied
JVMDUMP010I Snap dump written to /tmp/Snap.20200109.064150.283.0003.trc
JVMDUMP030W Cannot write dump to file <SPARK_HOME>/bin/jitdump.20200109.064150.283.0004.dmp: Permission denied
JVMDUMP007I JVM Requesting JIT dump using '/tmp/jitdump.20200109.064150.283.0004.dmp'
JVMDUMP010I JIT dump written to /tmp/jitdump.20200109.064150.283.0004.dmp
JVMDUMP013I Processed dump event "abort", detail "".
Bug: How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/20 03:07
Affects Version/s: 3.0.0

Trying to read data from a secured Kafka cluster using spark structured
 streaming. Also, using the below library to read the data -
 +*"spark-sql-kafka-0-10_2.12":"3.0.0-preview"*+ since it has the feature to
 specify our custom group id (instead of spark setting its own custom group
 id)

+*Dependency used in code:*+

        <groupId>org.apache.spark</groupId>
         <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
         <version>3.0.0-preview</version>

 

+*Logs:*+

Getting the below error - even after specifying the required JAAS
 configuration in spark options.

Caused by: java.lang.IllegalArgumentException: requirement failed:
 *Delegation token must exist for this connector*. at
 scala.Predef$.require(Predef.scala:281) at

org.apache.spark.kafka010.KafkaTokenUtil$.isConnectorUsingCurrentToken(KafkaTokenUtil.scala:299)
 at
 org.apache.spark.sql.kafka010.KafkaDataConsumer.getOrRetrieveConsumer(KafkaDataConsumer.scala:533)
 at
 org.apache.spark.sql.kafka010.KafkaDataConsumer.$anonfun$get$1(KafkaDataConsumer.scala:275)

 

+*Spark configuration used to read from Kafka:*+

val kafkaDF = sparkSession.readStream
 .format("kafka")
 .option("kafka.bootstrap.servers", bootStrapServer)
 .option("subscribe", kafkaTopic )
 
//Setting JAAS Configuration
.option("kafka.sasl.jaas.config", KAFKA_JAAS_SASL)
 .option("kafka.sasl.mechanism", "PLAIN")
 .option("kafka.security.protocol", "SASL_SSL")

// Setting custom consumer group id
.option("kafka.group.id", "test_cg")
 .load()

 

Following document specifies that we can disable the feature of obtaining
 delegation token -
 [https://spark.apache.org/docs/3.0.0-preview/structured-streaming-kafka-integration.html]


Tried setting this property *spark.security.credentials.kafka.enabled to*
 *false in spark config,* but it is still failing with the same error.
Bug: Fix the NoSuchElementException exception when enable AQE with InSubquery use case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/20 03:19
Affects Version/s: 3.0.0

After merged [https://github.com/apache/spark/pull/25854], we also need to handle the Insubquery case when build SubqueryMap in InsertAdaptiveSparkPlan rule. Otherwise we will  get the NoSuchElementException exception.
Bug: reset the metrics info of AdaptiveSparkPlanExec plan when enable aqe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/20 08:24
Affects Version/s: 3.0.0

Working on [https://github.com/apache/spark/pull/26813]. With on AQE, the metric info of AdaptiveSparkPlanExec does not reset when running the test DataFrameCallbackSuite#get numRows metrics by callback.
Bug: Disable OptimizeSkewJoin rule if introducing additional shuffle.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/20 01:47
Affects Version/s: 3.0.0

The OptimizeSkewedJoin will break the outputPartitioning of origin SMJ. And it may introduce additional shuffle after apply the OptimizeSkewedJoin. This PR will disable "OptimizeSkewedJoin" rule if introducing additional shuffle.
Bug: "Managed memory leak detected" even with no calls to take() or limit()
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/20 23:25
Affects Version/s: 2.3.2

Our Spark code is causing a "Managed memory leak detected" warning to appear, even though we are not calling take() or limit().

According to SPARK-14168 https://issues.apache.org/jira/browse/SPARK-14168 managed memory leaks should only be caused by not reading an iterator to completion, i.e. take() or limit()

Our exact warning text is: "2020-01-06 14:54:59 WARN Executor:66 - Managed memory leak detected; size = 2097152 bytes, TID = 118"
 The size of the managed memory leak is always 2MB.

I have created a minimal test program that reproduces the warning: 
{code:java}
import pyspark.sql
import pyspark.sql.functions as fx


def main():
    builder = pyspark.sql.SparkSession.builder
    builder = builder.appName("spark-jira")
    spark = builder.getOrCreate()

    reader = spark.read
    reader = reader.format("csv")
    reader = reader.option("inferSchema", "true")
    reader = reader.option("header", "true")

    table_c = reader.load("c.csv")
    table_a = reader.load("a.csv")
    table_b = reader.load("b.csv")

    primary_filter = fx.col("some_code").isNull()

    new_primary_data = table_a.filter(primary_filter)

    new_ids = new_primary_data.select("some_id")

    new_data = table_b.join(new_ids, "some_id")

    new_data = new_data.select("some_id")
    result = table_c.join(new_data, "some_id", "left")

    result.repartition(1).write.json("results.json", mode="overwrite")

    spark.stop()


if __name__ == "__main__":
    main()
{code}
Our code isn't anything out of the ordinary, just some filters, selects and joins.

The input data is made up of 3 CSV files. The input data files are quite large, roughly 2.6GB in total uncompressed. I attempted to reduce the number of rows in the CSV input files but this caused the warning to no longer appear. After compressing the files I was able to attach them below.
Bug: Duplicates cached RDD when create or replace an existing view
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/20 02:29
Affects Version/s: 2.0.2

We can reproduce by below commands:
{code}
beeline> create or replace temporary view temp1 as select 1
beeline> cache table temp1
beeline> create or replace temporary view temp1 as select 1, 2
beeline> cache table temp1
{code}

The cached RDD for plan "select 1" stays in memory forever until the session close. This cached data cannot be used since the view temp1 has been replaced by another plan. It's a memory leak.

assert(spark.sharedState.cacheManager.lookupCachedData(sql("select 1, 2")).isDefined)
assert(spark.sharedState.cacheManager.lookupCachedData(sql("select 1")).isDefined)
Bug: Impl Extremely Randomized Trees
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 17/Jan/20 08:09
Affects Version/s: 3.0.0

1, Extremely Randomized Trees or ExtraTrees is widely used and impled in Scikit-Learn and OpenCV;

2, ExtraTrees is quite similar to RandomForest, and the main difference lie in that,on each leaf, candidate splits (only one split in Scikit-Learn's impl) are drawn at random for each feature and the best of these randomly-chosen splits is selected.

Based on current impl of ensenble trees, it can be impled.
Bug: setDelegateCatalog should be called if catalog has implemented CatalogExtension
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/20 02:31
Affects Version/s: 3.1.0

CatalogManager.catalog calls Catalogs.load to load a catalog if it is not 'spark_catalog' . If the catalog has implemented CatalogExtension, setDelegateCatalog is not called when the catalog is loaded, which is not like that we have done for v2SessionCatalog, and that makes a confusion for customized session catalog, like iceberg SparkSessionCatalog.
Bug: Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests
Status: Open | Priority: Major | Resolution: 
Created: 07/Jan/20 21:20
Affects Version/s: 3.1.0

Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests

Currently in the design ExecutorResourceRequests and TaskREsourceRequests are mutable and users can update as they want. It would make sense to add api's to remove certain resource requirements from them. This would allow a user to create one ExecutorResourceRequests object and then if they want to just add/remove something from it they easily could without having to recreate all the requests in that.
Bug: Regression in the ORC benchmark
Status: Open | Priority: Major | Resolution: 
Created: 18/Jan/20 17:47
Affects Version/s: 3.0.0

New benchmark results generated in the PR [https://github.com/apache/spark/pull/27078] show regression ~3 times.

Before:
{code}
Hive built-in ORC                                   520            531           8          2.0         495.8       0.6X
{code}
https://github.com/apache/spark/pull/27078/files#diff-42fe5f1ef10d8f9f274fc89b2c8d140dL138
After:
{code}
Hive built-in ORC                                  1761           1792          43          0.6        1679.3       0.1X
{code}
https://github.com/apache/spark/pull/27078/files#diff-42fe5f1ef10d8f9f274fc89b2c8d140dR138
Bug: Constant propagation nullability issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/20 15:46
Affects Version/s: 2.3.0

There is a bug in constant propagation due to null handling:

SELECT * FROM t WHERE NOT(c = 1 AND c + 1 = 1) returns those rows where c is null due to 1 + 1 = 1 propagation, but it shouldn't.
Bug: Integrate event log compactor into Spark History Server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/20 07:50
Affects Version/s: 3.0.0

This issue is to track the effort on compacting old event logs (and cleaning up after compaction) without breaking guaranteeing of compatibility.

This issue depends on SPARK-29779 and SPARK-30479, and focuses on integrating event log compactor into Spark History Server and enable configurations.
Bug: can't resolve qualified column name with v2 tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 03:42
Affects Version/s: 3.0.0

When running queries with qualified columns like `SELECT t.a FROM t`, it fails to resolve for v2 tables.

v1 table is fine as we always wrap the v1 relation with a `SubqueryAlias`. We should do the same for v2 tables.
Bug: Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/20 17:08
Affects Version/s: 2.3.4

Local properties set via sparkContext are not available as TaskContext properties when executing  jobs and threadpools have idle threads which are reused

Explanation:
When SubqueryExec, the {{relationFuture}} is evaluated via a separate thread. The threads inherit the {{localProperties}} from sparkContext as they are the child threads.
These threads are controlled via the executionContext (thread pools). Each Thread pool has a default {{keepAliveSeconds}} of 60 seconds for idle threads.
Scenarios where the thread pool has threads which are idle and reused for a subsequent new query, the thread local properties will not be inherited from spark context (thread properties are inherited only on thread creation) hence end up having old or no properties set. This will cause taskset properties to be missing when properties are transferred by child thread via {{sparkContext.runJob/submitJob}}
Bug: normalize the column name in AlterTable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 04:24
Affects Version/s: 3.0.0

Because of case insensitive resolution, the column name in AlterTable may match the table schema but not exactly the same. To ease DS v2 implementations, Spark should normalize the column name before passing them to v2 catalogs, so that users don't need to care about the case sensitive config.
Bug: support hive style REPLACE COLUMN syntax
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 03:49
Affects Version/s: 3.1.0

We already support the hive style CHANGE COLUMN syntax, I think it's better to also support hive style REPLACE COLUMN syntax. Please refer to the doc: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL