av9ash/GitBugs huggingface dataset:
Bug: State is still needed in the event of executor failure
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/May/21 14:17
Affects Version/s: 3.1.1

[Graceful Decommission of Executors|https://spark.apache.org/docs/3.1.1/job-scheduling.html#graceful-decommission-of-executors] section states that:
{quote}a Spark executor exits either on failure or when the associated application has also exited. In both scenarios, all state associated with the executor is no longer needed and can be safely discarded.
{quote}
However, in the case of a flaky app with tasks causing occasional executor OOM, state _is_ needed to prevent triggering of the stage failure mechanism to regenerate missing blocks. Hence, the Shuffle Service is valuable in this scenario, not only in the dynamic resource allocation scenario.
Bug: Minor cleanup to replace BaseJoinExec with ShuffledJoin in CoalesceBucketsInJoin
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 09/May/21 00:00
Affects Version/s: 3.2.0

As title. We should use a more restrictive interface `ShuffledJoin` other than `BaseJoinExec` in `CoalesceBucketsInJoin`, as the rule only applies to sort merge join and shuffled hash join.
Bug: Data corruption in some cached compressed boolean columns
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 20/Aug/20 21:24
Affects Version/s: 2.3.4

I found that when sorting some boolean data into the cache that the results can change when the data is read back out.

It needs to be a non-trivial amount of data, and it is highly dependent on the order of the data.  If I disable compression in the cache the issue goes away.  I was able to make this happen in 3.0.0.  I am going to try and reproduce it in other versions too.

I'll attach the parquet file with boolean data in an order that causes this to happen. As you can see after the data is cached a single null values switches over to be false.

{code}
scala> val bad_order = spark.read.parquet("./bad_order.snappy.parquet")
bad_order: org.apache.spark.sql.DataFrame = [b: boolean]                        

scala> bad_order.groupBy("b").count.show
+-----+-----+
|    b|count|
+-----+-----+
| null| 7153|
| true|54334|
|false|54021|
+-----+-----+


scala> bad_order.cache()
res1: bad_order.type = [b: boolean]

scala> bad_order.groupBy("b").count.show
+-----+-----+
|    b|count|
+-----+-----+
| null| 7152|
| true|54334|
|false|54022|
+-----+-----+


scala> 

{code}
Bug: When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 17/Aug/20 06:34
Affects Version/s: 2.1.3

When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

eg:lit() function with cache() function.
 -----------------------------------
{code:java}
from pyspark.sql import Row
from pyspark.sql import functions as F

df_1 = spark.createDataFrame(Row(**x) for x in [{'col1': 'b'}]).withColumn("col2", F.lit(str(2)))
df_2 = spark.createDataFrame(Row(**x) for x in [{'col1': 'a', 'col3': 8}]).withColumn("col2", F.lit(str(1)))
df_3 = spark.createDataFrame(Row(**x) for x in [{'col1': 'b', 'col3': 9}]).withColumn("col2", F.lit(str(2)))
df_23 = df_2.union(df_3)
df_4 = spark.createDataFrame(Row(**x) for x in [{'col3': 9}]).withColumn("col2", F.lit(str(2)))

sel_col3 = df_23.select('col3', 'col2')
df_4 = df_4.join(sel_col3, on=['col3', 'col2'], how = "inner")
df_23_a = df_23.join(df_1, on=["col1", 'col2'], how="inner").cache() 
finaldf = df_23_a.join(df_4, on=['col2', 'col3'], how='left').filter(F.col('col3') == 9)
finaldf.show()
finaldf.select('col2').show() #Wrong result
{code}
 

Output
 -----------
{code:java}
>>> finaldf.show()
+----+----+----+
|col2|col3|col1|
+----+----+----+
| 2| 9| b|
+----+----+----+
>>> finaldf.select('col2').show() #Wrong result, instead of 2, got 1
+----+
|col2|
+----+
| 1|
+----+
+----+{code}
 lit() function without cache() function.
{code:java}
from pyspark.sql import Row
from pyspark.sql import functions as F

df_1 = spark.createDataFrame(Row(**x) for x in [{'col1': 'b'}]).withColumn("col2", F.lit(str(2)))
df_2 = spark.createDataFrame(Row(**x) for x in [{'col1': 'a', 'col3': 8}]).withColumn("col2", F.lit(str(1)))
df_3 = spark.createDataFrame(Row(**x) for x in [{'col1': 'b', 'col3': 9}]).withColumn("col2", F.lit(str(2)))
df_23 = df_2.union(df_3)
df_4 = spark.createDataFrame(Row(**x) for x in [{'col3': 9}]).withColumn("col2", F.lit(str(2)))

sel_col3 = df_23.select('col3', 'col2')
df_4 = df_4.join(sel_col3, on=['col3', 'col2'], how = "inner")
df_23_a = df_23.join(df_1, on=["col1", 'col2'], how="inner")
finaldf = df_23_a.join(df_4, on=['col2', 'col3'], how='left').filter(F.col('col3') == 9)
finaldf.show() 
finaldf.select('col2').show() #Correct result
{code}
 

Output
{code:java}
----------
>>> finaldf.show()
+----+----+----+
|col2|col3|col1|
+----+----+----+
| 2| 9| b|
+----+----+----+
>>> finaldf.select('col2').show() #Correct result, when df_23_a is not cached 
+----+
|col2|
+----+
| 2|
+----+
{code}
Bug: Partition length number overflow in `PartitionWriterStream`
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Aug/20 06:48
Affects Version/s: 3.0.0

A Spark user reported  `FetchFailedException: Stream is corrupted` error when they upgraded their workload to 3.0. The issue happens when the shuffle output data size from a single task is very large (~5GB). The issue is introduced by https://github.com/apache/spark/commit/abef84a868e9e15f346eea315bbab0ec8ac8e389 , the `PartitionWriterStream` defined the partition length to be an int value, while it should be a long value.
Bug: PySpark type hints support
Status: Resolved | Priority: Critical | Resolution: Done
Created: 21/Aug/20 09:00
Affects Version/s: 3.1.0

 https://github.com/zero323/pyspark-stubs demonstrates a lot of benefits to improve usability in PySpark by leveraging Python type hints.

By having the type hints in PySpark we can, for example:
- automatically document the input and output types
- leverage IDE for error detection and auto-completion
- have a cleaner definition and easier to understand.

This JIRA is an umbrella JIRA that targets to port https://github.com/zero323/pyspark-stubs and related items to smoothly run within PySpark.

It was also discussed in the dev mailing list:  http://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html
Bug: Add support for ESS in Spark sidecar
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Aug/20 22:21
Affects Version/s: 3.1.0

For Spark on Kubernetes there are multiple ways to achieve dynamic scaling. Unlike in YARN mode ESS on Spark on K8s as a sidecar would not help with dynamic scaling. Instead this helps with the situations where the executor JVM may be overloaded or crash. If we do this as a sidecar we can share the same mounts between the executor and the external shuffle storage.
Bug: Fix the trim logic in UTF8String.toInt/toLong did't handle Chinese characters correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/20 09:02
Affects Version/s: 3.0.0

The trim logic in Cast expression introduced in [https://github.com/apache/spark/pull/26622] will trim chinese characters unexpectly.

For example,  sql  select cast("1中文" as float) gives 1 instead of null
Bug: Filter by column alias in where clause
Status: Open | Priority: Major | Resolution: 
Created: 28/Aug/20 09:19
Affects Version/s: 3.1.0

{{group by}} and {{order by}} clause support it. but {{where}} does not support it:

{noformat}
spark-sql> select id + 1 as new_id from range(5) group by new_id order by new_id;
1
2
3
4
5
spark-sql> select id + 1 as new_id from range(5) where new_id > 2;
Error in query: cannot resolve '`new_id`' given input columns: [id]; line 1 pos 44;
'Project [('id + 1) AS new_id#5]
+- 'Filter ('new_id > 2)
   +- Range (0, 5, step=1, splits=None
spark-sql> select id + 1 as new_id, new_id + 1 as new_new_id from range(5);
Error in query: cannot resolve '`new_id`' given input columns: [id]; line 1 pos 25;
'Project [(id#12L + cast(1 as bigint)) AS new_id#10L, ('new_id + 1) AS new_new_id#11]
+- Range (0, 5, step=1, splits=None)
{noformat}

Teradata support this feature:
https://docs.teradata.com/reader/e79ET77~NzPDz~Ykinj44w/MKSYuTyx2UJWXzdHJf3~sQ
Bug: Add a AQEOptimizer for AdaptiveSparkPlanExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 12:55
Affects Version/s: 3.1.0

Currently, AdaptiveSparkPlanExec has implemented an anonymous RuleExecutor to apply the AQE specific rules on the plan. However, the anonymous class usually could be inconvenient to maintain and extend for the long term.
Bug: Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 27/Aug/20 12:17
Affects Version/s: 3.0.0

This is a proposal to add the following function to NewHadoopRDD:
{code:java}
def mapPartitionsWithInputSplitAndIndex[U: ClassTag](
    f: (InputSplit, Int, Iterator[(K, V)]) => Iterator[U],
    preservesPartitioning: Boolean = false
): RDD[U]
{code}
This new function would provide the input split with its partition index.

Having the partition index may be useful when trying to determine the mapping between part-XXXXX files and input splits.
Bug: Pyspark/cloudpickle.py - no module named 'wfdb'
Status: Open | Priority: Major | Resolution: 
Created: 20/Aug/20 21:32
Affects Version/s: 3.0.0

Running Spark in a Databricks notebook.

 

Ran into this issue when executing a cell:



(1) Spark Jobs

{code}
SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 4 times, most recent failure: Lost task 0.3 in stage 17.0 (TID 68, 10.139.64.5, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 177, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 466, in loads return pickle.loads(obj, encoding=encoding) File "/databricks/spark/python/pyspark/cloudpickle.py", line 1110, in subimport __import__(name) ModuleNotFoundError: No module named 'wfdb' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/databricks/spark/python/pyspark/worker.py", line 644, in main func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type) File "/databricks/spark/python/pyspark/worker.py", line 463, in read_udfs udfs.append(read_single_udf(pickleSer, infile, eval_type, runner_conf, udf_index=i)) File "/databricks/spark/python/pyspark/worker.py", line 254, in read_single_udf f, return_type = read_command(pickleSer, infile) File "/databricks/spark/python/pyspark/worker.py", line 74, in read_command command = serializer._read_with_length(file) File "/databricks/spark/python/pyspark/serializers.py", line 180, in _read_with_length raise SerializationError("Caused by " + traceback.format_exc()) pyspark.serializers.SerializationError: Caused by Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 177, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 466, in loads return pickle.loads(obj, encoding=encoding) File "/databricks/spark/python/pyspark/cloudpickle.py", line 1110, in subimport __import__(name) ModuleNotFoundError: No module named 'wfdb'
{code}
Bug: Add 'build' and 'project/build.properties' into cache key of SBT and Zinc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/20 01:51
Affects Version/s: 3.1.0

The version information is included in {{build}} and {{project/build.properties}}. We should better don't cache in this case. Looks like it can cause a build failure given https://github.com/apache/spark/pull/29286#issuecomment-679368436
Bug: Port pyspark-stubs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 11:21
Affects Version/s: 3.1.0

Port https://github.com/zero323/pyspark-stubs into PySpark. This was being discussed in dev mailing list. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html
Bug: Make SpecificInternalRow constructors faster by using while loops instead of maps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 23:23
Affects Version/s: 3.0.0

Two constructors in SpecificInternalRow can be made faster by using while loops instead of maps. This was originally noticed while working on SPARK-32532 and SPARK-32533 and will have impacts on performance of reading ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 for nested and array'd structs.
Bug: Set the downloaded artifact location to explicitly glob test reports
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Aug/20 05:52
Affects Version/s: 3.1.0

After SPARK-32357, now GitHub Actions reports the test results. However, seems some tests are not reported correctly. I attached an image.

Looks when you run tests via using tags to exclude/include tests, it still generates empty JUnit test reports. So, when tests are split by using the tags, it seems possible to overwrite the previous proper report.

We can avoid this problem by explicitly setting the downloaded artifact directory.
Bug: Remove the fork of action-surefire-report in test_report.yml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 11:46
Affects Version/s: 3.1.0

It was forked to have a custom fix https://github.com/HyukjinKwon/action-surefire-report/commit/c96094cc35061fcf154a7cb46807f2f3e2339476
in order to add the support of custom target commit SHA. It should be contributed back to the original plugin and avoid using the fork.
Bug: Remove the fork of action-download-artifact in test_report.yml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 11:47
Affects Version/s: 3.1.0

https://github.com/HyukjinKwon/action-download-artifact/commit/750b71af351aba467757d7be6924199bb08db4ed
in order to add the support to download all artifacts. It should be contributed back to the original
plugin and avoid using the fork.
Alternatively, we can use the official actions/download-artifact once they support to download artifacts
between different workloads, see also https://github.com/actions/download-artifact/issues/3
Bug: Add better api docs for stage level scheduling Resources
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/20 18:29
Affects Version/s: 3.1.0

A question came up when we added offheap memory to be able to set in a ResourceProfile executor resources.

[https://github.com/apache/spark/pull/28972/]

Based on that discussion we should add better api docs to explain what each one does. Perhaps point to the corresponding configuration .  Also specify the default config is used if not specified in the profiles for those that fallback.
Bug: SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing NULL values
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/20 10:37
Affects Version/s: 2.4.5

While writing to a target in SQL Server using Microsoft's SQL Server driver using dataframe.write API the target is storing NULL values for BIT columns.

 

Table definition

Azure SQL DB 

1)Create 2 tables with column type as bit

2)Insert some record into 1 table

Create a SPARK job 

1)Create a Dataframe using spark.read with the following query

select <bit_column> from <source_table>

2)Write the dataframe to a target table with bit type  as column.

 

Observation : Bit type is getting converted to NULL at the target
Bug: Insert wrong dates to Hive tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/20 20:18
Affects Version/s: 3.0.1

Run *spark-sql -S*:
{code:sql}
spark-sql> CREATE TABLE table1 (d date);
20/08/11 23:17:20 WARN HiveMetaStore: Location: file:/Users/maximgekk/proj/insert-date-into-hive-table/spark-warehouse/table1 specified for non-external table:table1
spark-sql> INSERT INTO table1 VALUES (date '2020-08-11');
spark-sql> SELECT * FROM table1;
1970-01-01
{code}
The expected result of the last statement is *2020-08-11* but got *1970-01-01*.
Bug: Spark SQL Infer Schema Performance
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/20 13:54
Affects Version/s: 2.4.6

When infer schema is enabled, it tries to list all the files in the table, however only one of the file is used to read schema informaiton. The performance is impacted due to list all the files in the table when the number of partitions is larger.

 

See the code in "[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala#L88|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala#88]", all the files in the table are input, however only one of the file's schema is used to infer schema.
Bug: Exclude _images and _sources that are generated by Sphinx in Jekyll build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/20 04:51
Affects Version/s: 3.1.0

For _images directory, after SPARK-31851, now we added some images to use within the pages built by Sphinx. It copies and images into _images directory. Later when Jeykill builds, the underscore directories are ignored by default which ends up with missing image in the main doc. I attached the image in the JIRA.

For _sources directory, see https://github.com/sphinx-contrib/sphinx-pretty-searchresults#source-links.
Bug: PySpark Structured Streaming Testing Support
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Aug/20 23:37
Affects Version/s: 2.4.4

Hello,

I have investigated a lot but couldn't get any help or resource on how to +{color:#172b4d}*test my pyspark Structured Streaming pipeline job*{color}+ (ingesting from Kafka topics to S3) and how to build  Continuous Integration (CI)/Continuous Deployment (CD).

1. Is it possible to test (unit test, integration test) pyspark structured stream?

2. How to build  Continuous Integration (CI)/Continuous Deployment (CD)?
Bug: Pyspark drop duplicate columns
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Aug/20 04:11
Affects Version/s: 3.0.0


Hi All,

This is one suggestion can we have a feature in pyspark to remove duplicate columns? 

I have come up with small code for that 

{code:python}
def drop_duplicate_columns(_rdd_df):
    column_names = _rdd_df.columns
    duplicate_columns = set([x for x in column_names if column_names.count(x) > 1])
    _rdd_df = _rdd_df.drop(*duplicate_columns)
    return _rdd_df

{code}

Your suggestions are appreciatd and can work on this PR, this would be my first contribution(PR) to Pyspark if you guys agree with it
Bug: Run all the tests at once, instead of having separate entrypoints.
Status: Open | Priority: Major | Resolution: 
Created: 08/Aug/20 07:44
Affects Version/s: 3.0.0

Started with this comment thread: https://github.com/apache/spark/pull/29121/files#r456683561

Each file is invoked separately and has a separate entry point: [https://github.com/apache/spark/blob/master/python/pyspark/ml/tests/test_wrapper.py#L120]

We would replace [https://github.com/apache/spark/blob/master/dev/run-tests.py#L470] this function call to the subprocess with something that would invoke the python tests.
Bug: deleted not existing hdfs locations when use spark sql to execute "insert overwrite" statement to dynamic partition
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Aug/20 06:40
Affects Version/s: 2.3.2

when execute insert overwrite table statement to dynamic partition :

 
{code:java}
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nostrict;
insert overwrite table tmp.id_name2 partition(dt) select * from tmp.id_name where dt='2001';
{code}
output log:
{code:java}
20/08/05 14:38:05 ERROR Hive: Exception when loading partition with parameters  partPath=hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/.hive-staging_hive_2020-08-05_14-38-00_715_3629476922121193803-1/-ext-10000/dt=2001,  table=id_name2,  partSpec={dt=2001},  loadFileType=REPLACE_ALL,  listBucketingLevel=0,  isAcid=false,  resetStatistics=false
org.apache.hadoop.hive.ql.metadata.HiveException: Directory hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/dt=2001 could not be cleaned up.
    at org.apache.hadoop.hive.ql.metadata.Hive.deleteOldPathForReplace(Hive.java:4666)
    at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:4597)
    at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2132)
    at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2588)
    at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2579)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/dt=2001 does not exist.
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1053)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
    at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1113)
    at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1110)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1120)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
    at org.apache.hadoop.hive.ql.metadata.Hive.cleanUpOneDirectoryForReplace(Hive.java:4681)
    at org.apache.hadoop.hive.ql.metadata.Hive.deleteOldPathForReplace(Hive.java:4661)
    ... 8 more
Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: Exception when loading 1 in table id_name2 with loadPath=hdfs://nameservice/user/hive/warehouse/tmp.db/id_name2/.hive-staging_hive_2020-08-05_14-38-00_715_3629476922121193803-1/-ext-10000;
{code}
it seems that spark doesn't test if the partitions hdfs locations whether exists before delete it.

and Hive can successfully execute the same sql.
Bug: Deduplicate configuration set/unset in test_sparkSQL_arrow.R
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/20 04:21
Affects Version/s: 3.0.0

Currently, there are many set/unset duplicated in `test_sparkSQL_arrow.R` test cases. We can just set once in globally and deduplicate such try-catch logics.