av9ash/GitBugs huggingface dataset:
Bug: Writing out empty or nested empty schemas in Datasource should be configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/22 01:59
Affects Version/s: 2.4.0

In SPARK-23372, we introduced a backwards incompatible change that would remove support for writing out empty or nested empty schemas in file based datasources. This introduces backward incompatibility for users who have been using a schema that met the above condition since the datasource supported it. Except for Parquet and text, other file based sources support this behavior.

 

We should either :
 * Make it configurable to enable/disable writing out empty schemas
 * Enable the validation check only for sources that do not support it - Parquet / Text
Bug: Set upperbound for pandas-stubs in CI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/22 00:27
Affects Version/s: 3.3.0

e.g., ) https://github.com/apache/spark/runs/5729037000?check_suite_focus=true

{code}
starting mypy annotations test...
annotations failed mypy checks:
python/pyspark/pandas/ml.py:62: error: Incompatible types in assignment (expression has type "Index", variable has type "MultiIndex")  [assignment]
python/pyspark/pandas/frame.py:6133: error: unused "type: ignore" comment
python/pyspark/pandas/frame.py:6292: error: Incompatible types in assignment (expression has type "Index", variable has type "MultiIndex")  [assignment]
Found 3 errors in 2 files (checked 325 source files)
{code}


Seems like the new release of pandas-stubs has a breaking change.
Bug: AttributeError: module 'pyspark.pandas' has no attribute 'Timestamp'
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Mar/22 16:13
Affects Version/s: 3.3.0

I am trying to replace pandas library with pyspark.pandas. 

Tried something like below - 
{code:java}
List[pd.Timestamp] {code}
 But it does not work and instead thrown the below error
{code:java}
AttributeError: module 'pyspark.pandas' has no attribute 'Timestamp'{code}
Bug: AttributeError: module 'pyspark.pandas' has no attribute 'Timedelta'
Status: Open | Priority: Major | Resolution: 
Created: 23/Mar/22 17:27
Affects Version/s: 3.2.1

After replacing pandas library with pyspark.pandas, I tried to use the below function :  
{code:java}
pd.Timedelta(days=30) {code}
But received the below error :
{code:java}
AttributeError: module 'pyspark.pandas' has no attribute 'Timedelta' {code}
Bug: pyspark.pandas.config.OptionError: "No such option: 'mode.chained_assignment'
Status: Open | Priority: Major | Resolution: 
Created: 23/Mar/22 16:41
Affects Version/s: 3.3.0

I replaced import pandas as pd to import pyspark.pandas as pd in my code. 
{code:java}
pd.set_option("mode.chained_assignment", None) {code}
The above command was working with pandas but this option is not available in  pyspark.pandas . 
{code:java}
pyspark.pandas.config.OptionError: "No such option: 'mode.chained_assignment'. Available options are [display.max_rows, 
compute.max_rows, compute.shortcut_limit, compute.ops_on_diff_frames, compute.default_index_type, compute.ordered_head, 
plotting.max_rows, plotting.sample_ratio, plotting.backend]" {code}
Bug: TypeError: Datetime subtraction can only be applied to datetime series
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/22 21:01
Affects Version/s: 3.2.1

I am trying to replace pandas with pyspark.pandas library, when I tried this :

pdf is a pyspark.pandas dataframe
{code:java}
pdf["date_diff"] = (pdf["date1"] - pdf["date2"])/pdf.Timedelta(days=30){code}
I got the below error :
{code:java}
File "C:\Users\abc\Anaconda3\envs\test\lib\site-packages\pyspark\pandas\data_type_ops\datetime_ops.py", line 75, in sub
raise TypeError("Datetime subtraction can only be applied to datetime series.") {code}
Bug: Complete the copy method in subclasses of InternalRow, ArrayData, and MapData to safely copy their instances.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/22 23:15
Affects Version/s: 3.3.0

Some subclasses of {{InternalRow}}, {{ArrayData}}, and {{MapData}} missing support for {{StructType}}, {{ArrayType}}, and {{MapType}} in their {{copy}} method.
We should complete them to safely copy their instances.
Bug: List of supported pandas APIs for pandas API on Spark docs.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/22 02:59
Affects Version/s: 3.3.0

In Modin, they have a [list of supported pandas API|https://modin.readthedocs.io/en/stable/supported_apis/dataframe_supported.html].

It would be great if we have also supported pandas API list so that users can easily find which API is available or not in pandas API on Spark
Bug: AttributeError: module 'pyspark.pandas' has no attribute 'DateOffset'
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/22 12:51
Affects Version/s: 3.3.0

I am working on replacing Pandas library to pyspark.pandas Library in my python repo in VS Code. But pyspark.pandas module does not seem to have DateOffset() module similar to what pandas has.

I tried this :
{code:java}
import pyspark.pandas as pd 
clean_df[self._config.general["info_col"]] = (clean_df["cycle_dt"] - pd.DateOffset(months=cycle_info_gap)        ).fillna(clean_df[self._config.general["time_col"]])  {code}
It results in the below error :
{code:java}
Un unanticipated error occurred: module 'pyspark.pandas' has no attribute 'DateOffset'{code}
Is there any alternative for this in pyspark.pandas?
Bug: Column name contains back tick `
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/22 17:51
Affects Version/s: 3.0.1

Try to modify the data frame to ensure column names have no special characters.
{code:java}
df.columns.map { columnName =>
   df.col(surroundingBackTickedName(columnName)).as(normalizeName(columnName))
}
{code}
*surroundingBackTickedName()* will enclose column name with backticks.

However, col() kept having issue with column name contains back tick because {*}parseAttributeName{*}() only takes backticks that appear in pair. I am wondering if there is a workaround
{code:java}
org.apache.spark.sql.AnalysisException: Cannot resolve column name "`COLNAME`2`" among (COLID, COLNAME!4, COLNAME#6, COLNAME$7, COLNAME%8, COLNAME'25, COLNAME(11, COLNAME)12, COLNAME*10, COLNAME+16, COLNAME,26, COLNAME-13, COLNAME/30, COLNAME:24, COLNAME;23, COLNAME<27, COLNAME=15, COLNAME>29, COLNAME?31, COLNAME@5, COLNAME`2){code}
Bug: to_timestamp should allow numeric types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/22 06:30
Affects Version/s: 3.3.0

SPARK-38240 mistakenly disallowed numeric type at to_timestamp. We should allow it back:

{code}
spark.range(1).selectExpr("to_timestamp(id)").show()
{code}


*Before*

{code}
+-------------------+
|   to_timestamp(id)|
+-------------------+
|1970-01-01 09:00:00|
+-------------------+
{code}


*After*

{code}
+-----------------+
| to_timestamp(id)|
+-----------------+
|             null|
+-----------------+
{code}
Bug: Spark 3 can not read mixed format partitions
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Mar/22 20:17
Affects Version/s: 3.0.0

Spark 3.x reads partitions with table's input format, which fails when the partition has a different input format than the table.

This is a regression introduced by SPARK-26630. Before that fix, Spark will use Partition InputFormat when creating HadoopRDD. With that fix, Spark uses only Table InputFormat when creating HadoopRDD, causing failures

Reading mixed format partitions is an import scenario, especially for format migration. It is also well supported in query engines like Hive and Presto.
Bug: NullPointerException when selecting a generator in a Stream of aggregate expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/22 00:11
Affects Version/s: 3.1.3

Assume this dataframe:
{noformat}
val df = Seq(1, 2, 3).toDF("v")
{noformat}
This works:
{noformat}
df.select(Seq(explode(array(min($"v"), max($"v"))), sum($"v")): _*).collect
{noformat}
However, this doesn't:
{noformat}
df.select(Stream(explode(array(min($"v"), max($"v"))), sum($"v")): _*).collect
{noformat}
It throws this error:
{noformat}
java.lang.NullPointerException
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$.$anonfun$containsAggregates$1(Analyzer.scala:2516)
  at scala.collection.immutable.List.flatMap(List.scala:366)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$.containsAggregates(Analyzer.scala:2515)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$$anonfun$apply$31.applyOrElse(Analyzer.scala:2509)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$$anonfun$apply$31.applyOrElse(Analyzer.scala:2508)
{noformat}
The only difference between the two queries is that the first one uses {{Seq}} to specify the varargs, whereas the second one uses {{Stream}}.
Bug: Download link on spark 3.2.1 for hadoop 3.2 is wrong
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/22 23:10
Affects Version/s: 3.2.1

When downloading spark 3.2.1 pre-built for hadoop, the dropdown reads:

{quote}
Pre-built for Apache Hadoop *3.3 and later*
{quote}

However the filename link reads

{quote}
spark-3.2.1-bin-*hadoop3.2*.tgz
{quote}

When downloading, the contents actually have hadoop 3.3.1 dependencies indicating the filename is incorrect.

https://spark.apache.org/downloads.html
Bug: Overflow occurs when reading ANSI day time interval from CSV file
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Mar/22 06:59
Affects Version/s: 3.3.0

*Problem:*

Overflow occurs when reading the following positive intervals, the results become to negative

interval '106751992' day     => INTERVAL '-106751990' DAY

INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR

interval '153722867281' minute => INTERVAL '-153722867280' MINUTE

 

*Reproduce:*
{code:java}
// days overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
 scala> spark.read.csv(path).show(false)
 +------------------------+
 |_c0                     |
 +------------------------+
 |interval '106751992' day|
 +------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +-------------------------+
 |c1                       |
 +-------------------------+
 |INTERVAL '-106751990' DAY|
 +-------------------------+
  // hour overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
 scala> spark.read.csv(path).show(false)
 +----------------------------+
 |_c0                         |
 +----------------------------+
 |INTERVAL +'+2562047789' hour|
 +----------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +---------------------------+
 |c1                         |
 +---------------------------+
 |INTERVAL '-2562047787' HOUR|
 +---------------------------+
 // minute overflow
 scala> val schema = StructType(Seq(StructField("c1",
   DayTimeIntervalType(DayTimeIntervalType.MINUTE, DayTimeIntervalType.MINUTE))))
 scala> spark.read.csv(path).show(false)
 +------------------------------+
 |_c0                           |
 +------------------------------+
 |interval '153722867281' minute|
 +------------------------------+
 scala> spark.read.schema(schema).csv(path).show(false)
 +-------------------------------+
 |c1                             |
 +-------------------------------+
 |INTERVAL '-153722867280' MINUTE|
 +-------------------------------+
{code}
 

*others:*

Also check the negative value is read to positive.

 

others:

should check the negative also
Bug: GeneratorNestedColumnAliasing does not work correctly for some expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/22 01:33
Affects Version/s: 3.2.1

[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala#L226]

The code to collect ExtractValue expressions is wrong. We should do it in a bottom up way instead of only check 2 levels. It can cause incorrect result if the expression looks like ExtractValue(ExtractValue(some_other_expr)).

 

An example to trigger the bug is:

 

input: <col1: array<struct<a: int, b: struct<a: struct<a: int, b: int>, b: int>>>>

 

Project(ExtractValue(ExtractValue(CaseWhen([col.a == 1, col.b]), "a"), "a")

- Generate(Explode(col1))

 

We will try to incorrectly push down the whole expression into the input of the Explode, now the input of CaseWhen has array<...> as input so we will get wrong result.
Bug: Fix PySpark documentation generation (missing ipython_genutils)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 01:53
Affects Version/s: 3.2.1

{code}
Extension error:
Could not import extension nbsphinx (exception: No module named 'ipython_genutils')
make: *** [Makefile:35: html] Error 2
                    ------------------------------------------------
      Jekyll 4.2.1   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
/__w/spark/spark/docs/_plugins/copy_api_dirs.rb:130:in `<top (required)>': Python doc generation failed (RuntimeError)
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `require'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `block in require_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `require_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:89:in `block in require_plugin_files'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `require_plugin_files'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:21:in `conscientious_require'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/site.rb:131:in `setup'
{code}

https://github.com/apache/spark/runs/5504729423?check_suite_focus=true
Bug: Move tests for the grouping error classes to QueryCompilationErrorsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/22 17:49
Affects Version/s: 3.4.0

Move tests for the error classes GROUPING_COLUMN_MISMATCH and GROUPING_ID_COLUMN_MISMATCH from DataFrameAggregateSuite to QueryCompilationErrorsSuite.
Bug: spark.sql.catalyst.analysis
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Mar/22 02:20
Affects Version/s: 2.4.4

create view step_29134910810  as select * from step_29134878417  limit 20(String), java.sql.SQLException: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolved object, tree: *

I don't know what I'm doing wrong

[~hyukjin.kwon]
Bug: Fix R tests under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/22 14:23
Affects Version/s: 3.3.0

[https://github.com/gengliangwang/spark/runs/5461227887?check_suite_focus=true]

 
{quote}1. Error (test_sparkSQL.R:2064:3): SPARK-37108: expose make_date expression i
2022-03-08T10:06:54.9600113Z Error in `handleErrors(returnStatus, conn)`: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 661.0 failed 1 times, most recent failure: Lost task 0.0 in stage 661.0 (TID 570) (localhost executor driver): java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): 13. If necessary set spark.sql.ansi.enabled to false to bypass this error.
{quote}
Bug: Invalid call to exprId
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/22 08:07
Affects Version/s: 2.4.4

java.sql.SQLException: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to exprId on unresolved object, tree: 'c_column_2
java.lang.RuntimeException: java.sql.SQLException: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to exprId on unresolved object, tree: 'c_column_2
    at com.bonc.ds.xquery.explorev2.stephandler.utils.HiveJdbcExecutor.createTableWithSql(HiveJdbcExecutor.java:602)
    at com.bonc.ds.xquery.explorev2.stephandler.operator.AbstractOperator.execute(AbstractOperator.java:183)

I see that this issue is fixed by 2.4.0.
I have no idea, i use hadoop3.1.4 spark 2.4.4
Bug: Improve user experience when working on data frames created from CSV and JSON in PERMISSIVE mode.
Status: Open | Priority: Major | Resolution: 
Created: 03/Mar/22 07:22
Affects Version/s: 3.2.1

In our data processing pipeline, we first process the user supplied data and eliminate invalid/corrupt records. So we parse JSON and CSV files in PERMISSIVE mode where all the invalid records are captured in "_corrupt_record". We then apply predicates on "_corrupt_record" to eliminate the bad records before subjecting the good records further in the processing pipeline.

We encountered two issues.
1. The introduction of "predicate pushdown" for CSV, does not take into account this system generated "_corrupt_column" and tries to push this down to scan resulting in an exception as the column is not part of base schema. 
2. Applying predicates on "_corrupt_column" results in a AnalysisException like following.
{code:java}
val schema = new StructType()
  .add("id",IntegerType,true)
  .add("weight",IntegerType,true) // The weight field is defined wrongly. The actual data contains floating point numbers, while the schema specifies an integer.
  .add("price",IntegerType,true)
  .add("_corrupt_record", StringType, true) // The schema contains a special column _corrupt_record, which does not exist in the data. This column captures rows that did not parse correctly.

val csv_with_wrong_schema = spark.read.format("csv")
  .option("header", "true")
  .schema(schema)
  .load("/FileStore/tables/csv_corrupt_record.csv")

val badRows = csv_with_wrong_schema.filter($"_corrupt_record".isNotNull)
7
val numBadRows = badRows.count()
 Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column
(named _corrupt_record by default). For example:
spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count()
and spark.read.schema(schema).csv(file).select("_corrupt_record").show().
Instead, you can cache or save the parsed results and then send the same query.
For example, val df = spark.read.schema(schema).csv(file).cache() and then
df.filter($"_corrupt_record".isNotNull).count().

{code:java}

For (1), we have disabled predicate pushdown.
For (2), we currently cache the data frame before using it , however, its not convenient and we would like to see a better user experience.
Bug: upgrade jackson due to CVE-2020-36518
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/22 14:52
Affects Version/s: 3.2.1

* https://github.com/FasterXML/jackson-databind/issues/2816
* only jackson-databind has a 2.13.2.1 release
* other jackson jars should stay at 2.13.2
Bug: High GC and memory footprint after switch to ZSTD
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Mar/22 16:31
Affects Version/s: 3.1.2

Hi All,

We started to switch our Spark pipelines to read parquet with ZSTD compression. 
After the switch we see that memory footprint is much larger than previously with SNAPPY.

Additionally GC stats of the jobs are much higher comparing to SNAPPY with the same workload as previously. 

Is there any configurations that may be relevant to read path, that may help in such cases ?
Bug: Improve the test coverage for pyspark/*.py
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/22 05:51
Affects Version/s: 3.3.0

Currently, there are several Python scripts under pyspark/ directory. (e.g. rdd.py, util.py, serializers.py, ...)

We could improve the test coverage by adding the missing tests for these scripts.
Bug: Use a disk-based store to save more information in live UI to help debug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/22 03:21
Affects Version/s: 3.3.0

In Spark, the UI lacks troubleshooting abilities. For example:

* AQE plan changes are not available

* plan description of a large plan is truncated

This is because the live UI depends on an in-memory KV store. We should always be worried

about the stability issues when adding more information to the store.

Therefore, it's better to add a disk-based store to save more information
Bug: Handle Hive's bucket spec case preserving behaviour
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/22 16:23
Affects Version/s: 3.3.0

{code}
CREATE TABLE t(
 c STRING,
 B_C STRING
)
PARTITIONED BY (p_c STRING)
CLUSTERED BY (B_C) INTO 4 BUCKETS
STORED AS PARQUET
{code}
then
{code}
SELECT * FROM t
{code}
fails with:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns B_C is not part of the table columns ([FieldSchema(name:c, type:string, comment:null), FieldSchema(name:b_c, type:string, comment:null)]
	at org.apache.hadoop.hive.ql.metadata.Table.setBucketCols(Table.java:552)
	at org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1098)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:764)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
	at org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:763)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listPartitionsByFilter$1(HiveExternalCatalog.scala:1287)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
	... 110 more
{code}
Bug: Add Shell Code Style Check Action
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Mar/22 09:07
Affects Version/s: 3.4.0

There is no shell code check in the current spark github actions. Some shell codes are written incorrectly and run abnormally in special cases. Besides, they cannot also pass the check of the shellcheck plugin, especially in IDEA or shellcheck actions.