av9ash/GitBugs huggingface dataset:
Bug: Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_column
Status: Open | Priority: Major | Resolution: 
Created: 31/Dec/22 01:38
Affects Version/s: 3.4.0

{code}
======================================================================
ERROR [0.901s]: test_column_accessor (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py", line 744, in test_column_accessor
    cdf.select(CF.col("z")[0], cdf.z[10], CF.col("z")[-10]).toPandas(),
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in toPandas
    return self._session.client.to_pandas(query)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
    return self._execute_and_fetch(req)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkArrayIndexOutOfBoundsException) [INVALID_ARRAY_INDEX] The index 10 is out of bounds. The array has 3 elements. Use the SQL function `get()` to tolerate accessing element at invalid index and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

======================================================================
ERROR [0.245s]: test_column_arithmetic_ops (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py", line 799, in test_column_arithmetic_ops
    cdf.select(cdf.a % cdf["b"], cdf["a"] % 2, 12 % cdf.c).toPandas(),
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in toPandas
    return self._session.client.to_pandas(query)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
    return self._execute_and_fetch(req)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkArithmeticException) [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

{code}
Bug: GROUP BY ALL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/22 23:01
Affects Version/s: 3.3.0

This patch implements GROUP BY ALL, similar to the one initially implemented in DuckDB. When specified, the analyzer automatically infers the grouping columns based on the expressions specified in the select clause: all expressions that don't include any aggregate expressions are pulled implicitly into the grouping columns. This avoids users having to specify individually the list of grouping columns in most cases.

Examples: 
{noformat}
select key, count, sum(score) from table group by all
-- rewritten to
select key, count, sum(score) from table group by key{noformat}
Bug: GRPC Errors on the client should be cleaned up
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/22 15:03
Affects Version/s: 3.4.0

When the server throws an exception we report a very deep stack trace that is not helpful for the user. 

We need to separate the cause from the user visible exception and wrap the error into custom exception instead of publishing the RPCError from GRPC
Bug: `min` fails on the minimal timestamp
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Dec/22 09:08
Affects Version/s: 3.4.0

The code below demonstrates the issue:

{code:python}
>>> from datetime import datetime, timezone
>>> from pyspark.sql.types import TimestampType
>>> from pyspark.sql import functions as F
>>> ts = spark.createDataFrame([datetime(1, 1, 1, 0, 0, 0, 0, tzinfo=timezone.utc)], TimestampType()).toDF("test_column")
>>> ts.select(F.min('test_column')).first()[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/dataframe.py", line 2762, in first
    return self.head()
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/dataframe.py", line 2738, in head
    rs = self.head(1)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/dataframe.py", line 2740, in head
    return self.take(n)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/dataframe.py", line 1297, in take
    return self.limit(num).collect()
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/dataframe.py", line 1198, in collect
    return list(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/serializers.py", line 152, in load_stream
    yield self._read_with_length(stream)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/serializers.py", line 174, in _read_with_length
    return self.loads(obj)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/serializers.py", line 472, in loads
    return cloudpickle.loads(obj, encoding=encoding)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/types.py", line 2010, in <lambda>
    return lambda *a: dataType.fromInternal(a)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/types.py", line 1018, in fromInternal
    values = [
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/types.py", line 1019, in <listcomp>
    f.fromInternal(v) if c else v
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/types.py", line 667, in fromInternal
    return self.dataType.fromInternal(obj)
  File "/Users/maximgekk/proj/apache-spark/python/pyspark/sql/types.py", line 279, in fromInternal
    return datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)
ValueError: year 0 is out of range
{code}
Bug: Create a TIME data type
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/22 16:05
Affects Version/s: 3.3.1

As a user of Spark SQL, I wish there was a {{TIME}} data type representing a time of day.

Spark has recently added support for a {{DATE}} data type, backed by both {{java.sql.Date}} and {{java.time.LocalDate}} Java types. The new {{TIME}} data type would be backed by {{java.time.LocalTime}}.

Today, the only alternative is to use {{TIMESTAMP}}. However, displaying rows with TIMESTAMP still shows a date in 1970 (ex: {{1970-01-01 18:20:00}}) which is both confusing and less readable.
Bug: Assign name to _LEGACY_ERROR_TEMP_2149
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 05:59
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: SPJ: Reduce Spark shuffle when only one side of a join is KeyGroupedPartitioning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 17:20
Affects Version/s: 3.3.1

When only one side of a SPJ (Storage-Partitioned Join) is {{{}KeyGroupedPartitioning{}}}, Spark currently needs to shuffle both sides using {{{}HashPartitioning{}}}. However, we may just need to shuffle the other side according to the partition transforms defined in {{{}KeyGroupedPartitioning{}}}. This is especially useful when the other side is relatively small.
Bug: DataSourceStrategy#selectFilters returns predicates in non-deterministic order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/22 00:04
Affects Version/s: 3.1.0

Method org.apache.spark.sql.execution.datasources.DataSourceStrategy#selectFilters, which is used to determine "pushdown-able" filters, does not preserve the order of the input {{Seq[Expression]}} nor does it return the same order across the same plans (modulo ExprId differences). This is resulting in CodeGenerator cache misses even when the exact same LogicalPlan is executed. 

The aforementioned method does not attempt to maintain the order of the input predicates, though it happens to do so when there are less than 5 pushdown-able {{Expression}} in the input (due to some "small maps" logic in {{{}scala.collection.TraversableOnce#toMap{}}}). 

Returning in the same order as the input will reduce churn on the CodeGenerator cache under prolonged workloads that execute queries that are very similar.
Bug: Split of API classes from Catalyst
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/22 00:52
Affects Version/s: 3.4.0

For the Spark Connect Scala Client we need a couple of classes that currently reside in Catalyst to be moved to a new sql/api project.

Concretely the following classes will be moved:
 * Row
 * DataType the entire hierarchy
 * Encoder
Bug: spark3 stagedir can't be change 
Status: Open | Priority: Major | Resolution: 
Created: 06/Dec/22 03:35
Affects Version/s: 3.2.2

i want't change different staging dir when write temporary data using , but spark3 seen can only write in table path

spark.yarn.stagingDir parameter only work when use spark2

 

in org.apache.spark.internal.io.FileCommitProtocol  file :  

  def getStagingDir(path: String, jobId: String): Path = {
    new Path(path, ".spark-staging-" + jobId)
  }
}
Bug: Test parity: pyspark.sql.tests.test_dataframe
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Dec/22 07:16
Affects Version/s: 3.4.0

After https://github.com/apache/spark/pull/39041 (SPARK-41528), we now reuses the same test cases, see {{python/pyspark/sql/tests/connect/test_parity_dataframe.py}}.

We should remove all the test cases defined there, and fix Spark Connect behaviours accordingly.
Bug: Test parity: pyspark.sql.tests.test_functions
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Dec/22 07:17
Affects Version/s: 3.4.0

After https://github.com/apache/spark/pull/39041 (SPARK-41528), we now reuses the same test cases, see {{python/pyspark/sql/tests/connect/test_parity_functions.py}}.

We should remove all the test cases defined there, and fix Spark Connect behaviours accordingly.
Bug: Implement training functions as input
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 18:39
Affects Version/s: 3.4.0

Sidenote: make formatting updates described in https://github.com/apache/spark/pull/39188

 

Currently, `Distributor().run(...)` takes only files as input. Now we will add in additional functionality to take in functions as well. This will require us to go through the following process on each task in the executor nodes:
1. take the input function and args and pickle them
2. Create a temp train.py file that looks like
{code:java}
import cloudpickle
import os
if _name_ == "_main_":
    train, args = cloudpickle.load(f"{tempdir}/train_input.pkl")
    output = train(*args)
    if output and os.environ.get("RANK", "") == "0": # this is for partitionId == 0
        cloudpickle.dump(f"{tempdir}/train_output.pkl") {code}
3. Run that train.py file with `torchrun`

4. Check if `train_output.pkl` has been created on process on partitionId == 0, if it has, then deserialize it and return that output through `.collect()`
Bug: The output column name of `groupBy.agg(count_distinct)` is incorrect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/22 11:28
Affects Version/s: 3.2.0


scala> val df = spark.range(1, 10).withColumn("value", lit(1))
df: org.apache.spark.sql.DataFrame = [id: bigint, value: int]

scala> df.createOrReplaceTempView("table")

scala> df.groupBy("id").agg(count_distinct($"value"))
res1: org.apache.spark.sql.DataFrame = [id: bigint, count(value): bigint]

scala> spark.sql(" SELECT id, COUNT(DISTINCT value) FROM table GROUP BY id ")
res2: org.apache.spark.sql.DataFrame = [id: bigint, count(DISTINCT value): bigint]

scala> df.groupBy("id").agg(count_distinct($"*"))
res3: org.apache.spark.sql.DataFrame = [id: bigint, count(unresolvedstar()): bigint]

scala> spark.sql(" SELECT id, COUNT(DISTINCT *) FROM table GROUP BY id ")
res4: org.apache.spark.sql.DataFrame = [id: bigint, count(DISTINCT id, value): bigint]
Bug: Support for User-defined Functions in Python
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Dec/22 09:42
Affects Version/s: 3.4.0

User-defined Functions in Python consist of (pickled) Python UDFs and (Arrow-optimized) Pandas UDFs. They enable users to run arbitrary Python code on top of the Apache Spark™ engine. Users only have to state "what to do"; PySpark, as a sandbox, encapsulates "how to do it".

Spark Connect Python Client (SCPC), as a client and server interface for PySpark will eventually replace the legacy API of PySpark. Supporting PySpark UDFs is essential for Spark Connect to reach parity with the PySpark legacy API.

See design doc [here|https://docs.google.com/document/d/e/2PACX-1vRXF8nTdjwH0LbYyp3b6Zt6STEKWsvfKSO7_s4foOB-3zJ2h4_06JF147hUPlADJxZ_X22RFxgZ-fRS/pub].
Bug: Union does not propagate Metadata output
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Dec/22 12:41
Affects Version/s: 3.1.2

Currently, the Union operator does not propagate any metadata output. This makes it impossible to access any metadata if a Union operator is used, even though the children have the exact same metadata output.
Example:

 
{code:java}
val df1 = spark.read.load(path1)
val df2 = spark.read.load(path2)
df1.union(df2).select("_metadata.file_path"). // <-- fails{code}
Bug: Numpy 1.24 breaks PySpark due to use of `np.bool` instead of `np.bool_` in many places
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 26/Dec/22 12:54
Affects Version/s: 3.3.0

In numpy 1.24, `numpy.bool` was removed (it was deprecated prior to 1.24). This causes many APIs in pyspark to stop working because an AttributeError is raised. The alternative is to use `numpy.bool_` (trailing underscore).
Bug: Remove the workaround of sql(...).collect back in PySpark tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/22 05:21
Affects Version/s: 3.4.0

See https://github.com/apache/spark/pull/39224/files#r1057436437

We don't have to `collect` for every `sql` but Spark Connect requires it. We should remove them out.
Bug: [SQL] ParquetFilters StringStartsWith push down matching string do not use UTF-8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 09:59
Affects Version/s: 3.4.0

Hello ~
 
I found a problem, but there are two ways to solve it.
 
The parquet filter is pushed down. When using the like '***%' statement to query, if the system default encoding is not UTF-8, it may cause an error.
 
There are two ways to bypass this problem as far as I know
1. spark.executor.extraJavaOptions="-Dfile.encoding=UTF-8"
2. spark.sql.parquet.filterPushdown.string.startsWith=false
 

The following is the information to reproduce this problem

The parquet sample file is in the attachment
{code:java}
spark.read.parquet("file:///home/kylin/hjldir/part-00000-30432312-7cdb-43ef-befe-93bcfd174878-c000.snappy.parquet").createTempView("tmp”)
spark.sql("select * from tmp where `1` like '啦啦乐乐%'").show(false) {code}
!image-2022-12-28-18-00-00-861.png|width=879,height=430!
 
  !image-2022-12-28-18-00-21-586.png|width=799,height=731!
 
I think the correct code should be:
{code:java}
private val strToBinary = Binary.fromReusedByteArray(v.getBytes(StandardCharsets.UTF_8)) {code}
Bug: Make consistent MR job IDs in FileBatchWriter and FileFormatWriter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/22 10:22
Affects Version/s: 3.0.0

[SPARK-26873|https://issues.apache.org/jira/browse/SPARK-26873] fix the consistent issue for FileFormatWriter,  but [SPARK-33402|https://issues.apache.org/jira/browse/SPARK-33402] break this requirement by introducing a random long, we need to address this to expects identical task IDs across attempts for correctness.

Also FileBatchWriter doesn't follow this requirement, need to fix it as well.
Bug: ClassCastException when config spark.sql.hive.metastore* properties under jdk17
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 27/Dec/22 05:59
Affects Version/s: 3.3.1

Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0

when config properties about spark.sql.hive.metastore* to use hive.metastore.version 3.1.2: 
*spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
*spark.sql.hive.metastore.version 3.1.2*
then start spark-shell with master = local[*] under jdk17 

try to select a hive table, will got error:

13:44:52.428 [main] ERROR org.apache.hadoop.hive.metastore.utils.MetaStoreUtils - Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader 'bootstrap')
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262) ~[hive-standalone-metastore-3.1.2.jar:3.1.2]
Bug: Improve PySpark errors
Status: Resolved | Priority: Major | Resolution: Done
Created: 20/Dec/22 10:21
Affects Version/s: 3.4.0

This ticket aims to introduce new PySpark framework to centralize the PySpark error message into single path and improve the error message more actionable and consistency.

This umbrella JIRA might includes:
 * Introduce new error framework for PySpark
 * Migrate existing errors generated by Python driver into error classes.
 * Migrate existing errors generated by Python worker into error classes.
 * Migrate existing errors generated by Py4J into error classes.
 * Introduce test utils for testing errors by its error class instead of error messages.
 * Improve the error messages.
 * Documentation for PySpark error framework.
Bug: SPJ: Spark shouldn't assume InternalRow implements equals and hashCode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 17:16
Affects Version/s: 3.3.1

Currently SPJ (Storage-Partitioned Join) actually assumes the {{InternalRow}} returned by {{HasPartitionKey}} implements {{equals}} and {{{}hashCode{}}}. We should remove this restriction.
Bug: Decimal.changePrecision produces ArrayIndexOutOfBoundsException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/22 21:04
Affects Version/s: 3.3.1

{{Reducing Decimal scale by more than 18 produces exception.}}
{code:java}
Decimal(1, 38, 19).changePrecision(38, 0){code}
{code:java}
java.lang.ArrayIndexOutOfBoundsException: 19
    at org.apache.spark.sql.types.Decimal.changePrecision(Decimal.scala:377)
    at org.apache.spark.sql.types.Decimal.changePrecision(Decimal.scala:328){code}
Reproducing with SQL query:
{code:java}
sql("select cast(cast(cast(cast(id as decimal(38,15)) as decimal(38,30)) as decimal(38,37)) as decimal(38,17)) from range(3)").show{code}
The bug exists for {{Decimal}} that is stored using compact long only, it works fine with {{Decimal}} that uses {{scala.math.BigDecimal}} internally.
Bug: Add the error class UNRESOLVED_ROUTINE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/22 17:14
Affects Version/s: 3.4.0

Add new error class UNRESOLVED_ROUTINE and replace the following error classes:
* _LEGACY_ERROR_TEMP_1041
* _LEGACY_ERROR_TEMP_1242
* _LEGACY_ERROR_TEMP_1243
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2422
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 16:32
Affects Version/s: 3.4.0

Choose a proper name to the legacy error class _LEGACY_ERROR_TEMP_2422, and double check that there are tests for the error class, and improve the error message format.
Bug: Assign name to _LEGACY_ERROR_TEMP_2141
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Any SparkThrowable (with an error class) not in error-classes.json is masked in SQLExecution.withNewExecutionId and end-user will see "org.apache.spark.SparkException: [INTERNAL_ERROR]" 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 01:47
Affects Version/s: 3.4.0

This change [here|https://github.com/apache/spark/pull/38302/files#diff-fdd1e9e26aa1ba9d1cc923ee7c84a1935dcc285502330a471f1ade7f3ad08bf9] means that any seen error is passed to SparkThrowableHelper.getMessage(...). Any SparkThrowable with an error class (for example, if a connector uses the spark error format i.e. see ErrorClassesJsonReader) will be masked as 
{code:java}
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot find main error class 'SOME_ERROR_CLASS'{code}
in SparkThrowableHelper.getMessage since errorReader.getMessageTemplate(errorClass) will fail for the error class not defined in error-classes.json.
Bug: spark3 read the one partition data and write to anthor partition cause error
Status: Open | Priority: Major | Resolution: 
Created: 22/Dec/22 08:53
Affects Version/s: 3.2.2

spark3 read the one partition data and write to anthor partition cause error
{code:java}
1 create temporary view t1 :

 select * from jt_ods.ods_ebi_stm_retail_settle_detail_full_di 
where dt = '2022-12-21' 
union all ( 
select * from jt_ods.ods_ebi_stm_retail_settle_detail_full_df as i 
where i.dt = '2022-12-20'  
and not exists(select 1 from jt_ods.ods_ebi_stm_retail_settle_detail_full_di as d where  d.dt = '2022-12-21' and i.id = d.id))

2 insert data :

 insert sql insert overwrite table jt_ods.ods_ebi_stm_retail_settle_detail_full_df partition(dt = '2022-12-21') select * from t distribute by rand() {code}
{code:java}
2022-12-22 16:29:48 Driver ERROR org.apache.spark.deploy.yarn.ApplicationMaster 
 User class threw exception: org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.
org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.
    at org.apache.spark.sql.errors.QueryCompilationErrors$.cannotOverwritePathBeingReadFromError(QueryCompilationErrors.scala:1834)
    at org.apache.spark.sql.execution.command.DDLUtils$.verifyNotReadPath(ddl.scala:980)
    at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:221) {code}