av9ash/GitBugs huggingface dataset:
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2020
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:37
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2020* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2021
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:37
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2021* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2024
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:38
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2024* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Support only decommission idle workers in standalone
Status: Open | Priority: Minor | Resolution: 
Created: 22/Apr/23 18:21
Affects Version/s: 3.4.0

Currently, standalone master web ui supports kill/decommission workers. But when graceful decommission is enabled, running task, shuffle and rdd migration could take long time. While waiting for running task, shuffle and rdd migration, decommissioned workers can't run new executors and decommissioned executors can't run new tasks. This caused lot of resource waste.

If only idle workers to be decommissioned, these workers could be shutdown and removed to save cost without waiting long decommissioning process.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2016
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:35
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2016* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Make improvement to LocalKMeans
Status: Open | Priority: Minor | Resolution: 
Created: 26/Apr/23 17:00
Affects Version/s: 3.3.0

There are two initializationMode in Kmeans, random mode and parallel mode.

The ParallelMode is using kmeansPlusPlus to generate the centers point, but the kMeansPlusPlus is a local method which runs in the driver.

If the scale of points is huge, the kMeansPlusPlus will run for a long time, because it is a single thread method running in the driiver.

We can make this method run in parallel to make it faster, such as using Parallel collections.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2018
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:36
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2018* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2015
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:35
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2015* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Upgrade mypy package in python pyspark dependencies
Status: Open | Priority: Minor | Resolution: 
Created: 19/Apr/23 08:04
Affects Version/s: 3.2.4

While working on this issue, I noticed that mypy is in a quite older version. Running it on version 1.2.0 does not show any problems with the current tests.

This will allow me to get better support from mypy authors on why mypy fails on the typing.io issue linked above.
Bug: Support IV and AAD optional parameters for aes_encrypt / ExpressionImplUtil
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/23 04:30
Affects Version/s: 3.5.0

There are some use cases where callers to aes_encrypt may want to provide initialization vectors (IVs) or additional authenticated data (AAD). The most common cases will be:
1. Ensuring that ciphertext matches values that have been encrypted by external tools. In those cases, the caller will need to provide an identical IV value.
2. For AES-CBC mode, there are some cases where callers want to generate deterministic encrypted output.
3. For AES-GCM mode, providing AAD fields allows callers to bind additional data to an encrypted ciphertext so that it can only be decrypted by a caller providing the same value. This is often used to enforce some context.

The proposed new API is the following:
 * aes_encrypt(expr, key [, mode [, padding [, iv [, aad]]]])

 * aes_decrypt(expr, key [, mode [, padding [, aad]]])

These fields are only supported for specific modes:
 * ECB: Does not support either IV or AAD and will return an error if either are provided.
 * CBC: Only supports an IV and will return an error if an AAD is provided
 * GCM: Supports either IV, AAD, or both.

If a caller is only providing an AAD to GCM mode, they would need to pass a null value in the IV field.
Bug: Spark SQL docs for explode_outer and posexplode_outer omit behavior for null/empty
Status: Open | Priority: Minor | Resolution: 
Created: 28/Apr/23 23:23
Affects Version/s: 3.4.0

The Spark SQL documentation for [explode_outer|https://spark.apache.org/docs/latest/api/sql/index.html#explode_outer] and [posexplode_outer|[https://spark.apache.org/docs/latest/api/sql/index.html#posexplode_outer|https://spark.apache.org/docs/latest/api/sql/index.html#posexplode_outer)]] omits mentioning that null or empty arrays produce nulls. The descriptions do not appear to be written down in a doc file and are likely pulled from the `ExpressionDescription` tags for the `Explode` and `PosExplode` generators when the `GeneratorOuter` wrapper is used.
Bug: ClientDistributedCacheManager doesn't set the LocalResourceVisibility.PRIVATE if isPublic throws exception
Status: Open | Priority: Minor | Resolution: 
Created: 22/Apr/23 09:40
Affects Version/s: 3.4.0

Hi Spark Team .

Currently *ClientDistributedCacheManager* *getVisibility* methods checks whether resource visibility can be set to private or public. 

In order to set  *LocalResourceVisibility.PUBLIC* ,isPublic checks permission of all the ancestors directories for the executable directory . It goes till the root folder to check permission of all the parents (ancestorsHaveExecutePermissions) 

checkPermissionOfOther calls  FileStatus getFileStatus to check the permission .

If the   FileStatus getFileStatus throws exception Spark Submit fails . It didn't sets the permission to Private.

if (isPublic(conf, uri, statCache))

{ LocalResourceVisibility.PUBLIC }

else

{ LocalResourceVisibility.PRIVATE }

Generally if the user doesn't have permission to check for root folder (specifically in case of cloud file system(GCS)  (for the buckets)  , methods throws error IOException(Error accessing Bucket).

 

*Ideally if there is an error in isPublic , which means Spark isn't able to determine the execution permission of all the parents directory , it should set the LocalResourceVisibility.PRIVATE.  However, it currently throws an exception in isPublic and hence Spark Submit fails*
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2014
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:34
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2014* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: CoalesceBucketsInJoin not working with SHUFFLE_HASH hint
Status: Open | Priority: Minor | Resolution: 
Created: 29/Apr/23 13:03
Affects Version/s: 3.3.2

h1. NOTICE: related to SPARK-43021
h1. What I did

I define the following code:

{{from pyspark.sql import SparkSession}}

{{spark = (}}
{{  SparkSession}}
{{    .builder}}
{{    .appName("Bucketing")}}
{{    .master("local[4]")}}
{{    .config("spark.sql.bucketing.coalesceBucketsInJoin.enabled", True)}}
{{    .config("spark.sql.autoBroadcastJoinThreshold", "-1")}}
{{    .getOrCreate()}}
{{)}}

{{# AQE prevents CoalesceBucketsInJoin in 3.3.2 in 3.3.2}}
{{spark.conf.set("spark.sql.adaptive.enabled", False)}}

{{df1 = spark.range(0, 100)}}
{{df2 = spark.range(0, 100, 2)}}

{{df1.write.bucketBy(4, "id").mode("overwrite").saveAsTable("t1")}}
{{df2.write.bucketBy(2, "id").mode("overwrite").saveAsTable("t2")}}

{{t1 = spark.table("t1")}}
{{t2 = spark.table("t2")}}

{{t2.join(t1.hint("SHUFFLE_HASH"), "id").explain()}}

{{== Physical Plan ==}}
{{*(3) Project [id#23L|#23L]}}
{{+- *(3) ShuffledHashJoin [id#23L|#23L], [id#21L|#21L], Inner, BuildRight}}
{{:- Exchange hashpartitioning(id#23L, 4), ENSURE_REQUIREMENTS, [plan_id=667]}}
{{: +- *(1) Filter isnotnull(id#23L)}}
{{: +- *(1) ColumnarToRow}}
{{: +- FileScan parquet spark_catalog.default.t2[id#23L|#23L] Batched: true, Bucketed: false (disabled by query planner), DataFilters: [isnotnull(id#23L)|#23L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/jovyan/notebooks/spark-warehouse/t2|file:///home/jovyan/notebooks/spark-warehouse/t2], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>}}
{{+- *(2) Filter isnotnull(id#21L)}}
{{+- *(2) ColumnarToRow}}
{{+- FileScan parquet spark_catalog.default.t1[id#21L|#21L] Batched: true, Bucketed: true, DataFilters: [isnotnull(id#21L)|#21L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/jovyan/notebooks/spark-warehouse/t1|file:///home/jovyan/notebooks/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 4 out of 4}}
h1. What happened

There is an Exchange node in the join plan
h1. What is expected

According to the docs about [{{spark.sql.bucketing.coalesceBucketsInJoin.enabled}}|https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration] there should not be any  Exchange/Shuffle nodes in the plan (the number of buckets of t1 is divided by the number of buckets of t2), but it doesn't say anything about what should happen if there are hints applied.

h1. Observation

# I checked other hints (MERGE and SHUFFLE_REPLICATE_NL) and they don't yield any Exchange nodes in the plan
# When I apply the hint to the t2 table ({{t2.hint("SHUFFLE_HASH").join(t1, "id")}}), no exchange happen for any hint.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2017
Status: Open | Priority: Minor | Resolution: 
Created: 24/Apr/23 09:36
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2017* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2022
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:38
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2022* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Adding support for MissingFiles when spark.sql.parquet.mergeSchema=true
Status: Open | Priority: Minor | Resolution: 
Created: 24/Apr/23 17:16
Affects Version/s: 3.4.0

Hello,

{{With spark.sql.files.ignoreMissingFiles=true we notice [readParquetFootersInParallel|https://github.com/apache/spark/blob/52c1068190803d856959ba563642a3e440cc086c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L438] can still encounter FileNotFoundException. }}

I notice function readParquetFootersInParallel handle the scenario where {{{*}spark.sql.files.ignoreCorruptFiles=true{*}.}}

My question: Would it be feasible to support the scenario where *spark.sql.files.ignoreMissingFiles=true* in the function: readParquetFootersInParallel  as well ? To prevent application failure due to {{FileNotFoundException{*}.{*}}}

 

Thank you
Bug: Correctly recurse into maps of maps and arrays of arrays in StructType.findNestedField
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/23 12:19
Affects Version/s: 3.4.0

[StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325] is unable to reach nested fields below two directly nested maps or arrays. Whenever it reaches a map or an array, it'll throw an `invalidFieldName` exception if the child is not a struct.

The following throws '{{{}Field name `a`.`element`.element`.`i` is invalid: `a`.`element`.`element` is not a struct.'{}}}, even though the access path is valid:
{code:java}
val schema = new StructType()
  .add("a", ArrayType(ArrayType(
    new StructType().add("i", "int"))))
findNestedField(Seq("a", "element", "element", "i"), schema) {code}
Bug: Remove typing.io namespace references as it is being removed
Status: Open | Priority: Minor | Resolution: 
Created: 17/Apr/23 07:08
Affects Version/s: 3.2.4

Python 3.11 gives a deprecation warning to the following:
{code:java}
 /python/3.11.1/lib/python3.11/site-packages/pyspark/broadcast.py:38: DeprecationWarning: typing.io is deprecated, import directly from typing instead. typing.io will be removed in Python 3.12.
    from typing.io import BinaryIO  # type: ignore[import]{code}
The only reference comes from:
{code:java}
spark % git grep typing.io 
python/pyspark/broadcast.py:from typing.io import BinaryIO  # type: ignore[import] {code}
I will fix the import so it does not cause any deprecation problem.

 

This is documeted in [1|[https://bugs.python.org/issue35089]], [2|https://docs.python.org/3/library/typing.html#typing.IO]
Bug: `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Apr/23 09:39
Affects Version/s: 3.5.0

both

```

build/mvn clean install -Dtest=none -DwildcardSuites=org.apache.spark.sql.ClientE2ETestSuite   

```

and 

```

build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"

```

 

will test failed when using Java 11&17 

 
{code:java}
- write jdbc *** FAILED ***
  io.grpc.StatusRuntimeException: INTERNAL: No suitable driver
  at io.grpc.Status.asRuntimeException(Status.java:535)
  at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at org.apache.spark.sql.SparkSession.execute(SparkSession.scala:458)
  at org.apache.spark.sql.DataFrameWriter.executeWriteOperation(DataFrameWriter.scala:257)
  at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:221)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:218) {code}