av9ash/GitBugs huggingface dataset:
Bug: Use AnalyzeTableCommand overwrite statistics information incorrectly
Status: Open | Priority: Major | Resolution: 
Created: 07/Feb/24 02:31
Affects Version/s: 2.4.4

When the size of the table changes but the total number of rows in the table does not change, I use the sql statement "analyze table student compute statistics" to analyze the external table statistics. The Statistics information returned by the sql statement "desc extended student" only contains the table Size information, excluding rowCounts information.

Specific operating instructions:
{code:sql}
Create external table

create table student(id int, name string, age int) row format delimited fields terminated by ','
lines terminated by '\n' location 'hdfs://nameservice/spark/student';


The contents of the external table file are as follows:
class1.txt:

1,'Jack',25
2,'Thompson',28


class2.txt:

3,'Davy',30
4,'Thompson',35


class3.txt：

5,'Curry',40
6,'Morgan',20


Import external table data

hdfs dfs -put 1.txt /spark/student
hdfs dfs -put 2.txt /spark/student


Analyze external table statistics

analyze table student compute statistics;
desc extended student;

Return results

Type EXTERNAL
Provider hive
Table Properties [transient_lastDdlTime=1707265554]
Statistics 56 bytes, 4 rows
Location hdfs://nameservice/spark/student
Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat org.apache.hadoop.mapred.TextInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties [serialization.format=,, line.delim=
, field.delim=,]
Partition Provider Catalog


Modify external table

hdfs dfs -rm /spark/student/student2.txt
hdfs dfs -put student3.txt /spark/student


Analyze the external table again

analyze table student compute statistics;
desc extended student;

Return results

Type EXTERNAL
Provider hive
Table Properties [transient_lastDdlTime=1707265719]
Statistics 55 bytes
Location hdfs://nameservice/spark/student
Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat org.apache.hadoop.mapred.TextInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties [serialization.format=,, line.delim=
, field.delim=,]
Partition Provider Catalog
{code}
Through the above operation results, I found that when the table size changes but the number of rows does not change, the statistics should include the new table size and the total number of rows. I don’t know if it is correct to only display the statistics of the table size.
Bug: missing timezone to coerce default values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/24 10:39
Affects Version/s: 3.5.0

```
create table src(key int, c string DEFAULT date'2018-11-17') using parquet;
Time taken: 0.133 seconds
spark-sql (default)> desc src;
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
```
Bug: Ignore --num-executors config if DYN_ALLOCATION_ENABLED is true and allow remove idle executors
Status: Open | Priority: Major | Resolution: 
Created: 01/Feb/24 08:46
Affects Version/s: 4.0.0

In https://issues.apache.org/jira/browse/SPARK-13723 , conf --num-executors will be used to compute initialExecutors and initial # of executors.

I think that may conflict to DYN_ALLOCATION_MIN_EXECUTORS and will not free the idle executors.

{code}
... 
  private def removeExecutors(executors: Seq[(String, Int)]): Seq[String] = synchronized {
        if (newExecutorTotal - 1 < minNumExecutors) {
          logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
            s"are only $newExecutorTotal executor(s) left (minimum number of executor limit " +
            s"$minNumExecutors)")
        } else if (newExecutorTotal - 1 < numExecutorsTargetPerResourceProfileId(rpId)) {
          logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " +
            s"are only $newExecutorTotal executor(s) left (number of executor " +
            s"target ${numExecutorsTargetPerResourceProfileId(rpId)})")
        } else {
          executorIdsToBeRemoved += executorIdToBeRemoved
          numExecutorsTotalPerRpId(rpId) -= 1
        }
  }
{code}
Bug: Replace UnsupportedOperationException by SparkUnsupportedOperationException in sql
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/24 07:35
Affects Version/s: 4.0.0

Replace all UnsupportedOperationException by SparkUnsupportedOperationException in the *sql* code base, and introduce new legacy error classes with the _LEGACY_ERROR_TEMP_ prefix.
Bug: Supporting broadcast of multiple filtering keys in DynamicPruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/24 21:53
Affects Version/s: 3.5.1

This PR extends `DynamicPruningSubquery` to support broadcasting of multiple filtering keys (instead of one as before). The majority of the PR is to simply generalise singularity to plurality.

Note: We actually do not use the multiple filtering keys `DynamicPruningSubquery` in this PR, we are doing this to make supporting DPP Null Safe Equality or multiple Equality predicates easier in the future.

In Null Safe Equality JOIN, the JOIN condition `a <=> b` is transformed to `Coalesce(key1, Literal(key1.dataType)) = Coalesce(key2, Literal(key2.dataType)) AND IsNull(key1) = IsNull(key2)`. In order to have the highest pruning efficiency, we broadcast the 2 keys `Coalesce(key, Literal(key.dataType))` and `IsNull(key)` and use them to prune the other side at the same time. 

Before, the `DynamicPruningSubquery` only has one broadcasting key and we only supports DPP for one `EqualTo` JOIN predicate, now we are extending the subquery to multiple broadcasting keys. Please note that DPP has not been supported for multiple JOIN predicates. 

Put it in another way, at the moment, we don't insert a DPP Filter for multiple JOIN predicates at the same time, only potentially insert a DPP Filter for a given Equality JOIN predicate.
Bug: String length (...) exceeds the maximum length (20000000)
Status: Open | Priority: Minor | Resolution: 
Created: 23/Feb/24 22:58
Affects Version/s: 3.5.0

Upgrading to Spark 3.5.0 introduced a regression for us where our query gateway (Livy) fails with an error:
{code:java}
com.fasterxml.jackson.core.exc.StreamConstraintsException: String length (20054016) exceeds the maximum length (20000000)

(sorry, unable to provide full stack trace){code}
The root of this problem is the breaking change in {{jackson}} that (in the name of "safety") introduced some JSON size limits, see: [https://github.com/FasterXML/jackson-core/issues/1014]

Looks like {{JSONOptions}} in Spark already [support configuring this limit|https://github.com/apache/spark/blob/c2dbb6d04bc9c781fb4a7673e5acf2c67b99c203/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala#L55-L58], but there seems to be no way to set it globally or pass it down to [{{DataFrame::toJSON()}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.toJSON.html] which our Apache Livy server is using when transmitting data.

Livy is an old project and transferring dataframes via JSON is super inefficient, and we really should move to something like Spark Connect, but I believe this issue can happen to many people working with basic GeoJSON data.

Spark can handle very large strings, and this arbitrary limit just gets in a way of output serialization for no good reason.
Bug: Add support for checkpointing to Spark Connect
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 27/Feb/24 15:07
Affects Version/s: 4.0.0

The {{sparkContext}} that underlies a given {{SparkSession}} is not accessible over Spark Connect. This means you cannot call {{spark.sparkContext.setCheckpointDir(...)}}, which in turn means you cannot checkpoint a DataFrame.

We should add support for this somehow to Spark Connect.
Bug: Upgrade Transport block cipher mode to GCM
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/24 17:53
Affects Version/s: 3.4.2

The cipher transformation currently used for encrypting RPC calls is an unauthenticated mode (AES/CTR/NoPadding). This needs to be upgraded to an authenticated mode (AES/GCM/NoPadding) to prevent ciphertext from being modified in transit.

The relevant line is here: [https://github.com/apache/spark/blob/a939a7d0fd9c6b23c879cbee05275c6fbc939e38/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java#L220]

GCM is relatively more computationally expensive than CTR and adds a 16-byte block of authentication tag data to each payload.
Bug: Add prefix into TemporaryDirectory to avoid flakiness
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/24 01:33
Affects Version/s: 4.0.0

Sometimes the test fail because the temporary directory names are same (https://github.com/apache/spark/actions/runs/8066850485/job/22036007390).

{code}
File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line ?, in pyspark.sql.dataframe.DataFrame.writeStream
Failed example:
    with tempfile.TemporaryDirectory() as d:
        # Create a table with Rate source.
        df.writeStream.toTable(
            "my_table", checkpointLocation=d)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.11/doctest.py", line 1353, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.dataframe.DataFrame.writeStream[3]>", line 1, in <module>
        with tempfile.TemporaryDirectory() as d:
      File "/usr/lib/python3.11/tempfile.py", line 1043, in __exit__
        self.cleanup()
      File "/usr/lib/python3.11/tempfile.py", line 1047, in cleanup
        self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
      File "/usr/lib/python3.11/tempfile.py", line 1029, in _rmtree
        _rmtree(name, onerror=onerror)
      File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
        os.rmdir(path, dir_fd=dir_fd)
    OSError: [Errno 39] Directory not empty: '/__w/spark/spark/python/target/4f062b09-213f-4ac2-a10a-2d704990141b/tmp29irqweq'
{code}
Bug: Support for configuring ShuffledHashJoin plan size Threshold
Status: Open | Priority: Minor | Resolution: 
Created: 01/Feb/24 09:36
Affects Version/s: 3.5.0

When we enable `spark.sql.join.preferSortMergeJoin=false`, we may get the following error.
 
{code:java}
org.apache.spark.SparkException: Can't acquire 1073741824 bytes memory to build hash relation, got 478549889 bytes
    at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotAcquireMemoryToBuildLongHashedRelationError(QueryExecutionErrors.scala:795)
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:581)
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.grow(HashedRelation.scala:813)
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.append(HashedRelation.scala:761)
    at org.apache.spark.sql.execution.joins.LongHashedRelation$.apply(HashedRelation.scala:1064)
    at org.apache.spark.sql.execution.joins.HashedRelation$.apply(HashedRelation.scala:153)
    at org.apache.spark.sql.execution.joins.ShuffledHashJoinExec.buildHashedRelation(ShuffledHashJoinExec.scala:75)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.init(Unknown Source)
    at org.apache.spark.sql.execution.WholeStageCodegenExec.$anonfun$doExecute$6(WholeStageCodegenExec.scala:775)
    at org.apache.spark.sql.execution.WholeStageCodegenExec.$anonfun$doExecute$6$adapted(WholeStageCodegenExec.scala:771)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915){code}
 
Because when converting SMJ to SHJ, it only determines whether the size of the plan is smaller than `conf.autoBroadcastJoinThreshold * conf.numShufflePartitions`. 
When the configured `numShufflePartitions` is large enough, it is easy to convert to SHJ. The executor build hash relation fails due to insufficient memory.
 
[https://github.com/apache/spark/blob/223afea9960c7ef1a4c8654e043e860f6c248185/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala#L505-L513]
Bug: AttributeError: module 'pandas' has no attribute 'Timstamp'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/24 02:30
Affects Version/s: 3.5.1

When using the pyspark.sql.types.TimestampType, if your value is a datetime.datetime object with a tzinfo, [this typo|https://github.com/apache/spark/blob/master/python/pyspark/sql/pandas/types.py#L996] breaks things.

 

I believe [this commit|https://github.com/apache/spark/commit/46949e692e863992f4c50bdd482d5216d4fd9221] introduced the bug 9 months ago

 

Full stack trace below:

 
{code:java}
File "/databricks/spark/python/pyspark/worker.py", line 1490, in main process() File "/databricks/spark/python/pyspark/worker.py", line 1482, in process serializer.dump_stream(out_iter, outfile) File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 531, in dump_stream return ArrowStreamSerializer.dump_stream( File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 107, in dump_stream for batch in iterator: File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 525, in init_stream_yield_batches batch = self._create_batch(series) File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 511, in _create_batch arrs.append(self._create_array(s, t, arrow_cast=self._arrow_cast)) File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 284, in _create_array series = conv(series) File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 1060, in <lambda> return lambda pser: pser.apply( # type: ignore[return-value] File "/databricks/python/lib/python3.10/site-packages/pandas/core/series.py", line 4771, in apply return SeriesApply(self, func, convert_dtype, args, kwargs).apply() File "/databricks/python/lib/python3.10/site-packages/pandas/core/apply.py", line 1123, in apply return self.apply_standard() File "/databricks/python/lib/python3.10/site-packages/pandas/core/apply.py", line 1174, in apply_standard mapped = lib.map_infer( File "pandas/_libs/lib.pyx", line 2924, in pandas._libs.lib.map_infer File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 1061, in <lambda> lambda x: conv(x) if x is not None else None # type: ignore[misc] File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 889, in convert_array return [ File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 890, in <listcomp> _element_conv(v) if v is not None else None # type: ignore[misc] File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 1010, in convert_struct return { File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 1011, in <dictcomp> name: conv(v) if conv is not None and v is not None else v File "/databricks/spark/python/pyspark/sql/pandas/types.py", line 1032, in convert_timestamp ts = pd.Timstamp(value) File "/databricks/python/lib/python3.10/site-packages/pandas/__init__.py", line 264, in __getattr__ raise AttributeError(f"module 'pandas' has no attribute '{name}'") AttributeError: module 'pandas' has no attribute 'Timstamp'
 {code}
Bug: Add concrete relation class for JDBC relation made in V1TableScan
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/24 11:18
Affects Version/s: 3.5.1

JDBCRelation's method toV1TableScan creates v1 anonymous relation that can use predicates and other pushdowns of JDBCRelation object (v2 relation).

That relation can later be logged to telemetry or to shown in Spark UI by that name. It is not descriptive enough, so the idea is to use concrete class.
Bug: Support Stage Isolated Executor JVM Profiling 
Status: Open | Priority: Minor | Resolution: 
Created: 29/Feb/24 09:58
Affects Version/s: 4.0.0

Different stages execute different codes, and hot spots will be ignored. Task code executed in stage is same.Stage-Isolated JVM profile analysis is easier to locate problems.Support stage-level Isolated profile files.Or, we can support start Executor JVM profile from UI.
Bug: Move DataSourceMetricsMixin from JDBC RDD to JDBC Relation
Status: Open | Priority: Minor | Resolution: 
Created: 29/Feb/24 10:03
Affects Version/s: 4.0.0

We added DataSourceMetricsMixin trait recently which is holder for sequence of SQLMetric objects.

We added it to JDBCRDD, so rdd can define which metrics it has and then those metrics would be added to metrics of RowDataSourceScanExec.

The problem that can appear in the future is that someone will want to do some transformation on RDD, like mapPartitions which returns new object of RDD that does not implement our interface. Since we cannot access parent of RDD, it is impossible to get those metrics in RowDataSourceScanExec.

The better solution would be to make relations implement mentioned trait and that should solve our problem, same trait can be used, because it is not linked with RDD by name or by any property.
Bug: Uses AbstractParser instead of CsvParser for CSV parser signature
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Feb/24 02:38
Affects Version/s: 4.0.0

https://github.com/uniVocity/univocity-parsers becomes inactive for the last 3 years, and we can't land some bug fixes anymore. Maybe we should leverage their interface, and have our CSV parser. This is a base work.

We should use the higher class if it fits for better maintenance in any event.
Bug: Proposal about moving on from the Shepherd terminology in SPIPs to "Mentor"
Status: Open | Priority: Minor | Resolution: 
Created: 28/Feb/24 15:21
Affects Version/s: 3.5.1

As an active member I am proposing a move to replace the current terminology "SPIP Shepherd" with the more respectful and inclusive term "SPIP Mentor." We have over the past few years have tried to replace some past terminologies with more acceptable ones.

While some may not find "Shepherd" offensive, it can unintentionally imply passivity or dependence on community members, which might not accurately reflect their expertise and contributions. Additionally, the shepherd-sheep dynamic might be interpreted as hierarchical, which does not align with the collaborative and open nature of Spark community.

*"SPIP Mentor"* better emphasizes the collaborative nature of the process, focusing on supporting and guiding members while respecting their strengths and contributions. It also avoids any potentially offensive or hierarchical connotations.
Bug: When creating Jira please use the word "mentor" instead of "Shepherd" in SPIP
Status: Open | Priority: Minor | Resolution: 
Created: 28/Feb/24 15:10
Affects Version/s: 3.3.4

As an active member I am proposing a move to replace the current terminology "SPIP Shepherd" with the more respectful and inclusive term "SPIP Mentor." We have over the past few years have tried to replace some past terminologies with more acceptable ones.

While some may not find "Shepherd" offensive, it can unintentionally imply passivity or dependence on community members, which might not accurately reflect their expertise and contributions. Additionally, the shepherd-sheep dynamic might be interpreted as hierarchical, which does not align with the collaborative and open nature of Spark community.

*"SPIP Mentor"* better emphasizes the collaborative nature of the process, focusing on supporting and guiding members while respecting their strengths and contributions. It also avoids any potentially offensive or hierarchical connotations.
Bug: Migrate CSV parsing off of Univocity
Status: Open | Priority: Minor | Resolution: 
Created: 27/Feb/24 03:23
Affects Version/s: 4.0.0

Univocity appears to be unmaintained.

As of February 2024:
 * The last release was [more than 3 years ago|https://github.com/uniVocity/univocity-parsers/releases].
 * The last commit to {{master}} was [almost 3 years ago|https://github.com/uniVocity/univocity-parsers/commits/master/].
 * The website is [down|https://github.com/uniVocity/univocity-parsers/issues/506].
 * There are [multiple|https://github.com/uniVocity/univocity-parsers/issues/494] [open|https://github.com/uniVocity/univocity-parsers/issues/495] [bugs|https://github.com/uniVocity/univocity-parsers/issues/499] on the tracker with no indication that anyone cares.

It's not urgent, but we should consider migrating to an actively maintained CSV library in the JVM ecosystem.

There are a bunch of libraries [listed here on this Maven Repository|https://mvnrepository.com/open-source/csv-libraries].

[jackson-dataformats-text|https://github.com/FasterXML/jackson-dataformats-text] looks interesting. I know we already use FasterXML to parse JSON. Perhaps we should use them to parse CSV as well.

I'm guessing we chose Univocity back in the day because it was the fastest CSV library on the JVM. However, the last performance benchmark comparing it to others was [from February 2018|https://github.com/uniVocity/csv-parsers-comparison/blob/5548b52f2cc27eb19c11464e9a331491e8ad4ba6/README.md#statistics-updated-28th-of-february-2018], so this may no longer be true.
Bug: Utilize BigDecimal to calculate the GPU resource 
Status: Open | Priority: Minor | Resolution: 
Created: 19/Feb/24 00:07
Affects Version/s: 4.0.0

To prevent precision errors, the current method of calculating GPU resources involves multiplying by 1E16 to convert doubles to Longs. If needed, it will also convert Longs back to doubles. This approach introduces redundancy in the code, especially for test code.

More details can be found at https://github.com/apache/spark/pull/44690#discussion_r1482301112
Bug: JDBCRDD does not correctly handle errors in getQueryOutputSchema
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Feb/24 23:23
Affects Version/s: 3.5.0

If there is an error executing statement.executeQuery(), it's possible that another error in one of the finally statements makes us not see the main error.
{code:java}
def getQueryOutputSchema(
      query: String, options: JDBCOptions, dialect: JdbcDialect): StructType = {
    val conn: Connection = dialect.createConnectionFactory(options)(-1)
    try {
      val statement = conn.prepareStatement(query)
      try {
        statement.setQueryTimeout(options.queryTimeout)
        val rs = statement.executeQuery()
        try {
          JdbcUtils.getSchema(rs, dialect, alwaysNullable = true,
            isTimestampNTZ = options.preferTimestampNTZ)
        } finally {
          rs.close()
        }
      } finally {
        statement.close()
      }
    } finally {
      conn.close()
    }
  } {code}
Bug: Wrong error message for incorrect ANSI intervals
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Feb/24 07:54
Affects Version/s: 3.4.2

When Spark SQL cannot recognise ANSI interval, it outputs wrong pattern for particular ANSI interval. For example, it cannot recognise year-month interval, but says about day-time interval:

{code:sql}
spark-sql (default)> select interval '-\t2-2\t' year to month;

Interval string does not match year-month format of `[+|-]d h`, `INTERVAL [+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: -	2-2	. (line 1, pos 16)

== SQL ==
select interval '-\t2-2\t' year to month
----------------^^^
{code}