av9ash/GitBugs huggingface dataset:
Bug: Upgrade to Parquet 1.14
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/May/24 11:12
Affects Version/s: 3.5.1

There's a bug in Parquet [(PARQUET-2454)|https://issues.apache.org/jira/browse/PARQUET-2454] where Parquet in Spark occasionally writes out truncated files with bytes missing at the end.

The fix was released in Parquet 1.14.0. [See changelog.|https://github.com/apache/parquet-mr/blob/master/CHANGES.md#version-1140]
Bug: Fix the python streaming data source timeout issue for large trigger interval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/24 19:41
Affects Version/s: 4.0.0

Currently we run long running python worker process for python streaming source and sink to perform planning, commit and abort in driver side. Testing indicate that current implementation cause connection timeout error when streaming query has large trigger interval

For python streaming source, keep the long running worker archaetecture but set the socket timeout to be infinity to avoid timeout error.

For python streaming sink, since StreamingWrite is also created per microbatch in scala side, long running worker cannot be attached to s StreamingWrite instance. Therefore we abandon the long running worker architecture, simply call commit() or abort() and exit the worker and allow spark to reuse worker for us.
Bug: Add CollectSetWIthLimit
Status: Open | Priority: Major | Resolution: 
Created: 20/May/24 21:13
Affects Version/s: 4.0.0

See [https://stackoverflow.com/questions/38730912/how-to-limit-functions-collect-set-in-spark-sql]

 

Some users want to collect a set but if the number of distinct elements is too large they may get a Cannot grow BufferHolder  error from trying to collect the set then trim it.

 

We should offer a collect set which pre-emptively does not add more elements than needed to reduce the amount of memory used.
Bug: AQE not working when joining dataframes with more than 2000 partitions
Status: Open | Priority: Major | Resolution: 
Created: 15/May/24 15:28
Affects Version/s: 3.3.2

We are joining 2 large dataframes with a considerable skew on the left side in one specific key (>2000 skew ratio).
{code:java}
left side num partitions: 10335
right side num partitions: 1241

left side num rows: 20181947343
right side num rows: 107462219 {code}
Since we have `{{{}spark.sql.adaptive.enabled{}}} ` we expect AQE to act during the join, dealing with the skewed partition automatically.

During their join, we can see the following log indicating that the skew was not detected since their statistics looks weirdly equal for min/median/max sizes:
{code:java}
OptimizeSkewedJoin: number of skewed partitions: left 0, right 0
 OptimizeSkewedJoin: 
Optimizing skewed join.
Left side partitions size info:
median size: 780925482, max size: 780925482, min size: 780925482, avg size: 780925482
Right side partitions size info:
median size: 3325797, max size: 3325797, min size: 3325797, avg size: 3325797
       {code}
Looking at this log line and the spark configuration possibilities, our two main hypotheses to work around this behavior and correctly detect the skew were:
 # Increasing the `minNumPartitionsToHighlyCompress` so that Spark doesn’t convert the statistics into a `CompressedMapStatus` and therefore is able to identify the skewed partition.
 # Allowing spark to use a `HighlyCompressedMapStatus`, but change other configurations such as `spark.shuffle.accurateBlockThreshold` and `spark.shuffle.accurateBlockSkewedFactor` so that even then the size of the skewed partitions/blocks is accurately registered and consequently used in the optimization.

We tried different values for `spark.shuffle.accurateBlockThreshold` (even absurd ones like 1MB) and nothing seem to work. The statistics indicates that the min/median and max are the same somehow and thus, the skew is not detected.

However, when forcibly reducing `spark.sql.shuffle.partitions` to less than 2000 partitions, the statistics looked correct and the optimized skewed join acts as it should:
{code:java}
OptimizeSkewedJoin: number of skewed partitions: left 1, right 0
OptimizeSkewedJoin: Left side partition 42 (263 GB) is skewed, split it into 337 parts.
OptimizeSkewedJoin: 
Optimizing skewed join.
Left side partitions size info:
median size: 862803419, max size: 282616632301, min size: 842320875, avg size: 1019367139
Right side partitions size info:
median size: 4320067, max size: 4376957, min size: 4248989, avg size: 4319766 {code}
Should we assume that the statistics are becoming corrupted when Spark uses `HighlyCompressedMapStatus`? Should we try another configuration property to try to work around this problem? (Assuming that fine tuning all dataframes in skewed joins in our ETL to have less than 2000 partitions is not an option)
Bug: Drop spark.sql.datetime.java8API
Status: Open | Priority: Major | Resolution: 
Created: 18/May/24 20:23
Affects Version/s: 4.0.0

The default value is *_enabled_* 

Dropping the old API in Spark 4.0.0 with other breaking changes would be prudent.

Supporting only the java.time API enables simplifications and future features like nanosecond timestamps without introducing a breaking change or complex behavior.
Bug: Make nestedTypeMissingElementTypeError case insensitive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/24 19:44
Affects Version/s: 3.5.0

When incorrectly declaring a complex data type using nested types (ARRAY, MAP and STRUCT), the query fails with a match error rather than `INCOMPLETE_TYPE_DEFINITION`. This is because the match is case sensitive,
Bug: Improve Alias.removeNonInheritableMetadata performance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/24 15:30
Affects Version/s: 4.0.0

`Alias.removeNonInheritableMetadata(...)` uses `MetadataBuilder`, which does expensive map manipulation (allocation, ++, remove). When `removeNonInheritableMetadata` is frequently called (e.g. wide VIEWs with many Aliases), it significantly slows down the code.
Bug: Rename Java Logger as SparkLogger 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/24 18:14
Affects Version/s: 4.0.0

Two new classes org.apache.spark.internal.Logger and org.apache.spark.internal.LoggerFactory were introduced from [https://github.com/apache/spark/pull/46301|https://github.com/apache/spark/pull/46301,]

Given that {{Logger}} is a widely recognized interface in Log4j, it may lead to confusion to have a class with the same name. To avoid this and clarify its purpose within the Spark framework, I propose renaming {{org.apache.spark.internal.Logger}} to {{{}org.apache.spark.internal.SparkLogger{}}}. Similarly, to maintain consistency, {{org.apache.spark.internal.LoggerFactory}} should be renamed to {{{}org.apache.spark.internal.SparkLoggerFactory{}}}.
Bug: Add source data type to connector.Cast expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/24 10:19
Affects Version/s: 4.0.0

Currently, 
V2ExpressionBuilder will build connector.Cast expression from catalyst.Cast expression.
Catalyst cast have expression data type, but connector cast does not have it.
Since some casts are not allowed on external engine, we need to know source and target data type, since we want finer granularity to block some unsupported casts.
Bug: [SQL][XML] Fix issues in skipping nested structure fields not selected in schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 22:14
Affects Version/s: 4.0.0

Previously, the XML parser can't skip nested structure data fields effectively when they were not selected in the schema. For instance, in the below example, `df.select("struct2").collect()` returns `Seq(null)` as `struct1` wasn't effectively skipped. This PR fixes this issue.
{code:java}
<ROW>
  <struct1>
    <innerStruct><field1>1</field1></innerStruct>
  </struct1>
  <struct2>
    <field2>2</field2>
  </struct2>
</ROW>{code}
Bug: Collate function support for non UTF8_BINARY strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/24 16:21
Affects Version/s: 4.0.0

When default collation level config is set to some collation other than UTF8_BINARY (i.e. UTF8_BINARY_LCASE) and when we try to execute COLLATE (or collation) expression, this will fail because it is only accepting StringType(0) as argument for collation name.
Bug: Regression e2e test with SPARK-47305
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/24 03:33
Affects Version/s: 4.0.0

In SPARK-47305, we fixed a bug in QO rule and added a test to demonstrate the issue and verify the fix works, but the scope of test was just to bugfix itself because we had no clear idea of (simpler) reproducer for e2e example.

We finally came up with simple reproducer which is e2e streaming query. We'd like to put this reproducer into test as regression test.
Bug: CSV expressions (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/24 04:06
Affects Version/s: 4.0.0

Enable collation support for *CSV* built-in string functions in Spark ({*}CsvToStructs{*}, {*}SchemaOfCsv{*}, {*}StructsToCsv{*}). First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. You will find these expressions in the *csvExpressions.scala* file, and they should mostly be pass-through functions. Implement the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *CSV* expressions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for string [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Fix error with aggregate function in With child
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/24 19:15
Affects Version/s: 4.0.0

Right now, if we have an aggregate function in the child of a With expression, we fail an assertion. However, queries like this used to work:

{code:sql}
select
    id between cast(max(id between 1 and 2) as int) and id
from range(10)
group by id
{code}
Bug: XML expressions (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/24 04:06
Affects Version/s: 4.0.0

Enable collation support for *XML* built-in string functions in Spark ({*}XmlToStructs{*}, {*}SchemaOfXml{*}, {*}StructsToXml{*}). First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. You will find these expressions in the *xmlExpressions.scala* file, and they should mostly be pass-through functions. Implement the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *XML* expressions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for string [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Reduce GitHub Action usage according to ASF project allowance
Status: Reopened | Priority: Major | Resolution: 
Created: 02/May/24 20:14
Affects Version/s: 4.0.0

h2. ASF INFRA POLICY
 - [https://infra.apache.org/github-actions-policy.html]

h2. MONITORING
 - [https://infra-reports.apache.org/#ghactions&project=spark&hours=168]

!Screenshot 2024-05-02 at 23.56.05.png|width=100%!

h2. TARGET
 * All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices.
 * All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20.
 * The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
 * The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours).

h2. DEADLINE
{quote}17th of May, 2024
{quote}
Bug: Analyzer bug with multiple ORDER BY items for input table argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/24 21:54
Affects Version/s: 3.5.0

Steps to reproduce:

 

{{from pyspark.sql.functions import udtf}}

{{@udtf(returnType="a: int, b: int")}}
{{class tvf:}}
{{  def eval(self, *args):}}
{{    yield 1, 2}}

 

{{SELECT * FROM tvf(}}
{{  TABLE(}}
{{    SELECT 1 AS device_id, 2 AS data_ds}}
{{    )}}
{{    WITH SINGLE PARTITION}}
{{    ORDER BY device_id, data_ds}}
{{ )}}


{{[UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.UNSUPPORTED_TABLE_ARGUMENT] Unsupported subquery expression: Table arguments are used in a function where they are not supported:}}
{{'UnresolvedTableValuedFunction [tvf], [table-argument#338 [], 'data_ds], false}}
{{   +- Project [1 AS device_id#336, 2 AS data_ds#337]}}
{{      +- OneRowRelation}}
Bug:  Recover compatibility with Spark Connect client 3.5 <> Spark Connect server 4.0
Status: Resolved | Priority: Major | Resolution: Done
Created: 02/May/24 09:32
Affects Version/s: 4.0.0

https://github.com/apache/spark/pull/46298#issuecomment-2087905857

There are test failures identified when you run Spark 3.5 Spark Connect client <> Spark Connect server 4.0.

They should ideally be compatible.
Bug: PrometheusServlet does not work well with queryName in spark streaming metric names
Status: Open | Priority: Major | Resolution: 
Created: 09/May/24 18:51
Affects Version/s: 3.3.4

PrometheusServlet does not work well with queryName in the metric name.

For driver metrics, the configuration `spark.metrics.namespace` allows the user to remove the app id from the metric name to build reusable dashboards and metric queries in external systems.

Similarly, we need a feature to replace the query name in the spark streaming metric names to make dashboards and metric queries reusable. Ideally, the query name would be generated as a tag, instead of part of the metric name.

This can be enabled if a feature flag is enabled, to not break existing users who are dependent of the existing metric format.
Bug: Always set the seed of dataframe.sample in Client side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/24 03:16
Affects Version/s: 3.4.3

the output dataframe of `sample` is not immutable in Spark Connect

 

In Spark Classic:
{code:java}
In [1]: df = spark.range(10000).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{code}
 

In Spark Connect:
{code:java}
In [1]: df = spark.range(10000).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]
 {code}
Bug: Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/24 11:54
Affects Version/s: 4.0.0

{code:java}
/**
 * <p>AbstractLogger constructor.</p>
 *
 * <p><strong>Important:</strong> the log level set within
 * the {@link org.rocksdb.Options} instance will be used as
 * maximum log level of RocksDB.</p>
 *
 * @param options {@link org.rocksdb.Options} instance.
 *
 * @deprecated Use {@link Logger#Logger(InfoLogLevel)} instead, e.g. {@code new
 *     Logger(options.infoLogLevel())}.
 */
@Deprecated
public Logger(final Options options) {
  this(options.infoLogLevel());
} {code}
Bug: Unify StreamingPythonRunner and PythonPlannerRunner
Status: Open | Priority: Major | Resolution: 
Created: 07/May/24 22:26
Affects Version/s: 4.0.0

We should unify the two driver side python runner for PySpark. To do this we should move out of StreamingPythonRunner and enhance PythonPlannerRunner with streaming support (multiple read - write loop)
Bug: Can not alter bucketed table if create table with upper case schema
Status: Open | Priority: Major | Resolution: 
Created: 06/May/24 04:19
Affects Version/s: 3.5.0

Running below SQL command throws exception

 
CREATE TABLE TEST1(
V1 BIGINT,
S1 INT)
PARTITIONED BY (PK BIGINT)
CLUSTERED BY (V1)
SORTED BY (S1)
INTO 200 BUCKETS
STORED AS PARQUET;

ALTER TABLE test1 SET TBLPROPERTIES ('comment' = 'This is a new comment.');

*Exception:*
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns V1 is not part of the table columns ([FieldSchema(name:v1, type:bigint, comment:null), FieldSchema(name:s1, type:int, comment:null)]
        at org.apache.hadoop.hive.ql.metadata.Table.setBucketCols(Table.java:552)
        at org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1145)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$alterTable$1(HiveClientImpl.scala:594)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:303)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:234)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:233)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:283)
        at org.apache.spark.sql.hive.client.HiveClientImpl.alterTable(HiveClientImpl.scala:587)
        at org.apache.spark.sql.hive.client.HiveClient.alterTable(HiveClient.scala:124)
        at org.apache.spark.sql.hive.client.HiveClient.alterTable$(HiveClient.scala:123)
        at org.apache.spark.sql.hive.client.HiveClientImpl.alterTable(HiveClientImpl.scala:93)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$alterTable$1(HiveExternalCatalog.scala:687)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
        ... 62 more
{code}
Bug: StateStore schema incompatibility between 3.2 and 3.4
Status: Open | Priority: Major | Resolution: 
Created: 01/May/24 16:18
Affects Version/s: 3.2.4

One our customer encountered some schema incompatibility problems when upgrading from Spark 3.2 to 3.4 with structured streaming application.

It seems in 3.4 `Encoders.bean()` includes properties with only getter with or without setter, whereas in 3.2, only properties with both getter and setter are included.

For example, here are schemas for an AtomicLong property/field generated by each version:
3.2: StructType(StructField(opaque,LongType,true),StructField(plain,LongType,true))
3.4: StructType(StructField(acquire,LongType,false),StructField(andDecrement,LongType,false),StructField(andIncrement,LongType,false),StructField(opaque,LongType,false),StructField(plain,LongType,false))

Note that the null ability flag also changes.
Primitive long schema has nullable=true in 3.2, but false in 3.4.

I am not sure if the issue is aware by the community before, and if there is workaround for that?

Thanks.
Bug: Disable structured logging for Interpreter by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/24 01:05
Affects Version/s: 4.0.0

Since there are plain text output from Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable structured logging for Interpreters by default.

 

spark-shell output when with structured logging enabled:

```

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT

      /_/

         

Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)

Type in expressions to have them evaluated.

Type :help for more information.

{"ts":"2024-05-04T01:11:03.797Z","level":"WARN","msg":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable","logger":"NativeCodeLoader"}

{"ts":"2024-05-04T01:11:04.104Z","level":"WARN","msg":"Service 'SparkUI' could not bind on port 4040. Attempting port 4041.","logger":"Utils"}

Spark context Web UI available at http://10.10.114.155:4041

Spark context available as 'sc' (master = local[*], app id = local-1714785064155).

Spark session available as 'spark'.

```

 

spark-shell output when without structured logging enabled:

```

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT

      /_/

         

Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)

Type in expressions to have them evaluated.

Type :help for more information.

24/05/03 18:11:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

24/05/03 18:11:35 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

Spark context Web UI available at http://10.10.114.155:4041

Spark context available as 'sc' (master = local[*], app id = local-1714785095892).

Spark session available as 'spark'.

```