av9ash/GitBugs huggingface dataset:
Bug: Parameterized queries fail for DESCRIBE & EXPLAIN w/ UNBOUND_SQL_PARAMETER error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/24 13:27
Affects Version/s: 4.0.0

*Succeeds:* scala> spark.sql("select ?", Array(1)).show();

*Fails:* spark.sql("describe select ?", Array(1)).show();

*Fails:* spark.sql("explain select ?", Array(1)).show();

Failures are of the form:

org.apache.spark.sql.catalyst.ExtendedAnalysisException: [UNBOUND_SQL_PARAMETER] Found the unbound parameter: _16. Please, fix `args` and provide a mapping of the parameter to either a SQL literal or collection constructor functions such as `map()`, `array()`, `struct()`. SQLSTATE: 42P02; line 1 pos 16; 'Project [unresolvedalias(posparameter(16))] +- OneRowRelation
Bug: Null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/24 00:08
Affects Version/s: 4.0.0

[INTERNAL_ERROR] Undefined error message parameter for error class: 'KAFKA_DATA_LOSS.COULD_NOT_READ_OFFSET_RANGE'

when groupId is null when we are about to throw KafkaException.couldNotReadOffsetRange error.

The error framework requires all params to be non-null.
Bug: try_to_timestamp should handle SparkUpgradeException
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 02/Apr/24 17:56
Affects Version/s: 3.4.3

Currently, try_to_timestamp will throw an exception on legacy timestamp input.
{code:java}
> SELECT try_to_timestamp('2016-12-1', 'yyyy-MM-dd')

org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0:
Fail to parse '2016-12-1' in the new parser.
You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid datetime string. SQLSTATE: 42K0B {code}
It should return null instead of error.
Bug: Add Java API Module for Spark Operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/24 06:50
Affects Version/s: kubernetes-operator-0.1.0

Spark Operator API refers to the [CustomResourceDefinition|https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/] __ that represents the spec for Spark Application in k8s.

This aims to add Java API library for Spark Operator, with the ability to generate yaml spec.
Bug: LIMIT can be improved
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/24 19:45
Affects Version/s: 3.5.1

Currently, LIMIT runs with a single task in GlobalLimit operator.  Many users decide to go around the problem by using SAMPLE as in: [https://towardsdatascience.com/stop-using-the-limit-clause-wrong-with-spark-646e328774f5]

 

Spark can be improved to:
1. Do a per-partition count;

2. Include K full partitions, plus 1 partial partitions.

While the 1 partial partition still requires a single task to run, the K full partitions would have dramatically reduced the N (of the LIMIT N) so it's a lot faster.
Bug: Avoid repeated calls to conf.resolver in resolveExpression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/24 16:41
Affects Version/s: 3.4.3

Consider a view with a large number of columns (~1000s). When resolving this view, looking at the flamegraph, observed repeated initializations of `conf` to obtain the `resolver` for each column of the view.

This can be easily optimized to reuse the same resolver (obtained once) for the various calls to `innerResolve` in `resolveExpression`.
Bug: Improve DeduplicateRelations performance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/24 11:26
Affects Version/s: 4.0.0

The `DeduplicateRelations` rule can be very slow on large plans due to the `missingInput` check on the whole plan, which was introduced by https://issues.apache.org/jira/browse/SPARK-37932 / https://github.com/apache/spark/pull/35684. Let's try to improve it.
Bug: Support recursive descent path in get_json_object function
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/24 11:36
Affects Version/s: 4.0.0

JSONPath borrows recursive descent syntax from E4X. We could use it to collect json object from json map string.
{code:java}
// json data
{"key1": {"b": {"c": "c1", "d": "d1", "e": "e1"}}}
{"key2": {"b": {"c": "c2", "d": "d2", "e": "e2"}}}

select get_json_object(data, '$..c'); -- [c1, c2]{code}
ref: https://goessner.net/articles/JsonPath/index.html#e2
Bug: Preserve case in Avro schema when using enableStableIdentifiersForUnionType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/24 22:37
Affects Version/s: 3.5.2

When enableStableIdentifiersForUnionType is enabled, all of the types are lowercased which creates a problem when field types are case-sensitive: 
{code:java}
Schema.createEnum("myENUM", "", null, List[String]("E1", "e2").asJava),
Schema.createRecord("myRecord2", "", null, false, List[Schema.Field](new Schema.Field("F", Schema.create(Type.FLOAT))).asJava){code}
 would become
{code:java}
struct<member_myenum: string, member_myrecord2: struct<f: float>>  {code}
but instead should be 
{code:java}
struct<member_myENUM: string, member_myRecord2: struct<F: float>>  {code}
Bug: Put removal of '!' as a synonym for 'NOT' on a keyword level under a config
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/24 04:29
Affects Version/s: 4.0.0

Recently we dissolved the lexer equivalence between '!' and 'NOT'.
! is a prefix operator and a synonym for NOT only in that case.
But NOT is used in many more cases in the grammar.
Given that there are a handful of known scenearios where users have exploited the undocumented loophole it's best to add a config.
Usage found so far is:

`c1 ! IN(1, 2)`
`c1 ! BETWEEN 1 AND 2`
`c1 ! LIKE 'a%'`

 But there are worse cases:
c1 IS ! NULL
CREATE TABLE T(c1 INT ! NULL)
or even
CREATE TABLE IF ! EXISTS T(c1 INT)
Bug: Upgrade built in hive to Hive-4.0
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/24 08:29
Affects Version/s: 3.5.1

Hive 4.0 has been released. It brings in a lot of new features, bug fixes and performance improvements. 

We would like to update the version of hive used in spark to hive-4.0

[https://lists.apache.org/thread/2jqpvsx8n801zb5pmlhb8f4zloq27p82]
Bug: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines
Status: Open | Priority: Major | Resolution: 
Created: 09/Apr/24 05:00
Affects Version/s: 4.0.0

SPIP doc: [https://docs.google.com/document/d/1v7sndtIHIBdzc4YvLPI8InXxhI7SnnAQ5HvmM2DGjVE/edit?usp=sharing]

 

Refined SPIP doc:

[https://docs.google.com/document/d/1oY26KtqXoJJNHbAhtmVgaXSVt6NlO6t1iWYEuGvCc1s/edit#heading=h.ud7930xhlsm6]

 

This [SPIP|https://docs.google.com/document/d/1v7sndtIHIBdzc4YvLPI8InXxhI7SnnAQ5HvmM2DGjVE/edit?usp=sharing] outlines the integration of Gluten's physical plan conversion, validation, and fallback framework into Apache Spark. The goal is to enhance Spark's flexibility and robustness in executing physical plans and to leverage Gluten's performance optimizations. Currently, Spark lacks an official cross-platform execution support for physical plans. Gluten's mechanism, which employs the Substrait standard, can convert and optimize Spark's physical plans, thus improving portability, interoperability, and execution efficiency.

The design proposal advocates for the incorporation of the TransformSupport interface and its specialized variants—LeafTransformSupport, UnaryTransformSupport, and BinaryTransformSupport. These are instrumental in streamlining the conversion of different operator types into a Substrait-based common format. The validation phase entails a thorough assessment of the Substrait plan against native backends to ensure compatibility. In instances where validation does not succeed, Spark's native operators will be deployed, with requisite transformations to adapt data formats accordingly. The proposal emphasizes the centrality of the plan transformation phase, positing it as the foundational step. The subsequent validation and fallback procedures are slated for consideration upon the successful establishment of the initial phase.

The integration of Gluten into Spark has already shown significant performance improvements with ClickHouse and Velox backends and has been successfully deployed in production by several customers.
Bug: Compute Current Time* expressions should be foldable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/24 15:50
Affects Version/s: 4.0.0

Following PR - https://github.com/apache/spark/pull/44261 changed "compute current time" family of expressions to be unevaluable, given that these expressions are supposed to be replaced with literals by QO. Unevaluable implies that these expressions are not foldable, even though they will be replaced by literals. 


If these expressions were used in places that require constant folding  (e.g. RAND()) new behavior would be to raise an error which is a regression comparing to behavior prior to spark 4.0.
Bug: Support column type in split function in scala and python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/24 23:42
Affects Version/s: 3.5.1

I have a use case to split a String typed column with different delimiters defined in other columns of the dataframe. SQL already supports this, but scala / python functions currently don't.

 

A hypothetical example to illustrate:
{code:java}
import org.apache.spark.sql.functions.{col, split}

val example = spark.createDataFrame(
    Seq(
      ("Doe, John", ", ", 2),
      ("Smith,Jane", ",", 2),
      ("Johnson", ",", 1)
    )
  )
  .toDF("name", "delim", "expected_parts_count")

example.createOrReplaceTempView("test_data")

// works for SQL
spark.sql("SELECT split(name, delim, expected_parts_count) AS name_parts FROM test_data").show()

// currently doesn't compile for scala, but easy to support
example.withColumn("name_parts", split(col("name"), col("delim"), col("expected_parts_count"))).show() {code}
 

Pretty simple patch that I can make a PR soon
Bug: Scala Spark Test Utils/Framework
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/24 01:11
Affects Version/s: 3.5.0

As users of Apache Spark in Scala, I've observed a gap in the availability of a dedicated testing framework similar to the one present for PySpark. While various open-source repositories and resources exist for PySpark testing also has Scala counterpart:
 * [spark-testing-base|[holdenk/spark-testing-base: Base classes to use when writing tests with Spark (github.com)|https://github.com/holdenk/spark-testing-base]]
 * [spark-fast-test|[MrPowers/spark-fast-tests: Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit) (github.com)|https://github.com/MrPowers/spark-fast-tests]]

 

The same level of support is lacking for Spark Scala. For example, spark-fast-test was last updated 2 years ago and still not supporting newer spark version. This cause some pain when I am trying to do Integration and performance testing on our cluster where we provide Fat Test Jar with provided Spark Jar of newer version.

I propose the development of an official testing utils tailored specifically for Spark Scala with similar if not the same feature providing by PySpark Test utils provided since 3.5.0. These utilities would empower Scala developers with the tools and capabilities necessary to write robust and reliable tests for their Spark applications.

 

Key Features to Include:
 * Utility Functions: Provide a suite of utility functions designed to simplify common testing tasks, such as assert_dataframe_equality and assert_schema_equality, thereby reducing boilerplate code and accelerating test development. In the Future this would also help us implement, consolidate the best practices for comparing two dataframes correctly and efficiently with automated testing and benchmarks. 
 * Comprehensive Documentation: Offer comprehensive documentation and examples to guide users in effectively utilizing the testing framework/utils, ensuring ease of adoption and accelerating the learning curve.

I am happy to contribute to this feature and would welcome any help. Also Let me know if This would require a SPIP
Bug: Remove unused import `spark/connect/common.proto` from `spark/connect/relations.proto`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/24 12:44
Affects Version/s: 4.0.0

fix compile waring:

 
{code:java}
spark/connect/relations.proto:26:1: warning: Import spark/connect/common.proto is unused. {code}
Bug: Parent classes for Spark Connect and Spark Classic
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/24 04:41
Affects Version/s: 4.0.0

For now, the operations like {{DataFrame.union(df1, df2)}} or {{isinstance(df, pyspark.sql.DataFrame}} does not work.

We should have a parent class to make those working so minimize the behaviour difference
Bug: Improve docstring of mapInPandas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/24 21:12
Affects Version/s: 4.0.0

Improve docstring of mapInPandas
 * "using a Python native function that takes and outputs a pandas DataFrame" is confusing cause the function takes and outputs "ITERATOR of pandas DataFrames" instead.
 * "All columns are passed together as an iterator of pandas DataFrames" easily mislead users to think the entire DataFrame will be passed together, "a batch of rows" is used instead.
Bug: Show offset value in TakeOrderedAndProjectExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/24 16:24
Affects Version/s: 3.4.0

Show the offset value in TakeOrderedAndProjectExec.

 

For example:

 

explain select * from test_limit_offset order by a  limit 2  offset 1;

plan

== Physical Plan ==

TakeOrderedAndProject(limit=3, orderBy=[a#171 ASC NULLS FIRST|#171 ASC NULLS FIRST], output=[a#171|#171])

+- Scan hive spark_catalog.bigdata_qa.test_limit_offset [a#171|#171], HiveTableRelation [`spark_catalog`.`test`.`test_limit_offset`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, Data Cols: [a#171|#171], Partition Cols: []]

 

No offset is displayed. If it is displayed, it will be more user-friendly
Bug: Replace equivalent expression to <=> in join condition for 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/24 07:01
Affects Version/s: 4.0.0

Replaces `t1.id is null and t2.id is null or t1.id = t2.id` to `t1.id <=> t2.id` in join condition for better performance.
Bug: Fix Aggregate bug in RewriteWithExpression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/24 23:39
Affects Version/s: 4.0.0

The following query will fail:

{code:SQL}
SELECT NULLIF(id + 1, 1)
from range(10)
group by id
{code}

This is because {{NullIf}} gets rewritten to {{With}}, then {{RewriteWithExpression}} tries to pull common expression {{id + 1}} out of the aggregate, resulting in an invalid plan.
Bug: Spark supporting Parquet V2
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/24 18:04
Affects Version/s: 3.2.0

Parquet V2 encoding made the parquet optimized. Read and write is now faster than Parquet V1.  We are using Dremio for reading and writing Parquet . Now due to Parquet V2 encoding read is 75% faster and write is 25% faster.

 

Apache Parquet-MR Writer version PARQUET_2_0, which is widely adopted by engines that write Parquet data, supports delta encodings. However, these encodings were not previously supported by Dremio's vectorized Parquet reader, resulting in decreased speed. Now, in version 24.3 and Dremio Cloud, when you use the Dremio SQL query engine on Parquet datasets, you’ll receive best-in-class performance.

[https://www.dremio.com/blog/vectorized-reading-of-parquet-v2-improves-performance-up-to-75/]
Bug: endsWith and startsWith don't work correctly for some collations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/24 21:40
Affects Version/s: 4.0.0

*CollationSupport.EndsWIth* and *CollationSupport.StartsWith* use {*}CollationAwareUTF8String.matchAt{*}, which operates byte offsets to compare prefixes/suffixes. This is not correct, since sometimes string parts (suffix/prefix) of different lengths are actually equal in context of case-insensitive and lower-case collations.

Example test cases that highlight the problem:

{{{}- *assertContains("The İo", "i̇o", "UNICODE_CI", true);* for *CollationSupportSuite.*{}}}{{{}{*}testContains{*}.{}}} 
{{{}- *assertEndsWith("The İo", "i̇o", "UNICODE_CI", true);* for *CollationSupportSuite.*{}}}{{{}{*}testEndsWith{*}.{}}}

{{The first passes, since it uses *StringSearch* directly, the second one does not.}}
Bug: Not working HDFS path for hive.metastore.jars.path
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/24 03:25
Affects Version/s: 3.4.2

I trying to use Hive Metastore version 3.1.3 with Spark version 3.4.2, but encountering an error when specifying the path to the metastore JARs on HDFS.

According to the official documentation, followed the guidelines and specified the path using an HDFS URI:
{code:java}
spark.sql.hive.metastore.version     3.1.3
spark.sql.hive.metastore.jars        path
spark.sql.hive.metastore.jars.path   hdfs://namespace/spark/hive3_lib/* {code}
However, when tested it, encountered an error stating that the URI schema in HiveClientImpl.scala is not file.
{code:java}
Caused by: java.lang.ExceptionInInitializerError: java.lang.IllegalArgumentException: URI scheme is not "file"
  at org.apache.spark.sql.hive.client.HiveClientImpl$.newHiveConf(HiveClientImpl.scala:1296)
  at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:174)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:139)
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:315)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:517)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:377)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:70)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:69)
  at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
  at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
  ... 143 more {code}
To resolve this, changed the spark.sql.hive.metastore.jars.path to a local file path instead of an HDFS path, and it worked fine. I think I followed the instructions correctly, but are there any specific configurations or preferences required to use HDFS paths?
Bug: Multiple bugs with map operations in combination with collations
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/24 15:38
Affects Version/s: 4.0.0

Following two queries produce different results (first succeeds, second throws an exceptions):
{code:java}
select map('a', 1, 'A' collate utf8_binary_lcase, 2); -- success
select map('a' collate utf8_binary_lcase, 1, 'A', 2); -- exception{code}
Following query results in 1:
{code:java}
select cast(map('a', 1, 'A', 2) as map<string collate utf8_binary_lcase, string>)['A' collate utf8_binary_lcase]; -- 1{code}
Bug: Avro does not register custom logical types on spark startup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/24 14:27
Affects Version/s: 4.0.0

It can happen that we resolve avro schema before we register avro logical types leading to two consecutive calls to resolve avro schema providing wrong results.

 

Example:

!image-2024-04-05-16-27-05-489.png!
Bug: Why does javaRDD().mapPartitions lead to the memory leak in this case?
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/24 19:16
Affects Version/s: 3.3.2

Hello Spark community. I have an Java Spark Structured Streaming application:
Unless I am doing silly mistake, the JedisCluster closed in the finally block, but still some memory leak. 
{code:java}
FlatMapFunction<Iterator<Row>, Row> myFunction = new MyFunction(jedisConfiguration);
StructType  structSchema = getSchema();

VoidFunction2<Dataset<Row>, Long> forEachFunc = (dataset, aLong) -> {
        Dataset<Row> dataset = getDataset();
        dataset.persist();
        JavaRDD<Row> processedRDD = dataset.javaRDD().mapPartitions(myFunction);
        Dataset<Row> processedDS = sparkSession().createDataFrame(processedRDD, structSchema);
        parquetWriter.write(processedDS);
        dataset.unpersist();
    };

DataStreamWriter<Row> dataStream = dataset
.writeStream()
.foreachBatch(forEachFunc)
.outputMode(outputMode)
.option("checkpointLocation", checkpointLocation);

....<stream dataStream> {code}
And the function
{code:java}
public class MyFunction implements FlatMapFunction<Iterator<Row>, Row> {

   <constructor with jedisConfiguration parameter>...

    @Override
    public Iterator<Row> call(Iterator<Row> rowIterator) throws Exception {

        List<Row> output;
        JedisCluster redis = new JedisCluster(jedisConfiguration);

        try {
            output = new ArrayList<>();

            while (rowIterator.hasNext()) {
                Row row = rowIterator.next();
                Long var1 = row.getAs("var1");
                Long var2 = row.getAs("var2");

                var redisKey = "some_key";
                var result = redis.hgetAll(redisKey);

                if (!result.isEmpty()) {
                    output.add(RowFactory.create(
                            var1,
                            var2,
                            result.getOrDefault("some_id", null)));
                }
            }
        } finally {
            if (redis != null) {
                try {
                    redis.close();
                } catch (Exception e) {
                    throw new RuntimeException("Failed to close Redis connection: " + e);
                }
            }
        }
        return output.iterator();
    }
} {code}
It actually works couple of days then dies. Can't figure out what does cause memory leak in the Driver?

Tested with Spark 3.3.2 and 3.5.0

Grafana board of the Driver's Memory Pool 
!Screenshot 2024-04-15 at 20.43.22.png|width=875,height=169!