av9ash/GitBugs huggingface dataset:
Bug: Override toString in LateralColumnAliasReference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/23 05:58
Affects Version/s: 3.5.0


{code:sql}
select id + 1 as a1, a1 + 2 as a3 from range(10);
{code}

Before:
{noformat}
Project [(id#2L + 1) AS a1#0, (lateralAliasReference('a1, a1, 'a1) + 2) AS a3#1]
+- Range (0, 10, step=1, splits=None)
{noformat}

After:
{noformat}
Project [(id#2L + 1) AS a1#0, (lateralAliasReference(a1) + 2) AS a3#1]
+- Range (0, 10, step=1, splits=None)
{noformat}
Bug: Reduce function doesn't infer Decimal type precision correctly
Status: Open | Priority: Major | Resolution: 
Created: 30/May/23 18:47
Affects Version/s: 3.4.0

A working reduce function example

 
{code:java}
select reduce(arrays, array(0, 0), (acc,x) -> array(get(acc,0)+get(x,0), get(acc,1)+get(x,1))) from (select array(array(1, 2), array(1000, 2000)) as arrays){code}
 

produces 
{code:java}
[1001,2002]{code}
 

But, if you introduce decimals, it breaks:
{code:java}
select reduce(arrays, array(0, 0), (acc,x) -> array(get(acc,0)+get(x,0), get(acc,1)+get(x,1))) from (select array(array(1, 2), array(1000.00, 2000)) as arrays); {code}
produces:
{code:java}
[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "reduce(arrays, array(0, 0), lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable()), lambdafunction(namedlambdavariable(), namedlambdavariable()))" due to data type mismatch: Parameter 3 requires the "ARRAY<INT>" type, however "lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable())" has the type "ARRAY<DECIMAL(13,2)>".; line 1 pos 0; Project [reduce(arrays#6, array(0, 0), lambdafunction(array((cast(get(lambda acc#7, 0) as decimal(10,0)) + get(lambda x#8, 0)), (cast(get(lambda acc#7, 1) as decimal(10,0)) + get(lambda x#8, 1))), lambda acc#7, lambda x#8, false), lambdafunction(lambda id#9, lambda id#9, false)) AS reduce(arrays, array(0, 0), lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable()), lambdafunction(namedlambdavariable(), namedlambdavariable()))#10] +- SubqueryAlias __auto_generated_subquery_name    +- Project [array(cast(array(1, 2) as array<decimal(12,2)>), array(cast(1000.00 as decimal(12,2)), cast(2000 as decimal(12,2)))) AS arrays#6]       +- OneRowRelation
 {code}
We've reproduced the issue even when trying to specify the decimal cast of all inputs in the array and the types in the accumulator.
Bug: Expression lineage not handling Window and AggregatePart operators
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 12/May/23 00:41
Affects Version/s: 3.3.2

Expression lineage logic (findExpressionAndTrackLineageDown) has missing handling for partial aggregation (AggregatePart) and Window operators. This is prohibiting optimizations such as bloom filters which depends on expression lineage
Bug: [BUG] MySQL jdbc cursor has not taken effect
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 24/May/23 15:44
Affects Version/s: 3.4.1

JDBCRDD.scala:  stmt.setFetchSize(options.fetchSize)

 
JdbcRDD.scala:
{code:java}
if (url.startsWith("jdbc:mysql:")){ 
// setFetchSize(Integer.MIN_VALUE) is a mysql driver specific way to force
// streaming results, rather than pulling entire resultset into memory.
// See the below URL
// dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html
  stmt.setFetchSize(Integer.MIN_VALUE) }
else
{ stmt.setFetchSize(100) }
{code}
Bug: Latest docker Spark image has critical CVE
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/May/23 18:44
Affects Version/s: 3.4.0

I pulled the latest spark 3.4.0 image from dockerhub, on 2023-04-28 and found after scanning on docker desktop that there are several critical CVE found (see screenshot).

It seems like some changes to github actions are needed to rebuild with updated dependencies on a regular cadence.

 

Notes:

Original project issue: https://issues.apache.org/jira/browse/SPARK-40513

[https://hub.docker.com/r/apache/spark/tags]

https://github.com/apache/spark-docker/actions
Bug: Web UI WholeStageCodegen duration is 0 ms
Status: Open | Priority: Major | Resolution: 
Created: 29/May/23 02:52
Affects Version/s: 3.2.0

In the Web UI when I’m running a query with limit clause, the duration of WholeStageCodegen operator is always 0 ms.
the corresponding feature is SPARK-13916, commitid: 76958d820f57d23e3cbb5b7205c680a5daea0499 . durationMs update only when we iterate the last row of partition, but when we only iterate a few rows, the duration will always be 0 ms

below  code will repetition the issue:

spark.sql("use tpcds1g")
spark.sql("""
select i_item_sk from item
limit 100
""").collect
Bug: Protobuf: Throw structured error when reading descriptor file in Python fails
Status: Open | Priority: Major | Resolution: 
Created: 29/May/23 02:26
Affects Version/s: 3.5.0

`_read_descriptor_set_file()` in `protobuf.functions` reads binary descriptor file from a file. It should throw structured Spark SQL error like _PROTOBUF_DESCRIPTOR_FILE_NOT_FOUND_ when the file is missing. It currently throws native Python error.
Bug: Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/May/23 21:37
Affects Version/s: 3.5.0

I launched spark-shell as so:
{noformat}
bin/spark-shell --driver-memory 8g --jars `find . -name "spark-avro*.jar" | grep -v test | head -1`
{noformat}
I got the below error trying to create an AVRO file:
{noformat}
scala> val df = Seq((1, 2), (3, 4)).toDF("a", "b")
val df = Seq((1, 2), (3, 4)).toDF("a", "b")
val df: org.apache.spark.sql.DataFrame = [a: int, b: int]

scala> df.write.mode("overwrite").format("avro").save("avro_file")
df.write.mode("overwrite").format("avro").save("avro_file")
java.lang.NoClassDefFoundError: scala/collection/immutable/StringOps
  at org.apache.spark.sql.avro.AvroFileFormat.supportFieldName(AvroFileFormat.scala:160)
  at org.apache.spark.sql.execution.datasources.DataSourceUtils$.$anonfun$checkFieldNames$1(DataSourceUtils.scala:75)
  at org.apache.spark.sql.execution.datasources.DataSourceUtils$.$anonfun$checkFieldNames$1$adapted(DataSourceUtils.scala:74)
  at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
  at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
  at org.apache.spark.sql.types.StructType.foreach(StructType.scala:105)
  at org.apache.spark.sql.execution.datasources.DataSourceUtils$.checkFieldNames(DataSourceUtils.scala:74)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:120)
...
scala> 
{noformat}
Bug: Upgrade cyclonedx-maven-plugin to 2.7.9
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/May/23 16:00
Affects Version/s: 3.5.0


https://github.com/CycloneDX/cyclonedx-maven-plugin/issues/326

{noformat}
Error:  Failed to execute goal org.cyclonedx:cyclonedx-maven-plugin:2.7.6:makeBom (default) on project spark-tags_2.12: Execution default of goal org.cyclonedx:cyclonedx-maven-plugin:2.7.6:makeBom failed: Unsupported class file major version 64 -> [Help 1]
{noformat}
Bug: Protobuf: Read descriptor file only once at the compile time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/23 22:55
Affects Version/s: 3.5.0

Protobuf functions read from the descriptor file many time (e.g. at each executor). This is unncessary and error prone (e.g. what if the contents change couple of days after the streaming query starts?).

 

It only needs to be read once.
Bug: unbase64 and unhex codegen are invalid with failOnError
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 21:45
Affects Version/s: 3.4.0

to_binary with hex and base64 generate invalid codegen:

{{spark.range(5).selectExpr('to_binary(base64(cast(id as binary)), "BASE64")').show()}}

results in

{{Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 1: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 1: Unknown variable or type "BASE64"}}

because this is the generated code:



/* 107 */         if (!org.apache.spark.sql.catalyst.expressions.UnBase64.isValidBase64(project_value_1)) {

/* 108 */           throw QueryExecutionErrors.invalidInputInConversionError(

/* 109 */             ((org.apache.spark.sql.types.BinaryType$) references[1] /* to */),

/* 110 */             project_value_1,

/* 111 */             BASE64,

/* 112 */             "try_to_binary");

/* 113 */         }
Bug: Spark Driver Bind Address is off-by-one
Status: Open | Priority: Major | Resolution: 
Created: 04/May/23 02:47
Affects Version/s: 3.3.3

I have the following environment variable set in my driver pod configuration:


{code:java}
SPARK_DRIVER_BIND_ADDRESS=10.244.0.53{code}
However, I see an off-by-one IP address being referred to in the Spark logs:


{code:java}
23/05/04 02:37:03 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.0.54:53140) with ID 1,  ResourceProfileId 0
23/05/04 02:37:03 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.0.54:32805 with 413.9 MiB RAM, BlockManagerId(1, 10.244.0.54, 32805, None){code}

I am not sure why this might be the case.
Bug: Maven test failed in ClientE2ETestSuite/CatalogSuite/StreamingQuerySuite without -Phive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 07:28
Affects Version/s: 3.5.0

 
{code:java}
build/mvn clean install -DskipTests
build/mvn test -pl connector/connect/client/jvm{code}
 

13 test failed with similar reasons:

 
{code:java}
- read and write *** FAILED ***
  io.grpc.StatusRuntimeException: INTERNAL: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.hive.execution.HiveFileFormat could not be instantiated
  at io.grpc.Status.asRuntimeException(Status.java:535)
  at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
  at scala.collection.Iterator.toStream(Iterator.scala:1417)
  at scala.collection.Iterator.toStream$(Iterator.scala:1416)
  at scala.collection.AbstractIterator.toStream(Iterator.scala:1431)
  at scala.collection.TraversableOnce.toSeq(TraversableOnce.scala:354)
  at scala.collection.TraversableOnce.toSeq$(TraversableOnce.scala:354)
  at scala.collection.AbstractIterator.toSeq(Iterator.scala:1431)
  at org.apache.spark.sql.SparkSession.execute(SparkSession.scala:489)
  ... {code}
Bug: Recover sh support in Dockerfiles 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/23 02:48
Affects Version/s: 3.5.0

rm /bin/sh && ln -sv /bin/bash /bin/sh

Does spark expect /bin/sh to be bash (in posix mode) and not just a generic posix compliant shell? Is there a bug with using dash? Minimally, this should do a dpkg --divert to ensure that users in dependent images don't accidentally revert it with package updates.

[1] https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388
[2] related origina change: https://github.com/apache-spark-on-k8s/spark/pull/444/files
Bug: Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 09:22
Affects Version/s: 3.5.0

env | grep SPARK_JAVA_OPT_ |...

This is susceptible to a few bugs particularly around newlines in values. I see two ways around it.
ensuring the matching name is at the start, -E '^SPARK_JAVA_OPT_', and running all the commands with null-terminated input and output: env -0 and -z on the other commands
or bash variable prefix expansion

for v in "${!SPARK_JAVA_OPT_@}"; do
    SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
done
Bug: Uses 'spark.sql.execution.arrow.maxRecordsPerBatch' in R createDataFrame with Arrow by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 04:44
Affects Version/s: 3.5.0

Now, createDataFrame uses `1` for numPartitions by default, which isn't realistic. Should use larger number for default partitions.

In PySpark, we chunk the input data by  'spark.sql.execution.arrow.maxRecordsPerBatch' size. Should better follow that in SparkR.
Bug: Subquery decorrelation rewriteDomainJoins failure from ConstantFolding to isnull
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/23 23:49
Affects Version/s: 3.4.0

We can get a decorrelation error because of rewrites that run in between DecorrelateInnerQuery and rewriteDomainJoins, that modify the correlation join conditions. In particular, ConstantFolding can transform `innercol <=> null` to `isnull(innercol)` and then rewriteDomainJoins does not recognize this and throws error Unable to rewrite domain join with conditions: ArrayBuffer(isnull(innercol#280)) because the isnull is not an equality, so it isn't usable for rewriting the domain join.

Can fix by recognizing `isnull(innercol)` as `innercol <=> null` in rewriteDomainJoins.

This area is also fragile in general and other rewrites that run between the two steps of decorrelation could potentially break their assumptions, so we may want to investigate longer-term follow ups for that.
Bug: Error while serializing ExecutorPeakMetricsDistributions into API JSON response
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/23 21:29
Affects Version/s: 3.3.3

When we try to get the ExecutorPeakMetricsDistributions [through the API|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/api.scala#L463] (/stages), there is a possibility of encountering an issue while serializing the StagesData into a JSON if the executor metrics are empty. 

 

The following error is thrown : 
{code:java}
Caused by: com.fasterxml.jackson.databind.JsonMappingException: -1 (through reference chain: scala.c
ollection.immutable.$colon$colon[0]->org.apache.spark.status.api.v1.StageData["executorMetricsDistri
butions"]->org.apache.spark.status.api.v1.ExecutorMetricsDistributions["peakMemoryMetrics"])
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
a:390)
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
a:349) {code}
This happens because the indices for the quartiles are populated incorrectly as -1 since the metrics itself are empty and this leads to this exception being thrown.
Bug: References to a specific side's key in a USING join can have wrong nullability
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 20:48
Affects Version/s: 3.3.2

Assume this data:
{noformat}
create or replace temp view t1 as values (1), (2), (3) as (c1);
create or replace temp view t2 as values (2), (3), (4) as (c1);
{noformat}
The following query produces incorrect results:
{noformat}
spark-sql (default)> select explode(array(t1.c1, t2.c1)) as x1
from t1
full outer join t2
using (c1);
1
-1      <== should be null
2
2
3
3
-1      <== should be null
4
Time taken: 0.663 seconds, Fetched 8 row(s)
spark-sql (default)> 
{noformat}
Similar issues occur with right outer join and left outer join.

{{t1.c1}} and {{t2.c1}} have the wrong nullability at the time the array is resolved, so the array's {{containsNull}} value is incorrect.

Queries that don't use arrays also can get wrong results. Assume this data:
{noformat}
create or replace temp view t1 as values (0), (1), (2) as (c1);
create or replace temp view t2 as values (1), (2), (3) as (c1);
create or replace temp view t3 as values (1, 2), (3, 4), (4, 5) as (a, b);
{noformat}
The following query produces incorrect results:
{noformat}
select t1.c1 as t1_c1, t2.c1 as t2_c1, b
from t1
full outer join t2
using (c1),
lateral (
  select b
  from t3
  where a = coalesce(t2.c1, 1)
) lt3;
1	1	2
NULL	3	4
Time taken: 2.395 seconds, Fetched 2 row(s)
spark-sql (default)> 
{noformat}
The result should be the following:
{noformat}
0	NULL	2
1	1	2
NULL	3	4
{noformat}
Bug: Expression "<lambda>(v)" within a window function doesn't raise a AnalysisException
Status: Open | Priority: Major | Resolution: 
Created: 23/May/23 00:39
Affects Version/s: 3.5.0

Expression "<lambda>(v)" within a window function doesn't raise a AnalysisException

 

See PandasUDFWindowParityTests.test_invalid_args for reproduction.
Bug: When encryption is enabled on the External Shuffle Service, then processing of push meta requests throws NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/23 02:48
Affects Version/s: 3.2.4

After enabling support for over-the-wire encryption for spark shuffle services, the meta requests for push-merged blocks fail with this error:
{code:java}
java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.spark.network.server.AbstractAuthRpcHandler.getMergedBlockMetaReqHandler(AbstractAuthRpcHandler.java:110)
	at org.apache.spark.network.crypto.AuthRpcHandler.getMergedBlockMetaReqHandler(AuthRpcHandler.java:144)
	at org.apache.spark.network.server.TransportRequestHandler.processMergedBlockMetaRequest(TransportRequestHandler.java:275)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:117)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:140)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
	at org.sparkproject.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at org.sparkproject.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) 
{code}
Bug: Upgrade Netty from 4.1.89 to 4.1.92
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 02:40
Affects Version/s: 3.5.0

* [https://netty.io/news/2023/03/14/4-1-90-Final.html]
 * [https://netty.io/news/2023/04/03/4-1-91-Final.html]
 * https://netty.io/news/2023/04/25/4-1-92-Final.html
Bug: Define a new error class and apply for the case where streaming query fails due to concurrent run of streaming query with same checkpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/23 20:25
Affects Version/s: 3.5.0

We are migrating to a new error framework in order to surface errors in a friendlier way to customers. This PR defines a new error class specifically for when there are concurrent updates to the log for the same batch ID
Bug: when shuffle hash join is enabled, q95 performance deteriorates
Status: Open | Priority: Major | Resolution: 
Created: 16/May/23 13:16
Affects Version/s: 3.1.2

Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used.

 
Performance difference:  from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin)
 

1. enable shuffledHashJoin, the execution plan is as follows:

!image-2023-05-16-21-28-44-163.png|width=935,height=64!

!image-2023-05-16-21-21-35-493.png|width=924,height=502!

2. disable shuffledHashJoin, the execution plan is as follows:

!image-2023-05-16-21-28-11-514.png|width=922,height=67!

!image-2023-05-16-21-22-16-170.png|width=934,height=477!

 

And when shuffledHashJoin is enabled, gc is very serious,

!image-2023-05-16-21-23-35-237.png|width=929,height=570!

 

but sortMergeJoin executes without this problem.

!image-2023-05-16-21-24-09-182.png|width=931,height=573!

 

Any suggestions on how to solve it？Thanks!
Bug: Spark Connect client cannot read from Hive metastore
Status: Open | Priority: Major | Resolution: 
Created: 19/May/23 03:44
Affects Version/s: 3.4.0

I created a Spark Connect shell in a pod using the following yaml.
{code:java}
apiVersion: v1
kind: Service
metadata:
  name: spark-connect-svc
  namespace: MY_NAMESPACE
spec:
  clusterIP: None
  selector:
    app: spark-connect-pod
    podType: spark-connect-driver


apiVersion: v1
kind: Pod
metadata:
  name: spark-connect-pod
  namespace: realtime-streaming
  labels:
    app: spark-connect-pod
    podType: spark-connect-driver
spec:
  restartPolicy: Never
  containers:
  - command:
    - sh
    - -c
    - /opt/spark/sbin/start-connect-server.sh --master k8s://https://MY_API_SERVER:443 --packages org.apache.spark:spark-connect_2.12:3.4.0 --conf spark.kubernetes.executor.limit.cores=1.0 --conf spark.kubernetes.executor.request.cores=1.0 --conf spark.executor.cores=1 --conf spark.executor.memory=6G --conf spark.kubernetes.container.image=MY_ECR_REPO/spark:3.4-prd  --conf spark.kubernetes.executor.podNamePrefix=spark-connect --num-executors=10 --conf spark.kubernetes.driver.pod.name=spark-connect-pod --conf spark.kubernetes.namespace=MY_NAMESPACE && tail -100f /opt/spark/logs/spark--org.apache.spark.sql.connect.service.SparkConnectServer-1-spark-connect-pod.out
    image: MY_ECR_REPO/spark-py:3.4-prd
    name: spark-connect-pod
 {code}
The Spark Connect server was successfully launched and I can connect to it using pyspark.

 

But when I want to add a Hive metastore config , it won't work.

 
{code:java}
>>> spark = SparkSession.builder.remote("sc://spark-connect-svc").config("spark.hive.metastore.uris", "thrift://hive-metastore:9083").getOrCreate()
>>> spark.sql("show databases").show()
+---------+
|namespace|
+---------+
|  default|
+---------+{code}
There're many databases under the hive metastore and I've tested with the local env, all works fine.