av9ash/GitBugs huggingface dataset:
Bug: py4j-0.10.9.5 often produces "Connection reset by peer"  in Spark 3.3.0
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Aug/22 11:24
Affects Version/s: 3.3.0

Without any visible reason I am getting this error in my Jupyter notebook (see stacktrace below) with pyspark kernel. Often it occurs even if no Spark operations are made, e.g. when I am working with multiprocessing Pool for a local piece of code that should parallelize on the cores of the driver, with no spark transformations/actions done in that jupyter cell.


INFO:py4j.clientserver:Error while sending or receiving.
Traceback (most recent call last):
  File "/opt/spark-3.3.0-bin-without-hadoop/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 503, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

INFO:py4j.clientserver:Closing down clientserver connection
INFO:root:Exception while sending command.
Traceback (most recent call last):
  File "/opt/spark-3.3.0-bin-without-hadoop/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 503, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/spark-3.3.0-bin-without-hadoop/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/opt/spark-3.3.0-bin-without-hadoop/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 506, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending
INFO:py4j.clientserver:Closing down clientserver connection
Bug: Security Vulnerability CVE-2018-11793 due to mesos-1.4.3-shaded-protobuf.jar
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/22 13:19
Affects Version/s: 3.3.0

Hello Team,
We are facing this vulnerability on Spark Installation 3.3.3 , Can we please upgrade the version of mesos in our installation to address this vulnerability. 


||Package||cve||cvss||severity||pkg_version||fixed_in_pkg||pkg_path||
|1|org.apache.mesos_mesos|CVE-2018-11793|7|high|1.4.0|1.7.1, 1.6.2, 1.5.2, 1.4.3|/opt/domino/spark/python/build/lib/pyspark/jars/mesos-1.4.0-shaded-protobuf.jar|


In our source code i found that the depedant version of mesos jar is 1.4.3 

user@ThinkPad-E14-02:~/Downloads/spark-master$ grep -ir mesos- * 
core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala: * TaskSchedulerImpl. We assume a Mesos-like model where the application gets resource offers as
*dev/deps/spark-deps-hadoop-2-hive-2.3:mesos/1.4.3/shaded-protobuf/mesos-1.4.3-shaded-protobuf.jar
dev/deps/spark-deps-hadoop-3-hive-2.3:mesos/1.4.3/shaded-protobuf/mesos-1.4.3-shaded-protobuf.jar
*
Bug: Security scanning spark v3.3.0 docker image results in DSA-5169-1 critical vulnerability
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Aug/22 15:57
Affects Version/s: 3.3.0

Dear Spark Team,

Whilst security scanning the docker image: docker.io/apache/spark:v3.3.0, I discovered the following vulnerability/scan results within the image :

      Type:            VULNERABILITY

      Name:            DSA-5169-1

      CVSS Score v3:   9.8

      Severity:        critical

The advice from [https://www.debian.org/security/2022/dsa-5169] suggests to upgrade the version of openssl to 1.1.1n-0+deb11u3

Steps to reproduce:
Install trivy [https://aquasecurity.github.io/trivy/v0.18.3/installation/]
trivy image docker.io/apache/spark:v3.3.0
Bug: StringCoding UTF8 decode slowly
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Aug/22 02:56
Affects Version/s: 3.1.2

When `UnsafeRow` is converted to `Row` at `org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.createExternalRow `,  UTF8String decoding and copyMemory  process are very slow.

Does anyone have any ideas for optimization?

!image-2022-08-22-10-56-54-768.png!

 

!image-2022-08-22-10-57-11-744.png!
Bug: unpersist cascades with Kryo, MEMORY_AND_DISK_SER and monotonically_increasing_id
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 24/Aug/22 00:36
Affects Version/s: 3.2.1

Unpersist of a parent dataset which has a column from _*monotonically_increasing_id*_ cascades to a child dataset when
 * joined on another dataset
 * kryo serialization is enabled
 * storage level is MEMORY_AND_DISK_SER
 * not all rows join????

 
{code:java}
import org.apache.spark.sql.functions.monotonically_increasing_id
import org.apache.spark.storage.StorageLevel

case class a(value: String, id: Long)

val storageLevel = StorageLevel.MEMORY_AND_DISK_SER // cascades
//val storageLevel = StorageLevel.MEMORY_ONLY // doesn't cascade

val acc = sc.longAccumulator("acc")
val parent1DS = spark.createDataset(Seq("a", "b", "c"))
                     .withColumn("id", monotonically_increasing_id)
                     .as[a]
                     .persist(storageLevel)

val parent2DS = spark.createDataset(Seq(1, 2, 3)) // 0,1,2 doesn't cascade
                     .persist(storageLevel)

val childDS = parent1DS.joinWith(parent2DS, parent1DS("id") === parent2DS("value"))
                       .map(i =>{ 
                                  acc.add(1) 
                                  i
                                }).persist(storageLevel)

childDS.count
parent1DS.unpersist
childDS.count

acc.value should be(2) {code}
Bug: KMeans: high variability in results despite high initSteps parameter value
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 26/Aug/22 13:35
Affects Version/s: 3.3.0

I'm running KMeans on a sample dataset using PySpark. I want the results to be fairly stable, so I play with the _initSteps_ parameter. My understanding is that the higher the number of steps for k-means|| initialization mode, the higher the number of iterations the algorithm runs and in the end selects the best model out of all iterations. And that's the behavior I observe when running sklearn implementation with _n_init_ >= 10. However, when running PySpark implementation, regardless of the number of partitions of underlying data frame (tested on 1, 4, 8 number of partitions), even when setting _initSteps_ to 10, 50, or let's say 500, the results I get with different seeds are different and trainingCost value I observe is sometimes far from the lowest.

As a workaround, to force the algorithm to iterate and select the best model I used a loop with dynamic seed.

SKlearn in each iteration gets the trainingCost near 276655.

PySpark implementation of KMeans gets there in the 2nd, 5th and 6th iteration, but all the remaining iterations yield higher values.

Does the _initSteps_ parameter work as expected? Because my findings suggest that something might be off here.

Let me know where I could upload this sample dataset (2MB)

 
{code:java}
import pandas as pd
from sklearn.cluster import KMeans as KMeansSKlearn
df = pd.read_csv('sample_data.csv')

minimum = 99999999
for i in range(1,10):
    kmeans = KMeansSKlearn(init="k-means++", n_clusters=5, n_init=10, random_state=i)
    model = kmeans.fit(df)
    print(f'Sklearn iteration {i}: {round(model.inertia_)}')from pyspark.sql 

import SparkSession
spark= SparkSession.builder \
    .appName("kmeans-test") \
    .config('spark.driver.memory', '2g') \
    .master("local[2]") \
    .getOrCreate()df1 = spark.createDataFrame(df)

from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler
assemble=VectorAssembler(inputCols=df1.columns, outputCol='features')
assembled_data=assemble.transform(df1)

minimum = 99999999
for i in range(1,10):
    kmeans = KMeans(featuresCol='features', k=5, initSteps=100, maxIter=300, seed=i, tol=0.0001)
    model = kmeans.fit(assembled_data)
    summary = model.summary
    print(f'PySpark iteration {i}: {round(summary.trainingCost)}'){code}
Bug: Use DataFrame's column for referring to DDL schema for from_csv() and from_json()
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Aug/22 12:58
Affects Version/s: 3.0.0

With spark's DataFrame api one has to explicitly pass the StrucType to functions like from_csv and from_json. This works okay in general.

In certain circumstances when schema depends on the one of the DataFrame's field, it gets complicated and one has to switch to RDD. This requires additional libraries to be added with additional parsing logic.

I am trying to explore a way to enable such use cases with DataFrame api and function itself.
Bug: SPLIT function on empty regex should truncate trailing empty string.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 22:14
Affects Version/s: 3.4.0

E.g. `select split('hello', '')` should convert to `['h', 'e', 'l', 'l', 'o']` instead of `['h', 'e', 'l', 'l', 'o', '']`. Requires explicit `limit` parameter to preserve trailing empty string.
Bug: ANALYZE TABLE makes some queries run forever
Status: Reopened | Priority: Major | Resolution: 
Created: 03/Aug/22 23:31
Affects Version/s: 3.2.2

I'm using TPCDS to run benchmarks, and after running ANALYZE TABLE (without the FOR ALL COLUMNS) some queries became really slow. For example query24 - [https://raw.githubusercontent.com/Agirish/tpcds/master/query24.sql] takes between 10~15min before running the ANALYZE TABLE.

After running ANALYZE TABLE I waited 24h before cancelling the execution.

If I disable spark.sql.cbo.joinReorder.enabled or 
spark.sql.cbo.enabled it becomes fast again.
It seems something in join reordering is not working well when we have table stats, but not column stats.

Rows Count:
store_sales - 2879966589
store_returns - 288009578
store - 1002
item - 300000
customer - 12000000
customer_address - 6000000
Bug: Switch base image from openjdk to eclipse-temurin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/22 08:18
Affects Version/s: 3.4.0

According to https://github.com/docker-library/openjdk/issues/505 and https://github.com/docker-library/docs/pull/2162, openjdk:8/11 is EOL and Eclipse Temurin replaces this.

The `openjdk` image is [not update anymore](https://adoptopenjdk.net/upstream.html) (the last releases were 8u342 and 11.0.16), will `remove the 11 and 8 tags (in October 2022, perhaps)` (we are using it in spark), so we have to switch this before it happens.
Bug: Use error classes in the compilation errors of GROUP BY a position
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/22 17:49
Affects Version/s: 3.4.0

Migrate the following errors in QueryCompilationErrors:
* groupByPositionRefersToAggregateFunctionError
* groupByPositionRangeError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Specify the column name when the data type is not supported by datasource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/22 14:09
Affects Version/s: 3.4.0

Currently, If the data type is not supported by the data source, the exception message thrown does not contain the column name, which is less clear for locating the problem, this Jira aims to optimize error message description
Bug: Use interruptible lock instead of synchronized in Executor.updateDependencies()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 21:07
Affects Version/s: 3.4.0

This patch modifies the synchronization in {{Executor.updateDependencies()}} in order to allow tasks to be interrupted while they are blocked and waiting on other tasks to finish downloading dependencies.

This synchronization was added years ago in [mesos/spark@{{{}7b9e96c{}}}|https://github.com/mesos/spark/commit/7b9e96c99206c0679d9925e0161fde738a5c7c3a] in order to prevent concurrently-launching tasks from performing concurrent dependency updates. If one task is downloading dependencies, all other newly-launched tasks will block until the original dependency download is complete.

Let's say that a Spark task launches, becomes blocked on a {{updateDependencies()}} call, then is cancelled while it is blocked. Although Spark will send a Thread.interrupt() to the canceled task, the task will continue waiting because threads blocked on a {{synchronized}} won't throw an InterruptedException in response to the interrupt. As a result, the blocked thread will continue to wait until the other thread exits the synchronized block. 

In the wild, we saw a case where this happened and the thread remained blocked for over 1 minute, causing the TaskReaper to kick in and self-destruct the executor.

This PR aims to fix this problem by replacing the {{synchronized}} with a ReentrantLock, which has a {{lockInterruptibly}} method.
Bug: Codegen compilation error when using split_part
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/22 21:29
Affects Version/s: 3.3.0

The following query throws an error:
{noformat}
create or replace temp view v1 as
select * from values
('11.12.13', '.', 3)
as v1(col1, col2, col3);

cache table v1;

SELECT split_part(col1, col2, col3)
from v1;
{noformat}
The error is:
{noformat}
22/08/19 14:25:14 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 42, Column 1: Expression "project_isNull_0 = false" is not a type
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 42, Column 1: Expression "project_isNull_0 = false" is not a type
	at org.codehaus.janino.Java$Atom.toTypeOrCompileException(Java.java:3934)
	at org.codehaus.janino.Parser.parseBlockStatement(Parser.java:1887)
	at org.codehaus.janino.Parser.parseBlockStatements(Parser.java:1811)
	at org.codehaus.janino.Parser.parseBlock(Parser.java:1792)
	at 
{noformat}
In the end, {{split_part}} does successfully execute, although in interpreted mode.
Bug: Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 02:46
Affects Version/s: 3.4.0

Avoid OOM issue as far as possible:
{code:java}
ObjectAggregationIterator INFO - Aggregation hash map size 128 reaches threshold capacity (128 entries), spilling and falling back to sort based aggregation. You may change the threshold by adjust option spark.sql.objectHashAggregate.sortBased.fallbackThreshold

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
#   Executing /bin/sh -c "kill 46725"...{code}
Bug: Further check the availability of RemoteBlockPushResolver API after call close
Status: Open | Priority: Major | Resolution: 
Created: 29/Aug/22 02:34
Affects Version/s: 3.4.0

Call the API of RemoteBlockPushResolver after it has been closed is a dangerous behavior.

For example, `applicationRemoved` will try to delete data from a closed DB instance, and when the DB instance is RocksDB, JVM will crash.

 

It needs to further check whether this scenario is possible.
Bug: Clean only MDC items set by Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 20:21
Affects Version/s: 3.4.0

Since SPARK-8981, Spark executor adds MDC support. Before setting MDC items, the executor cleans up all MDC items. But it causes an issue for other MDC items not set by Spark but from users at other places. It causes these custom MDC items not shown in executor log.
Bug: Logging isn't configurable via log4j2 with hadoop-provided profile
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Aug/22 17:44
Affects Version/s: 3.3.0

When building Spark with -Phadoop-provided (or using the 3.3.0 build without Hadoop), there is no slf implementation provided for log4j2, so the default log4j2 properties are ignored and logging isn't configurable via SparkContext.setLogLevel.

Reproduction on a fresh Ubuntu container:

 
{noformat}
apt-get update
apt-get install -y wget
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
wget https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-without-hadoop.tgz
tar -xvf hadoop-3.3.4.tar.gz -C /opt
tar -xvf spark-3.3.0-bin-without-hadoop.tgz -C /opt
export HADOOP_HOME=/opt/hadoop-3.3.4/
export SPARK_HOME=/opt/spark-3.3.0-bin-without-hadoop/
apt install -y openjdk-11-jre-headless python3
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/
export SPARK_DIST_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)
$SPARK_HOME/bin/pyspark
{noformat}
The default log level starts at INFO and you can't change it with sc.setLogLevel
Bug: Only return all physical plans after summitting pyspark script with several spark sql blocks inside
Status: Open | Priority: Major | Resolution: 
Created: 27/Aug/22 07:01
Affects Version/s: 2.1.3

Backgroud:

    In industry development environment, we got used to write several spark-sql blocks in one pyspark script. Without really submitting to cluster as running application, getting all the physical plans of a application is too hard to do within a parameter statement of "spark-submit" command.

 

Wish:

    I wish to add a parameter "–genPlan" as for "spark-submit" command, which can return all the physical plans of a application instead of submitting to run indeed. Also, other approach to settle the matter is wellcome.
Bug: Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/22 22:27
Affects Version/s: 3.4.0

Similar to https://issues.apache.org/jira/browse/SPARK-40054, we should refactor the {{{}try_add{}}}/{{{}try_subtract{}}}/{{{}try_multiply{}}}/{{{}try_divide{}}} functions so that the errors from their children will be shown instead of ignored.
 Spark SQL allows arithmetic operations between Number/Date/Timestamp/CalendarInterval/AnsiInterval (see the rule [ResolveBinaryArithmetic|https://github.com/databricks/runtime/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L501] for details). Some of these combinations can throw exceptions too: * Date + CalendarInterval
 * Date + AnsiInterval
 * Timestamp + AnsiInterval
 * Date - CalendarInterval
 * Date - AnsiInterval
 * Timestamp - AnsiInterval
 * Number * CalendarInterval
 * Number * AnsiInterval
 * CalendarInterval / Number
 * AnsiInterval / Number

This Jira is for the cases when both input data types are numbers.
Bug: Error messages produced from error-classes.json should not have hard-coded sentences as parameters
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/22 21:18
Affects Version/s: 3.4.0

Relevant comment: [https://github.com/apache/spark/pull/37621#discussion_r955101102.]

Example of hard coded string: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala#L512-L513
Bug: Executor connection issue in hybrid cloud deployment
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/22 08:59
Affects Version/s: 3.2.1

I understand that the issue is quite subtle and might be hard to debug, still I was not able to find issue with our infra, so I guess that is something inside the spark.

We deploy spark application in k8s and everything works well, if all the driver & executor pods are either in AWS or our DC, but in case they are split between datacenters something strange happens, for example, logs of one of the executors inside the DC
{code:java}
22/08/26 07:55:35 INFO TransportClientFactory: Successfully created connection to /172.19.149.92:39414 after 50 ms (1 ms spent in bootstraps)
22/08/26 07:55:35 TRACE TransportClient: Sending RPC to /172.19.149.92:39414
22/08/26 07:55:35 TRACE TransportClient: Sending request RPC 4860401977118244334 to /172.19.149.92:39414 took 3 ms
22/08/26 07:55:35 DEBUG TransportClient: Sending fetch chunk request 0 to /172.19.149.92:39414
22/08/26 07:55:35 TRACE TransportClient: Sending request StreamChunkId[streamId=1644979023003,chunkIndex=0] to /172.19.149.92:39414 took 0 ms
22/08/26 07:57:35 ERROR TransportChannelHandler: Connection to /172.19.149.92:39414 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.shuffle.io.connectionTimeout if this is wrong. {code}
The executor successfully creates connection & sends the request, but the connection was assumed dead. Even stranger the executor on ip 172.19.149.92 have sent the response back, which I can confirm with following logs
{code:java}
22/08/26 07:55:35 TRACE MessageDecoder: Received message ChunkFetchRequest: ChunkFetchRequest[streamChunkId=StreamChunkId[streamId=1644979023003,chunkIndex=0]]
22/08/26 07:55:35 TRACE ChunkFetchRequestHandler: Received req from /172.19.123.197:37626 to fetch block StreamChunkId[streamId=1644979023003,chunkIndex=0]
22/08/26 07:55:35 TRACE OneForOneStreamManager: Removing stream id 1644979023003
22/08/26 07:55:35 TRACE BlockInfoManager: Task -1024 releasing lock for broadcast_0_piece0
--
22/08/26 07:55:35 TRACE BlockInfoManager: Task -1024 releasing lock for broadcast_0_piece0
22/08/26 07:55:35 TRACE ChunkFetchRequestHandler: Sent result ChunkFetchSuccess[streamChunkId=StreamChunkId[streamId=1644979023003,chunkIndex=0],buffer=org.apache.spark.storage.BlockManagerManagedBuffer@79b43e2a] to client /172.19.123.197:37626 {code}
A few suspicious moments here:
 * connection to pod looks like /<IP>, while connection to driver looks like <POD_NAME>.<NAMESPACE>.svc/<IP>
 * Task *-1024* releasing lock for broadcast_0_piece0
Bug: Don't output the empty map of error message parameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/22 14:52
Affects Version/s: 3.4.0

In the current implementation, Spark output empty message parameters in the MINIMAL and STANDARD formats:

{code:json}
 org.apache.spark.SparkRuntimeException
 {
  "errorClass" : "ELEMENT_AT_BY_INDEX_ZERO",
  "messageParameters" : { }
 }
{code}
that contradict w/ the approach for other JSON fields.
Bug: Data Source V2: Support creating table with the duplicate transform with different arguments
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Aug/22 07:00
Affects Version/s: 3.2.2

This patch adds support to creating a table with duplicate transforms while with different arguments. For example, the following code will be failed before:
```scala
CREATE TABLE testcat.t (id int, `a.b` string) USING foo PARTITIONED BY (bucket(2, id), bucket(4, id))
```
Bug: Whether it is possible to support querying the status of a specific application in a subsequent version
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/22 07:28
Affects Version/s: 2.4.4

The current SparkAppHandler cannot support obtaining the application status in Standalone Cluster mode. One way is to query the status of the specified Driver through the StandaloneRestServer, but it cannot query the status of the specified application. Is it possible to add a method (eg: handleAppStatus) to the StandaloneRestServer by asking the Master Send the RequestMasterState message to get the state of the specified application. The current MasterWebUI should do this, but the premise is that it needs to use the same RpcEnv as the Master Endpoint. Many times we care about the status of the application rather than the status of the Driver, so we hope to add this function in subsequent versions to support obtaining the status of the specified application in Standalone cluster mode.