av9ash/GitBugs huggingface dataset:
Bug: Spark task execution with Java execution option has an injection problem
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/24 11:27
Affects Version/s: 3.5.3

1. We use spark to perform periodic calculation through Spark tasks preset in our system, and use the calculation results for reporting systemdisplay;

2. Due to difference data traffic model in different cities, we provide a webportal maintenance page, so that the maintenance staff could dynamically adjust parameters according to the actual situation. the maintenance staff only can set parameters for preset tasks, do not have right to access the workstation or  run command on workstation. 

3. The security department of the company finds that there has a risk of command injection in the webportal, which may cause maintenance staff to obtain the workstation permission or run command on workstation, so the security department expects us to solve this problem.

For example, if someone input special injection characters (' touch$IFS+ command, $IFS is linux command, $IFS specifies a space by default), then the injection will happen on the workstation.

1.Enter special characters like touch$IFS+command and pass them to the Spark

2.The workstation was successfully injected and the /tmp/zz34 file be created： 

!inject.png!

3.ExtraJavaOptions parameter was set by the sparkLauncher. setConf method submit the Spark program. Yarn task submission page displays that the parameters was passed to the Spark by spark.driver.extra.javaOptions parameter: 

!yarn.PNG!

4.We read the spark source code and found that Spark provides security protection for data such as memory parameters, but does not provide security protection for extraJavaParam parameter. In view of the above situation, does Spark have security risks? What do you think of this scenario?

!srcfilter.PNG!
Bug: Spark Connect should support str ndarray
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 05:17
Affects Version/s: 4.0.0

In [5]: spark.range(1).select(sf.lit(np.array(["a", "b"], np.str_))).schema
---------------------------------------------------------------------------
PySparkTypeError                          Traceback (most recent call last)
Cell In[5], line 1
----> 1 spark.range(1).select(sf.lit(np.array(["a", "b"], np.str_))).schema

File ~/Dev/spark/python/pyspark/sql/utils.py:272, in try_remote_functions.<locals>.wrapped(*args, **kwargs)
    269 if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
    270     from pyspark.sql.connect import functions
--> 272     return getattr(functions, f.__name__)(*args, **kwargs)
    273 else:
    274     return f(*args, **kwargs)

File ~/Dev/spark/python/pyspark/sql/connect/functions/builtin.py:271, in lit(col)
    269 elif isinstance(col, np.ndarray) and col.ndim == 1:
    270     if _from_numpy_type(col.dtype) is None:
--> 271         raise PySparkTypeError(
    272             errorClass="UNSUPPORTED_NUMPY_ARRAY_SCALAR",
    273             messageParameters=\{"dtype": col.dtype.name},
    274         )
    276     # NumpyArrayConverter for Py4J can not support ndarray with int8 values.
    277     # Actually this is not a problem for Connect, but here still convert it
    278     # to int16 for compatibility.
    279     if col.dtype == np.int8:

PySparkTypeError: [UNSUPPORTED_NUMPY_ARRAY_SCALAR] The type of array scalar 'str32' is not supported.
Bug: Fix Javascript and image imports in the documentation site
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/24 04:59
Affects Version/s: 4.0.0

In [SPARK-49378|https://github.com/apache/spark/pull/47864/], we introduced a change to break apart the Structured Streaming programming guide. This created several pages under `/streaming`.

To make this change work, we had to modify some file-paths in script files; previously, they used to rely on the fact that every page were siblings. But after the nesting of `/streaming` was added, this assumption was broken. We introduced the `rel_path_to_root` Jekyll variable to make this work, and we used it in most places.

However, we [inadvertently modified|https://github.com/apache/spark/pull/47864/files#diff-729ad9c4e852768f70b7c45195e7e5f8271a7f3146df73045e441d024b907819R201-R202] the paths to our main Javascript file and AnchorJS import to be absolute. These should instead be prefixed with `rel_path_to_root`. (The net effect is that the language-specific code blocks aren't rendering properly on any of the pages.)

Also, the images in the Structured Streaming programming guide use `/img`, which is not correct. Those also need `rel_path_to_root`.
Bug: Factor out NormalizePlan to a separate class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 13:04
Affects Version/s: 4.0.0

Factor out plan normalization functions to later use those in single-pass Analyzer testing.

In the single-pass Analyzer proposal we discussed one of the ways to test the single-pass Analyzer - to run tests in dual mode (both single-pass and fixed-point Analyzers) and compare the logical plans. To do that we need to perform plan normalization.

Context: https://issues.apache.org/jira/browse/SPARK-49834
Bug: Some warn messages should actually use logger instead of the warnings module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/24 07:13
Affects Version/s: 4.0.0

Some warnings messages in the Pyspark connect module are meant more as informational. They are now using the 'warnings' module which incorrectly pops up not in the log output but rather stdout.

The warnings module should be used only to produce user facing warnings because of an action they have taken.
Bug: Duplicate JVM options
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 01:15
Affects Version/s: 3.5.3

JVM options in spark.driver.defaultJavaOptions and spark.executor.defaultJavaOptions end up being added multiple times to the command line due to the way the Java module options and IPv6 options are added.

{{$ pyspark --conf spark.driver.defaultJavaOptions="-Dfoo=bar"}}

{{>>> spark.conf.get('spark.driver.extraJavaOptions')}}

{{'-Djava.net.preferIPv6Addresses=false -Dfoo=bar -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dfoo=bar'}}

{{>>> spark.conf.get('spark.driver.defaultJavaOptions')}}

{{'-Dfoo=bar'}}
Bug: LEGACY store assignment policy should not be allowed in write with DataFrameWriteV2 API
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 15/Oct/24 12:02
Affects Version/s: 3.5.3

 LEGACY store assignment policy is disallowed in data source v2. However, it's allowed to write with DataFrameWriteV2 API. The following code will work without errors.
{code:java}
spark.sql("set spark.sql.storeAssginmentPolicy=LEGACY")
df.writeTo($v2Table).append() {code}
Bug: Fix Identity Column metadata get
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/24 08:53
Affects Version/s: 4.0.0

[https://github.com/apache/spark/blob/fae0c12fea8866c000d1bf6263fbf5cae8526691/sql/[…]n/scala/org/apache/spark/sql/catalyst/util/IdentityColumn.scala|https://github.com/apache/spark/blob/fae0c12fea8866c000d1bf6263fbf5cae8526691/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IdentityColumn.scala#L64] Getting identity column metadata should call {{getLong/getBoolean}} rather than {{getString.toLong/getString.toBoolean.}}
Bug: Don't resolve globs in SparkSubmit cluster deploy mode
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/24 16:22
Affects Version/s: 3.5.0

I was trying to use Spark in cluster deploy mode with K8s Operator and S3, but stumbled upon an issue ([log|[https://gist.github.com/sdwbgn/c3a8c4e687dc30ad03511d81211abc7e]]) when Spark Submit tried to authorize to S3 to resolve globs. So it requires having configured S3 access on Spark Submit.

It seems redundant as the same resolution will also happen on the driver itself, when job will be submitted. So it seems reasonable to disable on spark-submit in cluster mode.
Bug: Session variable not properly evaluated in qualify clause
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Oct/24 07:56
Affects Version/s: 3.4.3

Session variable are not properly evaluated in qualify clause. Query ??select count(*) over() as cnt from values(1) t (c1) qualify c1 = session.v_myvar?? should work but it isn't. This should be fixed.
Bug: Use stack-based iterative computation to avoid creating many Scala List objects for deep expression trees
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/24 15:50
Affects Version/s: 4.0.0

In some use cases with deep expression trees, the driver's heap shows many `{{{}scala.collection.immutable.$colon$colon`{}}} objects from the heap.  The objects are allocated due to deep recursion in the {{gatherCommutative}} method which uses {{flatmap}} recursively. Each invocation of {{flatmap}} creates a new temporary Scala collection. Our claim is based on the following stack trace (>1K lines) of a thread in the driver below, truncated here for brevity:
 
{code:java}
"HiveServer2-Background-Pool: Thread-9867" #9867 daemon prio=5 os_prio=0 tid=0x00007f35080bf000 nid=0x33e7 runnable [0x00007f3393372000]   java.lang.Thread.State: RUNNABLE   	at scala.collection.immutable.List$Appender$1.apply(List.scala:350)	at scala.collection.immutable.List$Appender$1.apply(List.scala:341)	at scala.collection.immutable.List.flatMap(List.scala:431)	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression.gatherCommutative(Expression.scala:1479)	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression.$anonfun$gatherCommutative$1(Expression.scala:1479)	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression$$Lambda$5280/143713747.apply(Unknown Source)	at scala.collection.immutable.List.flatMap(List.scala:366)....	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression.gatherCommutative(Expression.scala:1479)	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression.$anonfun$gatherCommutative$1(Expression.scala:1479)	at org.apache.spark.sql.catalyst.expressions.CommutativeExpression$$Lambda$5280/143713747.apply(Unknown Source)	at scala.collection.immutable.List.flatMap(List.scala:366).... {code}
Bug: AQE hang forever when collecting twice on a failed plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/24 21:26
Affects Version/s: 4.0.0

When we collect twice in a failed AQE plan, no new query stage will be created, and no stage will be submitted either. We will be waiting for a finish event forever, which will never come because that query stage has already failed in the previous run.
Bug: Integrating Open Policy Agent with Spark
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 19:45
Affects Version/s: 3.5.3

Integrating [Open Policy Agent|https://www.openpolicyagent.org/] with Spark will enable fine-grained access control, compliance, and security policies with external data sources. 

One way to do this is to leverage SparkSessionExtensions to add a check rule against the spark logical plan prior to its execution. By accessing the logical plan and extracting the user_id, we can determine the user, actions (read, write, etc.), and resources. With this, we can construct and send an OPA http request, allowing granular authorization.
Bug: Fix ColumnarArray.copy() to read nulls from the correct offset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/24 16:16
Affects Version/s: 3.5.3

{{ColumnarArray}} represents an array containing elements from {{data[offset]}} to {{{}data[offset + length){}}}. When copying the array, the null flag should also be read starting from {{offset}} rather than 0.
Bug: Refactor relation resolution code out of Analyzer.scala
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/24 09:52
Affects Version/s: 4.0.0

Analyzer.scala is 4K+ lines long, so it makes sense to gradually split it.

We can move resolveRelation(...) and some of its dependencies out to a separate class, because it's reasonably self-contained.
Bug: Fix STRING type meaning in CREATE/ALTER commands with session level default collation
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/24 19:04
Affects Version/s: 4.0.0

Currently spark uses default collation for all STRING type occurrences in SQL. We want to limit this to only DML statements (top level UPDATE, DELETE, INSERT, MERGE). This ticket makes sure that CREATE or ALTER of a TABLE, VIEW, or FUNCTION do not get affected by the default session collation.
Bug: Named parameter invocation does not obey case--in-sensitivity rules for identifiers
Status: Open | Priority: Major | Resolution: 
Created: 11/Oct/24 06:14
Affects Version/s: 3.4.0

The following should work as long as identifiers are case-insensitive (which is the default):

{{SELECT mask('AaBb123-&^ % 서울 Ä', lowerchar => 'z', otherchar => 'X');}}

Returns this error:
{quote}[[UNRECOGNIZED_PARAMETER_NAME|https://docs.databricks.com/error-messages/error-classes.html#unrecognized_parameter_name]] Cannot invoke function `mask` because the function call included a named argument reference for the argument named `lowerchar`, but this function does not include any signature containing an argument with this name. Did you mean one of the following? [`lowerChar` `otherChar` `upperChar`]. SQLSTATE: 4274K

 
{quote}
Bug: MERGE INTO does not push down the predicate of the WHEN NOT MATCHED BY SOURCE THEN DELETE clause (Iceberg)
Status: Open | Priority: Major | Resolution: 
Created: 11/Oct/24 08:44
Affects Version/s: 3.5.2

h2. Description

When writing to an Iceberg table using MERGE INTO command, and using WHEN NOT MATCHED BY SOURCE AND ... THEN DELETE clause, it is doing a full Iceberg table scan, even though the table is well partitioned. Without this clause, there is no full table scan.

I’m using Spark 3.5.2 and Iceberg 1.6.0.
h2. An example

Table schema is as such:
CREATE TABLE iceberg_table (
  part_col_1 STRING,
  part_col_2 STRING,
  part_col_3 TIMESTAMP,
  some_col_x INT,
  some_col_y STRING,
  main_value_col BIGINT
)
USING iceberg
PARTITIONED BY (part_col_1, part_col_2, days(part_col_3)){{}}
 
MERGE INTO command is as follows:
MERGE INTO iceberg_table t
USING new_data_to_upsert s
ON
t.part_col_1 = 'some_part_value_a' AND
t.part_col_2 = 'some_part_value_b' AND
t.part_col_3 = s.part_col_3 AND
t.some_col_x = s.some_col_x AND
t.some_col_y = s.some_col_y
WHEN MATCHED THEN UPDATE SET t.main_value_col = s.main_value_col
WHEN NOT MATCHED BY TARGET THEN INSERT *
WHEN NOT MATCHED BY SOURCE AND
t.part_col_1 = 'some_part_value_a' AND
t.part_col_2 = 'some_part_value_b' AND
t.part_col_3 >= to_timestamp('2024-09-25 00:00:00') AND
t.part_col_3 < to_timestamp('2024-09-25 04:00:00')
THEN DELETE
 
View "{{{}new_data_to_upsert"{}}} has the same columns as the table, and all its records have
 * {{part_col_1}} equal "{{{}some_part_value_a"{}}},
 * {{part_col_2}} equal "{{{}some_part_value_b"{}}},
 * {{part_col_3}} in the range {{{}[2024-09-25 00:00:00; 2024-09-25 04:00:00){}}}.

I also {{.persist()}} it before doing {{{}.createOrReplaceTempView("new_data_to_upsert"){}}}.

Query plan shows for the {{iceberg_table}} scan the following:
(1) BatchScan iceberg_table
Output [8]: [some_col_x#511, part_col_3#512, part_col_1#513, part_col_2#514, some_col_y#515, main_value_col#516L, _file#520]
iceberg_table (branch=null) [filters=true, groupedBy=]{{}}
As far as I understand {{filters=true}} is the problem - no predicate push-down.
When I try a regular SELECT or DELETE, it shows smth like:
filters=[part_col_1 = 'some_part_value_a', part_col_2 = 'some_part_value_b', part_col_3 >= 1727222400000, part_col_3 < 1727236800000]
 
If I remove "{{{}WHEN NOT MATCHED BY SOURCE AND … THEN DELETE"{}}} clause from the query, the plan is different and the BatchScan steps to the Iceberg table do have {{filters}} specified. No matter what I've tried with the predicate in this clause, it never resulted in the predicate push-down.
 
I've created an issue in the Iceberg GitHub repo as well, but nobody has responded so far: https://github.com/apache/iceberg/issues/11248
Bug: Use different ShuffleOrigin for the shuffle required from stateful operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/24 06:14
Affects Version/s: 4.0.0

ShuffleOrigin was introduced to determine whether the shuffle can be adjusted or not, and if it can be, in which way.
 
ENSURE_REQUIREMENTS used by the rule `EnsureRequirements` denotes that the shuffle can be adjusted as long as the requirements are still ensured. Here the requirements do not include the number of partitions so AQE can possibly optimize the number (assuming that the number of partitions isn't a strong requirement), but for stateful operators, the number of partitions is a strong requirement and it must not be adjusted.
 
To prevent any further modification of AQE to break stateful operator (as same as the reason we introduced StatefulOpClusteredDistribution), we want to explicitly mark the shuffle backing up stateful operator as different ShuffleOrigin.
Bug: [M1] Spark Connect improvements
Status: Open | Priority: Major | Resolution: 
Created: 09/Oct/24 09:42
Affects Version/s: 4.0.0

In SPARK-48452 we have investigated the approach for Spark Connect and came up with some initial/basic solution - i.e. we are wrapping results in the LocalRelation with a isSqlScript param set to true. Our LogicalPlan extends the Command hierarchy and this way we didn't have to alter a lot on the SparkConnect end.

However, this is not the ideal/proper design and we should improve it:
 * Introduce proper LogicalPlan hierarchy for eager execution.
 * Remove isSqlScript flag from LocalRelation.
 * Investigate other options to return results through Spark Connect (i.e. result streaming; anything else?).
 * Anything else?
Bug: NormalizeFloatingNumbers degrades nullability of expressions with nested structs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/24 00:58
Affects Version/s: 4.0.0

For an AttributeReference where the dataType is a nested struct such that the internal struct requires normalization (has a floating type expression), we end up not correctly propagating the nullability of an expression.

For example, for an expression like:
{code:java}
namedStruct("struct", namedStruct("double", <DoubleType-field>) {code}
The dataType prior to normalization is:
{code:java}
StructType(StructField("struct", StructType(StructField("double", DoubleType, true, {})), false, {})){code}
whereas post-normalization, the dataType becomes:
{code:java}
StructType(StructField("struct", StructType(StructField("double", DoubleType, true, {})), true, {})) 
{code}
 

We ended up converting the `nullable` attribute of the "double" field from `false` to `true`.
Bug: Correct documentation and default for event log task metrics accumulator logging flag from SPARK-42204
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/24 18:20
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-42204 intended only to add a flag but not to change default behavior, but I made a mistake related to double-negation and accidentally set the opposite of the intended default.

Thus, this Jira / PR corrects the code to roll back the unintentional behavior change and updates the documentation accordingly.
Bug: Fix deadlock between shuffle dependency initialization and explain string generation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/24 00:22
Affects Version/s: 4.0.0

We hit a deadlock between shuffle dependency initialization and explain string generation, in which both code paths trigger lazy variable instantiation while visiting some tree nodes in different orders and thus acquiring object locks in a reversed order.

The hashcode of plan node is implemented as lazy val. So the fix is to remove hash code computation from explain string generation so to break the chain of lazy variable instantiation.
Bug: Avoid validateNullability failure for BooleanSimplification
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Oct/24 22:38
Affects Version/s: 4.0.0

```

case a And b if Not(a).semanticEquals(b) =>  If(IsNull(a), Literal.create(null, a.dataType), FalseLiteral) case a And b if a.semanticEquals(Not(b)) => If(IsNull(b), Literal.create(null, b.dataType), FalseLiteral) case a Or b if Not(a).semanticEquals(b) => If(IsNull(a), Literal.create(null, a.dataType), TrueLiteral) case a Or b if a.semanticEquals(Not(b)) => If(IsNull(b), Literal.create(null, b.dataType), TrueLiteral)

```

in BooleanSimplification change expressions from non-nullable to nullable if a/b is non-nullable. This is a degradation of optimizer and will block further optimization for non-nullable column/expressions.
Bug: Bump Hadoop libraries to to 3.3.9
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/24 16:32
Affects Version/s: 3.4.3

CVE-2024-47554 is fixed in that version of the library. Could please you confirm whether Spark is affected by this vulnerability and if so, are there any plans to update the dependency?

{code}
│ commons-io:commons-io (hadoop-client-runtime-3.3.4.jar)      │ CVE-2024-47554      │ HIGH     │          │ 2.8.0             │ 2.14.0                                     │ apache-commons-io: Possible denial of service attack on      │
│                                                              │                     │          │          │                   │                                            │ untrusted input to XmlStreamReader                           │
│                                                              │                     │          │          │                   │                                            │ https://avd.aquasec.com/nvd/cve-2024-47554                   │
{code}
h4. Steps to reproduce
{code}
trivy image spark
{code}
Bug: Change classifyException function signature: add isRuntime argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/24 13:40
Affects Version/s: 3.5.3

We want to change the signature of the [{*}classifyException{*}|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala#L746] function, so that it can return either AnalysisException or SparkRuntimeException. For this we should add a parameter *isRuntime* in the function signature, and change the return value to {*}Throwable with SparkThrowable{*}.

*classifyException* currently returns AnalysisException. We want to be able to call it from the execution part of the code, and get SparkRuntimeException.