av9ash/GitBugs huggingface dataset:
Bug: need guidance for vulnerability CVE-2022-42889 in spark 3.0.0 version
Status: Closed | Priority: Major | Resolution: Invalid
Created: 25/Oct/22 08:26
Affects Version/s: 3.0.0

Hi Spark team,

 

[~dongjoon]  [~bjornjorgensen] 

 

 

We are using spark 3.0.0 on AWS EMR service to run our spark jobs. 

spark-core_2.12:3.0.0  has transitive dependency on commons-text 1.6 and this is flagged as critical severity CVE-2022-42889.

As per Jira SPARK-40801 , commons text has been upgraded and spark 3.4.0 is released.

We are dependent on AWS EMR service and changing EMR version and spark version is big task for us considering all downstream dependent applications

We know spark 3.0.0 is EOL for you but would really appreciate if could provide guidance on it.

We have few queries and need inputs from spark dev team to handle this issue on priority at our end 

 

 
 * Does spark-core 3.0.0 use StringSubstitutor API and {*}do we need to worry about this{*}?
 * which lib or code within spark core 3.0.0 triggers StringSubstitutor  method ?
 * I searched for spark source code for usage of StringSubstitutor  and found one reference here [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala] in master branch but this class  is not available in spark 3.0.0 tags. As per link - [https://blogs.apache.org/security/entry/cve-2022-42889] , If you rely on software that uses a version of commons-text prior to 1.10.0, you are likely still not vulnerable: you are only affected when this software uses the {{StringSubstitutor}} API without properly sanitizing any untrusted input.
 *  *Please confirm if spark 3.0.0 does not use {{StringSubstitutor}} API from commons-text but just have dependency marked in POM* [https://github.com/apache/spark/blob/3fdfce3120f307147244e5eaf46d61419a723d50/pom.xml#L506]  for other API use from commons-text?
 * in case , can we include the apache commons text 1.10.0 as explicit dependency on our applications POMs and add common text 1.6 in exclusions for spark-core , will it work ?
Bug: Upgrade FasterXML jackson-databind to 2.14
Status: Closed | Priority: Major | Resolution: Resolved
Created: 05/Oct/22 12:44
Affects Version/s: 3.4.0

[CVE-2022-42003|https://nvd.nist.gov/vuln/detail/CVE-2022-42003]


[Github|https://github.com/FasterXML/jackson-databind/issues/3590]
Bug: Row-level Runtime Filtering cannot be enabled when externalTable has no stats
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Oct/22 02:53
Affects Version/s: 3.3.0

When using external tables, Row-level Runtime Filtering cannot be enabled anyway without performing analysis on the table to generate statistics. In actual use, external tables often do not have statistics, but it is also hoped that the execution efficiency can be improved through Row-level Runtime Filtering. The reason why Row-level Runtime Filtering cannot be enabled is: 'InjectRuntimeFilter' calls the 'satisfyByteSizeRequirement' method to determine whether the application side plan's aggregated scan size meets the requirements, and because there is no statistical data, the application side plan's aggregated scan size is equal to 0, which cannot satisfy the requirement to enable Row -level Runtime Filtering requirements.
In order to enable Row-level Runtime Filtering even when the external table has no statistics, add the RUNTIME_FILTER_{_}ENABLED_WHEN_NO_STATS parameter. When RUNTIME_FILTER{_}_ENABLED_WHEN_NO_STATS is configured to true and the external table has no statistics, the 'satisfyByteSizeRequirement' method returns true so that Row-level Runtime Filtering may be enabled .
Bug: Migrate type check failures of math expressions onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 13:33
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the math expressions:
1. HashExpression (2): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L271-L275
2. RoundBase (1): 
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L1484
Bug: spark-shell fails with "bad array subscript" in cygwin or msys bash session
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/22 20:15
Affects Version/s: 3.3.0

A spark pull request [spark PR|https://github.com/apache/spark/pull/38167] fixes this issue, and also fixes a build error that is also related to _*cygwin*_  and *msys/mingw* bash *sbt* sessions.

If a Windows user tries to start a *_spark-shell_* session by calling the bash script (rather than the *_spark-shell.cmd_* script), it fails with a confusing error message.  Script _*spark-class*_ calls _*launcher/src/main/java/org/apache/spark/launcher/Main.java* to_ generate command line arguments, but the launcher produces a format appropriate to the *_.cmd_* version of the script rather than the _*bash*_ version.

The launcher Main method, when called for environments other than Windows, interleaves NULL characters between the command line arguments.   It should also do so in Windows when called from the bash script.  It incorrectly assumes that if the OS is Windows, that it is being called by the .cmd version of the script.

The resulting error message is unhelpful:

 
{code:java}
[lots of ugly stuff omitted]
/opt/spark/bin/spark-class: line 100: CMD: bad array subscript
{code}
The key to _*launcher/Main*_ knowing that a request is from a _*bash*_ session is that the _*SHELL*_ environment variable is set.   This will normally be set in any of the various Windows shell environments ({_}*cygwin*{_}, {_}*mingw64*{_}, {_}*msys2*{_}, etc) and will not normally be set in Windows environments.   In the _*spark-class.cmd*_ script, _*SHELL*_ is intentionally unset to avoid problems, and to permit bash users to call the _*.cmd*_ scripts if they prefer (it will still work as before).
Bug: Async log purge
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/22 23:13
Affects Version/s: 3.4.0

Purging old entries in both the offset log and commit log will be done asynchronously.

 

For every micro-batch, older entries in both offset log and commit log are deleted. This is done so that the offset log and commit log do not continually grow.  Please reference logic here

 

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L539] 

 

The time spent performing these log purges is grouped with the “walCommit” execution time in the StreamingProgressListener metrics.  Around two thirds of the “walCommit” execution time is performing these purge operations thus making these operations asynchronous will also reduce latency.  Also, we do not necessarily need to perform the purges every micro-batch.  When these purges are executed asynchronously, they do not need to block micro-batch execution and we don’t need to start another purge until the current one is finished.  The purges can happen essentially in the background.  We will just have to synchronize the purges with the offset WAL commits and completion commits so that we don’t have concurrent modifications of the offset log and commit log.
Bug: Check error classes in DDLParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 11:08
Affects Version/s: 3.4.0

Check error classes in DDLParserSuite by using checkError(). For instance, replace

{code:scala}
intercept("CREATE TABLE my_tab (id bigint) SKEWED BY (id) ON (1,2,3)",
      "CREATE TABLE ... SKEWED BY")
{code}
by
{code:scala}
checkError(
      exception = parseException("CREATE TABLE my_tab (id bigint) SKEWED BY (id) ON (1,2,3)"),
      errorClass = "...",
      parameters = Map.empty,
      context = ...)
{code}
at https://github.com/apache/spark/blob/1a1341910249d545365ba3d6679c6943896dde22/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala#L546
Bug: Migrate type check failures of interval expressions onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 15:19
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the interval expressions:
1. Average (1):
https://github.com/apache/spark/blob/47d119dfc1a06ee2d520396129b4f09bc22d3fb7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala#L78
2. ApproxCountDistinctForIntervals (3):
https://github.com/apache/spark/blob/08123a3795683238352e5bf55452de381349fdd9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala#L80-L91
Bug: Migrate type check failures of string expressions onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 15:03
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the string and regexp expressions:
1. Elt (3):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L276-L284
2. RegExpReplace (2):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L597-L604
Bug: CVE-2022-42889 upgrade commons text library to 1.10.0 in spark 3.0.0
Status: Closed | Priority: Major | Resolution: Invalid
Created: 20/Oct/22 19:35
Affects Version/s: 3.0.0

Hi Team,

 

We use spark-core_2.12:3.0.0  which has transitive dependency on commons-text 1.6 and this is flagged as CVE-2022-42889.

 

We have our spark application built using maven using spark-core_2.12:3.0.0. 

Need clarifications on below :
 * Does spark-core use StringSubstitutor and do we need to worry about this?
 * If its getting used , then which lib or code within spark core triggers it ?
 * can we include the apache commons text 1.10.0 as explicit dependency on our POM and add common text 1.6 in exclusions for spark-core , will it work ?
 * Upgrading the another spark version which may have commons text upgraded to 1.10.0 is not feasible and big task for us considering all dependent application using 3.0.0 version
Bug: Avoid introducing too many partitions when bucketed scan disabled by sql planner
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Oct/22 09:53
Affects Version/s: 3.4.0

For bucket tables with huge size, lots of partitions maybe generated if bucketed scan disabled by sql planner. We can add one limit(default as BUCKETING_MAX_BUCKETS) to reduce the partitions for non-bucketed scan to avoid too many tasks introduced in single stage.
Bug: Migrate type check failures of bloom_filter_agg() onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/22 07:01
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in bloom_filter_agg():
https://github.com/apache/spark/blob/1f4e4c812a9dc6d7e35631c1663c1ba6f6d9b721/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala#L66-L76
Bug: Support Keytab file passing as a k8s secret
Status: Open | Priority: Major | Resolution: 
Created: 19/Oct/22 05:14
Affects Version/s: 3.3.0

Currently I can use 
{code:java}
--conf spark.kerberos.keytab=local://{code}
It would be useful if I can make keytab file as a kubernetes secret and then reference the secret here instead of keytab file. Keytab file needs to be mounted using templates and it gets to be tedious needlessly when running driver on cluster mode.

 

I can help if directed on how to get this done.
Bug: Add additional checkpoint rename file check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/22 03:14
Affects Version/s: 3.4.0

We encountered an issue recently that one customer's structured streaming job failed to read delta file. 

The temporary file exists but it was not successfully renamed to final delta file path. 

We currently don't check if renamed file exists but assume it successful. As the result, failing to read delta file assumed to be committed in last batch makes re-triggering the job impossible. 

We should be able to do a check against checkpoint renamed file to prevent such difficulty in advance.
Bug: to_utc_timestamp creates null column
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Oct/22 17:22
Affects Version/s: 3.2.0

We are in the process of migrating our PySpark applications from Spark version 3.1.2 to Spark version 3.2.0. 

This bug is present in version 3.2.0. We do not see this issue in version 3.1.2.

 

*Minimal example to reproduce bug*

Below is a minimal example of applying to_utc_timestamp() on String column with timestamp data
{code:java}
from pyspark.sql.types import StringType
from pyspark.sql.functions import *

# Source data
columns = ["id","timestamp_field"]
data = [("1", "2022-10-17T00:00:00+0000"), ("2", "2022-10-17T00:00:00+0000")]
source_df = spark.createDataFrame(data).toDF(*columns)
source_df.createOrReplaceTempView("source")
print("Source:")
print(source_df.show())

# Execute query
query = """
SELECT
    id, 
    timestamp_field as original,  
    to_utc_timestamp(timestamp_field, 'UTC')     AS received_timestamp
FROM source
"""
df = spark.sql(query)
print("Transformed:")
print(df.show())
print(df.count()) {code}
*Post Execution*

The source data has a column called _timestamp_field_ which is a string type.
{code:java}
Source:
+---+--------------------+                                                      
| id|     timestamp_field|
+---+--------------------+
|  1|2022-10-17T00:00:...|
|  2|2022-10-17T00:00:...|
+---+--------------------+
{code}
The query applies to_utc_timestamp() on timestamp_field to create a new column. The new column is null.
{code:java}
Transformed:
+---+--------------------+------------------+
| id|            original|received_timestamp|
+---+--------------------+------------------+
|  1|2022-10-16T00:00:...|              null|
|  2|2022-10-16T00:00:...|              null|
+---+--------------------+------------------+ {code}
–

 

*Questions*
 * Did the to_utc_timestamp function get any new changes in spark version 3.2.0? We don't see this issue in spark 3.1.2
 * Can we apply any spark settings to resolve this?
 * Is there a new preferred function in spark 3.2.0 that replaces to_utc_timestamp?
Bug: Missing handling a catalog name in destination tables in `RenameTableExec`
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Oct/22 14:43
Affects Version/s: 3.1.3

Currently `RenameTableExec` only handles an empty namespace for destination tables as the following current spec:

(3.3.0 is picked up) [https://github.com/apache/spark/blob/v3.3.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RenameTableExec.scala#L48]

 
{code:java}
# L48
    val qualifiedNewIdent = if (newIdent.namespace.isEmpty) {
      Identifier.of(oldIdent.namespace, newIdent.name)
    } else newIdent{code}
This part doesn't handle the case where a destination table is specified catalog.db.table. For example, Apache Iceberg is expected to handle the destination table as catalog.db.table in the document; [https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--rename-to|https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--rename-to.]

 

If catalog.db.table is passed to "ALTER TABLE <src> RENAME TO <dst>" query, there's a difference of handling namespaces between source and destination tables. Specifically, source tables can be correctly handled as *[db]* for its namespace, but destination tables are handled as *[catalog, db]* for its namespace.
Bug: STORED AS serde in CREATE TABLE LIKE view does not work
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/22 07:06
Affects Version/s: 3.2.2

SPARK-29839 added the feat for supporting STORED AS in CREATE TABLE LIKE, and we could use this syntax to create a table with specified serde based on a existing *table or view.*

I found that if we create a table based on a {*}view{*}, the serde of created table is always parquet.  However, if we use *USING syntax*  (SPARK-29421) to create a table with specified serde based a {*}view{*}, we can get the correct serde.

I think we shoud fix this to add specified serde for the created table when using *create table like view* *stored as* syntax.
Bug: Spark 3.3.0 doesn't works with Hive 3.1.2
Status: Reopened | Priority: Major | Resolution: 
Created: 10/Oct/22 17:10
Affects Version/s: 3.3.0

Hive 2.3.9 is impacted with CVE-2021-34538, so trying to use the Hive 3.1.2.

Using Spark 3.3.0 with Hadoop 3.3.4 and Hive 3.1.2, getting below error when starting the Thriftserver

 
{noformat}
Exception in thread "main" java.lang.IllegalAccessError: tried to access class org.apache.hive.service.server.HiveServer2$ServerOptionsProcessor from class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:92)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala){noformat}
Using below command to start the Thriftserver

 

*spark-class org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal*

 

Have set the SPARK_HOME correctly.

 

The same works well with Hive 2.3.9, but fails when we upgrade to Hive 3.1.2.
Bug: Topic-partition in offset deleted before commiting
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/22 22:53
Affects Version/s: 3.2.2

Structured streaming subscribed with "subscribePattern". I see that structured streaming maintains two folders inside checkpoint location, offsets and commits.

When a batch arrives the offset folder gets updated with "topic name:\{partition:offset}" for that batch. And when this offset is properly processed. The commits folder gets updated with the batch id that was just processed.

In our case data was sent to that topic, the offset folder was updated but the topic got deleted before the offset was processed and commit was updated and now we get the message,

org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition topicName-1 could be determined

Even when we restart the stream, it still tries to fetch the same topic-partition and throws this error.

Is there a way to handle this kind of situation?
Bug: Check error classes in DDL parsing tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 12:03
Affects Version/s: 3.4.0

Check error classes in DDL command tests by using checkError(). For instance
 - AlterNamespaceSetPropertiesParserSuite
 - AlterTableDropPartitionParserSuite
 - AlterTableRenameParserSuite
 - AlterTableRecoverPartitionsParserSuite
 - DescribeTableParserSuite
 - TruncateTableParserSuite
 - AlterTableSetSerdeParserSuite
 - ShowPartitionsParserSuite

[https://github.com/apache/spark/blob/414771d4e8b52d0a76a7729d005794dc04f1e075/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterNamespaceSetPropertiesParserSuite.scala#L43]
Bug: Check error classes in PlanParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 11:34
Affects Version/s: 3.4.0

Check error classes in PlanParserSuite by using checkError(). For instance, replace

{code:scala}
intercept("EXPLAIN logical SELECT 1", "Unsupported SQL statement")
{code}
by
{code:scala}
checkError(
      exception = parseException("EXPLAIN logical SELECT 1"),
      errorClass = "...",
      parameters = Map.empty,
      context = ...)
{code}
at https://github.com/apache/spark/blob/35d00df9bba7238ad4f409999617fae4d04ddbfd/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala#L225
Bug:   Add `@ExtendedLevelDBTest` to the leveldb relevant  tests in the yarn module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/22 15:27
Affects Version/s: 3.2.2

SPARK-40490 make  the test case related to `YarnShuffleIntegrationSuite` starts to verify the registeredExecFile reload test scenario again，so we need to add `@ExtendedLevelDBTest` for the test case using LevelDB so that the `MacOs/Apple Silicon` can skip relevant tests through `-Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest`
Bug: Check error classes in ExpressionParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 11:31
Affects Version/s: 3.4.0

Check error classes in ExpressionParserSuite by using checkError(). For instance, replace

{code:scala}
intercept("a like 'pattern%' escape '##'", message)
{code}
by
{code:scala}
checkError(
      exception = parseException("a like 'pattern%' escape '##'"),
      errorClass = "...",
      parameters = Map.empty,
      context = ...)
{code}
at https://github.com/apache/spark/blob/c4a0360daf546baa9cc970ef805f52c40f384fe3/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala#L200
Bug: Migrate type check failures of percentile expressions onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 15:30
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the percentile expressions:
1. Percentile (4):
https://github.com/apache/spark/blob/3f3201a7882b817a8a3ecbfeb369dde01e7689d8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L122-L129
2. PercentileBase (3):
https://github.com/apache/spark/blob/8559f88f8ed9d22751975150a6d5735653a1e528/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala#L87-L93