av9ash/GitBugs huggingface dataset:
Bug: ANSI SQL mode: use plain string representation on casting Decimal to String
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/22 02:25
Affects Version/s: 3.4.0

Currently, casting decimal as string type will result in Strings with exponential notations if the adjusted exponent is less than -6. This is consistent with BigDecimal.toString [https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#toString] 

This is different from external databases like PostgreSQL/Oracle/MS SQL server. It doesn't compliant with the ANSI SQL standard either. 

-I suggest always using the plain string in the casting.-

After discussions, I decide to put the new behavior on ANSI SQL mode.
Bug: Introduce parametric singleton for DataType
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jul/22 17:40
Affects Version/s: 3.4.0

As per [https://github.com/apache/spark/pull/37173#discussion_r921990449|https://github.com/apache/spark/pull/37173#discussion_r921990449,] ,

we ought to make `DataTypeSingleton` parametric.

Otherwise, we will create a new DataType instance (with parameters in constructors, e.g. `Varchar`, `DecimalTime`) every time it's called.
Bug: How to treat/eliminate CVE-2021-4048 (reported for arpack_combined_all-0.1.jar)
Status: Open | Priority: Major | Resolution: 
Created: 15/Jul/22 14:49
Affects Version/s: 3.3.0

The following CVE is reported for arpack_combined_all-0.1.jar which is used in  org.apache.spark:spark-graphx_2.13 which in turn is used in mllib : [https://nvd.nist.gov/vuln/detail/CVE-2021-4048]

Questions: how relevant is this issue, can it be safely ignored?

It seems that arpack_combined_all-0.1.jar is really needed because when removing it from the CLASSPATH, a NoClassDefFoundError: org/netlib/blas/Sdot is reported.
Bug: Fix performance gap and inconsistent behavior between using Scala 2.12 and Scala 2.13
Status: Open | Priority: Major | Resolution: 
Created: 15/Jul/22 02:31
Affects Version/s: 3.4.0

After SPARK-39576, we can run benchmark GitHub Actions to to generate Scala 2.13 benchmark results, so that we can compare with the result of Scala 2.12 one by one to find the performance gap and fix them.

But if the performance gap is bug of Scala 2.13, we should report it to the Scala community and try to find a workaround way in Spark. If the workaround way not exists due we need to support two Scala versions now, we can left TODO and fix them after Spark only supports Scala 2.13.

On the other hand, by comparing the UTs or run Scala 2.13 compiled Spark in the production environment, we may find some behavioral differences, they can also be classified as subTasks of the current umbrella.
Bug: Support adding maven packages while pip
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/22 10:03
Affects Version/s: 3.3.0

The goal is to support adding maven packages to a pip installable package, including adding jars and resolvers, so that when spark gets booted up it can automatically look for the maven packages in the classpath and install corresponding dependencies.

 

This idea comes up because currently for a python package, which depends on jars like pyspark internally use reflection on spark source code, if we want to make it work, there're two steps: 1. pip install the python package. 2. Add the jar into spark configuration while we start spark session, for example through spark.jars.packages.

If we can support the proposed functionality, we could ideally just add the package name and resolver while we pip install the package, and when spark session starts, it can look for those configurations inside python classpath and install them if they're not existed. This will simplify the process of all python developers who internally depends on maven packages and make pyspark more user-friendly.
Bug: NPE on invalid patterns from the regexp functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/22 06:30
Affects Version/s: 3.3.0

The example below reproduces the issue:

{code:sql}
spark-sql> SELECT regexp_extract('1a 2b 14m', '(?l)');
22/07/12 19:07:21 ERROR SparkSQLDriver: Failed in [SELECT regexp_extract('1a 2b 14m', '(?l)')]
java.lang.NullPointerException: null
	at org.apache.spark.sql.catalyst.expressions.RegExpExtractBase.getLastMatcher(regexpExpressions.scala:768) ~[spark-catalyst_2.12-3.3.0.jar:3.3.0]
{code}
Bug: NotExists subquery failed with conflicting attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/22 13:28
Affects Version/s: 3.1.3

{code:sql}
select * from
(
select v1.a, v1.b, v2.c
from v1
inner join v2
on v1.a=v2.a) t3
where not exists (
  select 1
  from v2
  where t3.a=v2.a and t3.b=v2.b and t3.c=v2.c
){code}
This query throws AnalysisException
{code:java}
org.apache.spark.sql.AnalysisException: Found conflicting attributes a#266 in the condition joining outer plan:
  Join Inner, (a#250 = a#266)
:- Project [_1#243 AS a#250, _2#244 AS b#251]
:  +- LocalRelation [_1#243, _2#244, _3#245]
+- Project [_1#259 AS a#266, _3#261 AS c#268]
   +- LocalRelation [_1#259, _2#260, _3#261]and subplan:
  Project [1 AS 1#273, _1#259 AS a#266, _2#260 AS b#267, _3#261 AS c#268#277]
+- LocalRelation [_1#259, _2#260, _3#261] {code}
Bug: Include the origin logical plan for LogicalRDD if it comes from DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/22 01:45
Affects Version/s: 3.4.0

When Spark converts the DataFrame to LogicalRDD for some reason (e.g. foreachBatch sink), Spark just picks the RDD from the origin DataFrame and discards the (logical/physical) plan.

The origin logical plan can be useful for several use cases, including:

1. wants to connect the overall logical plan into one
2. inherits plan stats from origin logical plan
Bug: Prune filter condition if compare with rand is deterministic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/22 04:27
Affects Version/s: 3.4.0

The SQL

{code:java}
SELECT * FROM tab WHERE rand(1) < 2
{code}

evaluate rand(1) < 2 for rows one by one.
In fact, we can prune the filter condition.
Bug: Add the REGEXP_INSTR function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 10:55
Affects Version/s: 3.4.0

The function should return the position of the specified occurrence of the regular expression pattern in the input string. If no match is found, returns 0. See other DBMSs:
 - MariaDB: [https://mariadb.com/kb/en/regexp_instr/]
 - Oracle: [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions129.htm]
 - DB2: [https://www.ibm.com/docs/en/db2/11.5?topic=functions-regexp-instr]
 - Snowflake: [https://docs.snowflake.com/en/sql-reference/functions/regexp_instr.html]
 - BigQuery: [https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_instr]
 - Redshift: [https://docs.aws.amazon.com/redshift/latest/dg/REGEXP_INSTR.html]
 - Exasol DB: [https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/regexp_instr.htm]
- Vertica: [https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/RegularExpressions/REGEXP_INSTR.htm]
Bug: Binary array operations can be faster if one side is a constant
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/22 17:24
Affects Version/s: 3.3.0

Array operations such as [ArraysOverlap|https://github.com/apache/spark/blob/79f133b7bbc1d9aa6a20dd8a34ec120902f96155/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L1367] are optimized to put all the elements of the smaller array into a HashSet, if elements properly support equals. 
However, if one of the arrays is a constant, we could do much better as we don't have to reconstruct the HashSet for each row, we could construct it just once and send it to all the executors. This would improve runtime by a constant factor.
Bug: Add the REGEXP_SUBSTR function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/22 11:49
Affects Version/s: 3.4.0

The function should search a string for a regular expression pattern and returns it or NULL of it is not found. See other DBMSs:
- Oracle: https://docs.oracle.com/cd/B12037_01/server.101/b10759/functions116.htm
- DB2: https://www.ibm.com/docs/en/db2/11.5?topic=functions-regexp-substr
- Snowflake: https://docs.snowflake.com/en/sql-reference/functions/regexp_substr.html
- BigQuery: https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_substr
- Redshift: https://docs.aws.amazon.com/redshift/latest/dg/REGEXP_SUBSTR.html
- MariaDB: https://mariadb.com/kb/en/regexp_substr/
- Exasol DB: https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/regexp_substr.htm
Bug: spark-core: sonatype-2021-1215 & sonatype-2021-1216 vulnerabilities from com.twitter:chill
Status: Open | Priority: Major | Resolution: 
Created: 10/Jul/22 23:06
Affects Version/s: 3.1.3

Our static application security test showed that Spark Core has the following vulnerabilities due to a transitive dependency on com.esotericsoftware:kryo-shaded@4.0.2 via com.twitter:chill@0.10.0.

[34733 - kryo:DeserializeStringFuzzer: Uncaught exception in com.esotericsoftware.kryo.serializers.FieldSerializer.read - oss-fuzz (chromium.org)|https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34733]

[34646 - kryo:DeserializeCollectionsFuzzer: Uncaught exception with empty stacktrace - oss-fuzz (chromium.org)|https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34646]

Both vulnerabilities were addressed in kryo@5..3.0+. Once chill has been upgraded ([Upgrade com.esotericsoftware:kryo-shaded:jar:4.0.2 to avoid security risk · Issue #665 · twitter/chill (github.com)|https://github.com/twitter/chill/issues/665]), Spark Core should be updated as well.
Bug: Reduce memory overhead of TransportCipher$EncryptedMessage's byteRawChannel buffer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/22 01:05
Affects Version/s: 3.0.0

With Spark's encryption enabled ({{{}spark.network.crypto.enabled{}}} set to {{true}} and {{spark.network.crypto.saslFallback}} set to {{{}false{}}}), I ran into memory usage problems in the external shuffle service. 

This was caused by a problem that is very similar to SPARK-24801: each {{TransportCipher$EncryptedMessage}} eagerly initializes a buffer that is used during the encryption process. This buffer is only used once {{transferTo}} is called, but it is eagerly initialized in the {{EncryptedMessage}} constructor. This leads to high memory usage when there are many messages queued in an outgoing channel.

One possible fix would be to mimic SPARK-24801 and make the initialization lazy. However, we can actually go one step further and share a single re-used buffer across multiple messages. This is safe because those messages are _already_ sharing a different buffer which is accessed in the same write paths.
Bug: Support push topK through outer join
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jul/22 12:39
Affects Version/s: 3.4.0

Supports push down local limit and local sort from TopK through other join:
  - for a left outer join, the references of ordering of TopK come from the left side and
  the limits of TopK is smaller than left side 
  - for a right outer join, the references of ordering of TopK come from the right side and
  the limits of TopK is smaller than right side
Bug: TakeOrderedAndProjectExec should respect child output ordering
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/22 09:23
Affects Version/s: 3.4.0

TakeOrderedAndProjectExec should respect child output ordering to avoid unnecessary sort. For example:  TakeOrderedAndProjectExec on the top of SortMergeJoin.
{code:java}
SELECT * FROM t1 JOIN t2 ON t1.c1 = t2.c2 ORDER BY t1.c1 LIMIT 100;
{code}
Bug: Did not find value which can be converted into java.lang.String
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/22 11:29
Affects Version/s: 3.2.0

Hi, I have a problem with loading the ALS model in pyspark. I wrote it in a detail at this link. Could you help me with it? Thanks.

 

[https://stackoverflow.com/questions/72868619/apache-spark-loading-als-model-did-not-find-value-which-can-be-converted-into]
Bug: Reuse exchange across subqueries is broken with AQE if subquery side exchange materialized first
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jul/22 05:46
Affects Version/s: 3.3.0

When trying to reuse Exchange of a subquery in main plan, if the Exchange inside subquery materialize first then main ASPE node won't have that stage info (in [stageToReplace|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L243]) to replace in current logical plan. This will cause AQE to produce new candidate physical plan without reusing the exchange present inside subquery. And depending on how complex the inner plan is (no. of exchanges) AQE could choose plan without ReusedExchange. 

We have seen with multiple queries with our private build. This can happen in DPP also.
Bug: Streaming Deduplication should not check the schema of "value"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/22 04:17
Affects Version/s: 3.1.2

When we use dropDuplicate() in the streaming query, specifying the columns explicitly would perform deduplication against the columns rather than all columns. 

For the structure of state in streaming deduplication, we construct the key from "specified" columns and value as empty row (since it's not used at all). That said, once the query specifies the columns in dropDuplicate(), all other columns should not affect the operation.

Unfortunately, even we use the empty row as value of the state store, we register the "all columns" as the schema for the value on state store, which leads incorrect behavior from checking schema for state store. (This is figured out as a long-standing issue, it's from the initial implementation of StreamingDeduplicateExec.)

Specifically, columns for DataFrame which is applied to streaming deduplicate should be same across the lifetime of the query, whereas the only specified columns should be same actually.

It would be ideal to change the value schema to be empty, but the change itself may not be sufficient since schema file has been already written for older streaming queries. We may need to allow state schema compatibility checker to ignore value schema if required (either config or parameter of method if feasible).
Bug: Unable to set zstd compression level while writing parquet files
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jul/22 10:06
Affects Version/s: 3.2.0

While writing zstd compressed parquet files, the following setting `spark.io.compression.zstd.level` does not have any affect with regards to the compression level of zstd.

All files seem to be written with the default zstd compression level, and the config option seems to be ignored.

Using the zstd cli tool, we confirmed that setting a higher compression level for the same file tested in spark resulted in a smaller file.
Bug: DataFrameReader API cannot read files from hadoop archives (.har)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jul/22 12:52
Affects Version/s: 3.0.3

Reading a file from an hadoop archive using the DataFrameReader API returns an empty Dataset:
{code:java}
scala> val df = spark.read.textFile("har:///user/preaudc/logs/lead/jp/2022/202207.har/20220719")
df: org.apache.spark.sql.Dataset[String] = [value: string]
scala> df.count
res7: Long = 0 {code}
 

On the other hand, reading the same file, from the same hadoop archive, but using the RDD API yields the correct result:
{code:java}
scala> val df = sc.textFile("har:///user/preaudc/logs/lead/jp/2022/202207.har/20220719").toDF("value")
df: org.apache.spark.sql.DataFrame = [value: string]
scala> df.count
res8: Long = 5589 {code}
Bug: Rename trait Unevaluable
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jul/22 17:56
Affects Version/s: 3.3.0

I came upon `trait Unevaluable` which is defined in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala

Unevaluable is not a word.

There are `valuable`, `invaluable` but I have never seen Unevaluable.

This issue renames the trait to Unevaluatable
Bug: Provides a good error during create Index with different dtype elements
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Jul/22 07:15
Affects Version/s: 3.2.2

PANDAS

 
{code:java}
>>> import pandas as pd >>> pd.Index([1,2,'3',4]) Index([1, 2, '3', 4], dtype='object') >>> 
 {code}
PYSPARK

 

 
{code:java}
Using Python version 3.8.13 (default, Jun 29 2022 11:50:19)
Spark context Web UI available at http://172.25.179.45:4042
Spark context available as 'sc' (master = local[*], app id = local-1658301116572).
SparkSession available as 'spark'.
>>> from pyspark import pandas as ps
WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
>>> ps.Index([1,2,'3',4])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/indexes/base.py", line 184, in __new__
    ps.from_pandas(
  File "/home/spark/spark/python/pyspark/pandas/namespace.py", line 155, in from_pandas
    return DataFrame(pd.DataFrame(index=pobj)).index
  File "/home/spark/spark/python/pyspark/pandas/frame.py", line 463, in __init__
    internal = InternalFrame.from_pandas(pdf)
  File "/home/spark/spark/python/pyspark/pandas/internal.py", line 1469, in from_pandas
    ) = InternalFrame.prepare_pandas_frame(pdf, prefer_timestamp_ntz=prefer_timestamp_ntz)
  File "/home/spark/spark/python/pyspark/pandas/internal.py", line 1570, in prepare_pandas_frame
    spark_type = infer_pd_series_spark_type(reset_index[col], dtype, prefer_timestamp_ntz)
  File "/home/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 360, in infer_pd_series_spark_type
    return from_arrow_type(pa.Array.from_pandas(pser).type, prefer_timestamp_ntz)
  File "pyarrow/array.pxi", line 1033, in pyarrow.lib.Array.from_pandas
  File "pyarrow/array.pxi", line 312, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert '3' with type str: tried to convert to int64
 {code}
I understand that pyspark pandas need the dtype to be the same, but we need a good error msg or something to tell the user how to avoid.
Bug: Use AmazonKinesisClientBuilder.withCredentials instead of new AmazonKinesisClient(credentials)
Status: In Progress | Priority: Minor | Resolution: 
Created: 19/Jul/22 05:41
Affects Version/s: 3.4.0

[warn] /home/runner/work/spark/spark/connector/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala:108:25: [deprecation @ org.apache.spark.examples.streaming.KinesisWordCountASL.main.kinesisClient | origin=com.amazonaws.services.kinesis.AmazonKinesisClient.<init> | version=] constructor AmazonKinesisClient in class AmazonKinesisClient is deprecated

[warn] /home/runner/work/spark/spark/connector/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala:224:25: [deprecation @ org.apache.spark.examples.streaming.KinesisWordProducerASL.generate.kinesisClient | origin=com.amazonaws.services.kinesis.AmazonKinesisClient.<init> | version=] constructor AmazonKinesisClient in class AmazonKinesisClient is deprecated

[warn] /home/runner/work/spark/spark/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala:142:24: [deprecation @ org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.client | origin=com.amazonaws.services.kinesis.AmazonKinesisClient.<init> | version=] constructor AmazonKinesisClient in class AmazonKinesisClient is deprecated

[warn] /home/runner/work/spark/spark/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:58:18: [deprecation @ org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient.client | origin=com.amazonaws.services.kinesis.AmazonKinesisClient.<init> | version=] constructor AmazonKinesisClient in class AmazonKinesisClient is deprecated
Bug: Use ArrowType.Decimal(precision, scale, bitWidth) instead of ArrowType.Decimal(precision, scale)
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Jul/22 07:35
Affects Version/s: 3.4.0

[warn] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:48:49: [deprecation @ org.apache.spark.sql.util.ArrowUtils.toArrowType | origin=org.apache.arrow.vector.types.pojo.ArrowType.Decimal.<init> | version=] constructor Decimal in class Decimal is deprecated
Bug: vis-timeline @ 4.2.1 vulnerable to XSS attacks
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jul/22 05:59
Affects Version/s: 3.2.1

Spark UI includes visjs/vis-timeline package@4.2.1, which is vulnerable to XSS attacks ([Cross-site Scripting in vis-timeline · CVE-2020-28487 · GitHub Advisory Database|https://github.com/advisories/GHSA-9mrv-456v-pf22]). This version should be replaced with the next non-vulnerable issue - [Release v7.4.4 · visjs/vis-timeline (github.com)|https://github.com/visjs/vis-timeline/releases/tag/v7.4.4] or higher.