av9ash/GitBugs huggingface dataset:
Bug: Move ArrowColumnVectorSuite to org.apache.spark.sql.vectorized
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Feb/22 20:21
Affects Version/s: 3.2.2

Currently ArrowColumnVector is under org.apache.spark.sql.vectorized. However, ArrowColumnVectorSuite is under org.apache.spark.sql.execution.vectorized.

 

Proposing to move ArrowColumnVectorSuite to org.apache.spark.sql.vectorized so that the package names match
Bug: Ability to turn off auto commit in JDBC source for read only operations
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Feb/22 12:46
Affects Version/s: 3.2.1

Currently, all the jdbc connections on executors side work always with auto commit option set to true.

However, there are cases where this mode makes hard to use JdbcRelationProvider at all, i.e. reading huge datasets from Postgres (a whole result set is collected regardless of a fetch size when autocommit is set to true https://jdbc.postgresql.org/documentation/91/query.html#query-with-cursor )

So the proposal is following:
 # Add a boolean option "autocommit" to JDBC Source allowing a user to turn off autocommit mode for read only operations.
 # Add guards which prevent using this option in DML operations.
Bug: select doesnt validate if the column already exists
Status: Open | Priority: Minor | Resolution: 
Created: 02/Feb/22 06:32
Affects Version/s: 3.2.1

 

Select doesnt validate whether the alias column is already present in the dataframe. 

After which, we cannot do anything in that dataframe on that column. 
df4 = df2.select(df2.firstname, df2.lastname) --> throws analysis exception
df4.show()
 

However drop will not let you drop the said column. 

 

Scenario to reproduce :
df2 = df1.select("*", (df1.firstname).alias("firstname"))   ---> this will add same column
df2.show() 
df2.drop(df2.firstname) --> this will give AnalysisException: Reference 'firstname' is ambiguous, could be: firstname, firstname.
 

 

Is this expected behavior .
  !select vs drop.png!
!image-2022-02-02-06-28-23-543.png!
Bug: pyspark DataFrame.replace() is sensitive to column name case in pyspark 3.2 but not in 3.1
Status: Open | Priority: Minor | Resolution: 
Created: 04/Feb/22 16:37
Affects Version/s: 3.2.0

The `subset` argument for `DataFrame.replace()` accepts one or more column names. In pyspark 3.2 the case of the column names must match the column names in the schema exactly or the replacements will not take place. In earlier versions (3.1.2 was tested) the argument is case insensitive.

Minimal example:

{{replace_dict = \{'wrong': 'right'}}}

{{df = spark.createDataFrame(}}
{{  [['wrong', 'wrong']], }}
{{  schema=['case_matched', 'case_unmatched']}}
{{)}}
{{df2 = df.replace(replace_dict, subset=['case_matched', 'Case_Unmatched'])}}

 

In pyspark 3.2 (tested 3.2.0, 3.2.1 via pip on windows and 3.2.0 on Databricks) the result is:
|case_matched|case_unmatched|
|right|wrong|

While in pyspark 3.1 (tested 3.1.2 via pip on windows and 3.1.2 on Databricks) the result is:
|case_matched|case_unmatched|
|right|right|

I believe the expected behaviour is that shown in pyspark 3.1 as in all other situations column names are accepted in a case insensitive manner.
Bug: Upgrade h2 from 2.0.204 to 2.1.210 in /sql/core
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Feb/22 12:02
Affects Version/s: 3.3.0

H2 have 2 CVE security issues.

[Arbitrary code execution in H2 Console|https://github.com/advisories/GHSA-h376-j262-vhq6]

[RCE in H2 Console|https://nvd.nist.gov/vuln/detail/CVE-2021-42392]
  
If we change sql/core/pom.xml

line 156 from <version>2.0.204</version> to <version>2.1.210</version>

This will build and all test are passed.
Bug: How do I get a lot of results in KDE
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 16/Feb/22 02:45
Affects Version/s: 2.4.5

I have a pyspark.DataFrame, I have converted one of the columns to RDD and performed KDE, I need to get all the KDE estimates of the column and add a new column in the DataFrame for subsequent work, how can I do it by Spark?
Bug: Check existence of file before untarring/zipping
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Feb/22 16:29
Affects Version/s: 3.2.1

Some discussion related to Hadoop revealed that its FileUtil.unTar and unZip methods might have a problem if the source file does not actually exist.

It's trivial to check for this in the one place we use it in Spark, and just explicitly error if the source file does not exist.
Bug: Update comments of ExecutorAllocationClient
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 21/Feb/22 01:08
Affects Version/s: 3.2.1

The class comment of ExecutorAllocationClient is out of date.
{code:java}
This is currently supported only in YARN mode. {code}
Nowadays, this is supported in the following modes: Spark's Standalone, YARN-Client, YARN-Cluster, Mesos, Kubernetes.

 

In my opinion, this comment should be updated.
Bug: Add hash probes metrics for shuffled hash join
Status: In Progress | Priority: Trivial | Resolution: 
Created: 28/Feb/22 17:17
Affects Version/s: 3.3.0

For `HashAggregate` there's a SQL metrics to track number of hash probes per looked-up key. It would be better to add a similar metrics for shuffled hash join as well, to get some idea of hash probing performance.
Bug: Replace pattern matches on boolean expressions with conditional statements
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Feb/22 03:52
Affects Version/s: 3.3.0

Before

 
{code:java}
val bool: Boolean
  bool match {
    case true => do something when bool is true
    case false => do something when bool is false
  } {code}
 

 

After

 
{code:java}
val bool: Boolean
  if (bool) {     
    do something when bool is true   
  } else {     
    do something when bool is false   
  } {code}
Bug: Unintended exception thrown in pyspark.ml.LogisticRegression.getThreshold
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 17/Feb/22 15:48
Affects Version/s: 2.4.0

If {{LogisticRegression.getThreshold}} is called with model having multiple thresholds we suppose to raise an exception,
{code:python}
ValueError: Logistic Regression getThreshold only applies to binary classification ...
{code}
However, {{thresholds}} ({{{}List[float]{}}}) are incorrectly passed to {{{}str.join{}}}, resulting in unintended {{TypeError}}


{code:python}
>>> from pyspark.ml.classification import LogisticRegression
... 
... model = LogisticRegression(thresholds=[1.0, 2.0, 3.0])
>>> model.getThreshold()
Traceback (most recent call last):
  Input In [7] in <module>
    model.getThreshold()
  File /path/to/spark/python/pyspark/ml/classification.py:1003 in getThreshold
    + ",".join(ts)
Type Error: sequence item 0: expected str instance, float found

{code}
Bug: Fix KubernetesUtils#uploadFileToHadoopCompatibleFS use passed in `delSrc` and `overwrite`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Feb/22 08:28
Affects Version/s: 3.3.0

KubernetesUtils#uploadFileToHadoopCompatibleFS defines the input parameters `

delSrc` and `overwrite`,  but constants(false and true) are used when call `

FileSystem.copyFromLocalFile(boolean delSrc, boolean overwrite, Path src, Path dst) ` method.

`
Bug: FoldablePropagation rule and ConstantFolding rule leads to wrong aggregated result
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 22/Aug/24 05:44
Affects Version/s: 3.2.1

{code:java}
SELECT  cast(-1 AS BIGINT) AS ele1
FROM    (            
       SELECT  array(1, 5, 3, 123, 255, 546, 64, 23) AS t        
   ) LATERAL VIEW explode(t) tmp AS ele
WHERE   ele=-1 {code}
This query returns an empty result. However, the following query returns 1.  This result seems wrong.
{code:java}
SELECT  count(DISTINCT ele1)
FROM    (            
    SELECT  cast(-1 as bigint) as ele1            
    FROM    (                        
        SELECT  array(1, 5, 3, 123, 255, 546, 64, 23) AS t                       
        ) LATERAL VIEW explode(t) tmp AS ele            
    WHERE   ele = -1        
) {code}
By plan change log, I find that it is FoldablePropagation rule and ConstantFolding rule that optimize Aggregate expression to `Aggregat [[cast(count(distinct -1) as string) AS count(DISTINCT ele)#7|#7]] ]`.

 

Is this result right?  Does it need to be fixed?
Bug: Deprecate SaprkR
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/Aug/24 04:11
Affects Version/s: 4.0.0

Deprecate SparkR
- Discussion: https://lists.apache.org/thread/qjgsgxklvpvyvbzsx1qr8o533j4zjlm5
- Vote: https://lists.apache.org/thread/3c8qxks26kqflsjh0gtjo3nldk686vtq
Bug: Use API Group `spark.apache.org`
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 17/Aug/24 21:43
Affects Version/s: kubernetes-operator-0.1.0

K8s convention follows domain name styles instead of package name styles.
{code}
-apiVersion: org.apache.spark/...
+apiVersion: spark.apache.org/...
{code}
Bug: Redact `Spark Command` output in `launcher` module
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Aug/24 01:54
Affects Version/s: 3.4.3

When `launcher` module shows `Spark Command`, there is no redaction. Although Spark Cluster is supposed to be in a secure environment, this could be collected by a centralized log system. We need to do a proper redaction like we do for `Launch Command:` output.
{code:java}
$ SPARK_NO_DAEMONIZE=1 SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=" sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/logs/spark-dongjoon-org.apache.spark.deploy.master.Master-1-M3-Max.local.out
Spark Command: /Users/dongjoon/.jenv/versions/17/bin/java -cp /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/conf/:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/slf4j-api-2.0.13.jar:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/* -Dspark.master.rest.enabled=true -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4= -Xmx1g org.apache.spark.deploy.master.Master --host M3-Max.local --port 7077 --webui-port 8080
{code}
Bug: Non-daemon timer thread prevents Spark driver JVM from stopping
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/24 04:46
Affects Version/s: 3.5.2

It is observed that when using [Spark Torch Distributor|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.torch.distributor.TorchDistributor.html], Spark driver pod could hang around after calling {_}spark.stop(){_}. Although the Spark Context was shutdown, the JVM was still running.

The reason was that there is a non-daemon timer thread named {_}BarrierCoordinator barrier epoch increment timer{_}, which prevented the driver JVM from stopping. In SPARK-46895 we replaced the timer with non-daemon single thread scheduled executor, but the issue still exists.

We should use daemon single thread scheduled executor instead.
Bug: Remove assert when translating expressions in DataSourceStrategy when
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/24 13:21
Affects Version/s: 4.0.0

In `DataSourceV2Strategy`, when we translate filters, we initialise `V2ExpressionBuilder` and assert that return type of translation is instance of Predicate.

Given that `V2ExpressionBuilder` does not have compiler gurantees of returning Predicates, this makes it very easy to create bugs, and make regressions.

E.g. we add a new translation for expression which returns boolean, but forget to inherit it from predicate. Now suddenly queries that have this expression in their filter will start failing - because we will try to push it down and assert will fail.

Instead of this approach - we should have option to swallow and absorb bugs from V2ExpressionBuilder by returning `None` whenever we encounter expression which is Predicate.

This way, it is as if we are saying:

If translated expression is not of type `Predicate` act as if we didn't know how to translate expression for this particular example.
Bug: Refactor plotting implementations to support connect
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 09/Aug/24 10:00
Affects Version/s: 4.0.0

1, Some plots depends on MLlib and don't work with Spark Conenct. We need to reimplement ML-based plots with Spark SQL, so they can be compatible with Spark Connect;

2, Further computation optimization, e.g.:
 * compute all necessary metrics for some plots in single pass of the whole dataset, so we can improve the performance;
 * optimize existing sampling algorithm
Bug: Rename/Reorg Connect Client Packages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/24 01:46
Affects Version/s: 4.0.0

Rename sql/connect/common to sql/connect/client

Move connector/connect/client/jvm to sql/connect/client-standalone

Move the remaining connector/connect directories (bin and docs) to sql/connect.
Bug: 0000-01-01 is a valid date
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 15/Aug/24 14:02
Affects Version/s: 3.4.3


{code:java}
scala> val df = spark.sql("SELECT date_add('0001-01-01', -2)")
df: org.apache.spark.sql.DataFrame = [date_add(0001-01-01, -2): date]


scala> df.show()
+------------------------+
|date_add(0001-01-01, -2)|
+------------------------+
|              0000-12-30|
+------------------------+


scala> val df = spark.sql("SELECT date_add('0000-01-01', -2)")
df: org.apache.spark.sql.DataFrame = [date_add(0000-01-01, -2): date]

scala> df.show()
+------------------------+
|date_add(0000-01-01, -2)|
+------------------------+
|             -0001-12-30|
+------------------------+
{code}
year 0 is not legal, it never existed
Bug: Communicate RocksDB State Store CheckpointID Between Driver and Executor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/24 19:16
Affects Version/s: 4.0.0

A incremental step to implement RocksDB state store checkpoint format V2.

Once conf STATE_STORE_CHECKPOINT_FORMAT_VERSION is set to be higher than version 2, the executor returns checkpointID to the driver (only done for RocksDB). The driver stores is locally. For the next batch, the checkpointID is sent to the executor to be used to load the state store. If the local version of the executor doesn't match the uniqueID, it will reload from the checkpoint.
Bug: Avoid redundant array transform for identical expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/24 06:41
Affects Version/s: 3.3.4

Our customer encounters significant performance regression when migrating from Spark 3.2 to Spark 3.4 on a `Insert Into` query which is analyzed as a `AppendData` on an Iceberg table.

We found that the root cause is in Spark 3.4, `TableOutputResolver` resolves the query with additional `ArrayTransform` on an `ArrayType` field. The `ArrayTransform`'s lambda function is actually an identical function, i.e., the transformation is redundant.
Bug: Create shared interface for SparkSession.Builder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/24 14:03
Affects Version/s: 4.0.0

Create an interface for the SparkConnect.Builder: org.apache.spark.sql.api.SparkSessionBuilder.

We need to be mindful of the different methods both implementations have. One apprach would be to support the union of all methods, and log warnings (or throw errors) if things are not supported.
Bug: Remove special casing for AVRO functions in Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/24 20:07
Affects Version/s: 4.0.0

We currently special case AVRO functions (to_avro/from_avro) in the FunctionRegistry and in the SparkConnectPlanner.

There are a couple of tasks here:
 * Unify Catalyst dummy expressions and AVRO expressions. We should try to retain the validation and the annotations.
 * Add all the required constructors
 * Register the functions using extensions
Bug: Catch errors when failing to write to external data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/24 15:21
Affects Version/s: 4.0.0

Catch non-fatal exceptions when trying to save the results of a query into a data source. For example, 
{code:java}
com.crealytics.spark.excel{code}
 sometimes throws:
{code:java}
scala.MatchError{code}
which is not very user-friendly.
Bug: [M1] Further exception improvements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/24 12:00
Affects Version/s: 4.0.0

We need to remove line numbers manually added to exceptions. Currently some exceptions print the line number twice.

Label exceptions need to use backquotes, same as with variables, i.e. need to check if toSQLId, toSQLStmt, and similar methods are applied to all identifiers.

Maybe add some tests for \{LINE} numbers?
Bug: Add memory based thresholds for shuffle spill
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/24 04:02
Affects Version/s: 4.0.0

We can only determine the number of spills by configuring {{{}spark.shuffle.spill.numElementsForceSpillThreshold{}}}. In some scenarios, the size of a row may be very large in the memory.
Bug: Remove special casing for Protobuf functions in Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/24 20:09
Affects Version/s: 4.0.0

Tasks to perform:
 * Remove the special casing from the connect planner.
 * Register the protobuf functions under system.internal using session extensions.
 * Add the needed constructors to the protobuf expressions.
 * Update protobuf functions and make them use unresolved function path
Bug: Support Transpose DataFrame API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/24 01:49
Affects Version/s: 4.0.0

Support Transpose as Scala/Python DataFrame API in both Spark Connect and Classic Spark.

Transposing data is a crucial operation in data analysis, enabling the transformation of rows into columns. This operation is widely used in tools like pandas and numpy, allowing for more flexible data manipulation and visualization.

While Apache Spark supports unpivot and pivot operations, it currently lacks a built-in transpose function. Implementing a transpose operation in Spark would enhance its data processing capabilities, aligning it with the functionalities available in pandas and numpy, and further empowering users in their data analysis workflows.

Please see [https://docs.google.com/document/d/1QSmG81qQ-muab0UOeqgDAELqF7fJTH8GnxCJF4Ir-kA/edit] for a detailed design.
Bug: to_date ... too slow
Status: Open | Priority: Major | Resolution: 
Created: 18/Aug/24 15:50
Affects Version/s: 2.4.7

The *to_date* _ _built-in udf is creating new _ParseException_ instances every time a string value cam't be parsed. _ParseException_ extends {_}Exception{_}, which in turn, extends {_}Throwable{_}, that calls the *fillInStackTrace* method. This method is not only one of the most expensive methods in Java, but it's also synchronized. That means, that could introduce some overhead.

Here's a stracktrace as example:

_java.lang.Throwable.fillInStackTrace(Native Method)_
*_java.lang.Throwable.fillInStackTrace(Throwable.java:783) => holding Monitor(java.text.ParseException@176695786})_*
_java.lang.Throwable.<init>(Throwable.java:265)_
_java.lang.Exception.<init>(Exception.java:66)_
_java.text.ParseException.<init>(ParseException.java:63)_
_java.text.DateFormat.parse(DateFormat.java:366)_
_org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.agg_doAggregateWithKeys_0$(Unknown Source)_
_org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.processNext(Unknown Source)_
_org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)_
_org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:645)_
_scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)_
_org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:187)_
_org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)_
_org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)_
_org.apache.spark.scheduler.Task.run(Task.scala:123)_
_org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:413)_
_org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1334)_
_org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:419)_
_java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)_
_java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)_
_java.lang.Thread.run(Thread.java:748)_

 

Because empty strings also throw _ParseException_ errors, trying to parse several date fields with empty values from a large dataframe could take the Spark task much more time than expected (and needed).

 

The following is strongly suggested:

 
 # Add some kind of warning in the to_date documentation page, clearly stating that lots of non-valid string date values will introduce serious performance issues. Note: the same warning should be also included in the documentation for udfs.
 # Add some kind of check or control to the string values before parsing in order to prevent the parsing of the string and the unnecessary creation of lots of exceptions. At least, for empty values, if checking valid string dates is considered a costly operation.
Bug: Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/24 09:03
Affects Version/s: 4.0.0

Cast from structs to variant objects should not be legal since variant objects are unordered bags of key-value pairs while structs are ordered sets of elements of fixed types. Therefore, casts between structs and variant objects do not behave like casts between structs. Example (produced by Serge Rielau):


{code:java}
scala> spark.sql("SELECT cast(named_struct('c', 1, 'b', '2') as struct<b int, c int>)").show()
+------------------------+
|named_struct(c, 1, b, 2)|

+------------------------+
|{1, 2}|

+------------------------+

Passing a struct into VARIANT loses the position
scala> spark.sql("SELECT cast(named_struct('c', 1, 'b', '2')::variant as struct<b int, c int>)").show()
+-----------------------------------------+
|CAST(named_struct(c, 1, b, 2) AS VARIANT)|

+-----------------------------------------+
|{2, 1}|

+-----------------------------------------+
{code}

Casts from maps to variant objects should also not be legal since they represent completely orthogonal data types. Maps can represent a variable number of key value pairs based on just a key and value type in the schema but in objects, the schema (produced by schema_of_variant expressions) will have a type corresponding to each value in the object. Objects can have values of different types while maps cannot and objects can only have string keys while maps can also have complex keys.

We should therefore prohibit the existing behavior of allowing explicit casts from structs and maps to variants as the variant spec currently only supports an object type which is remotely compatible with structs and maps. We should introduce a new expression that converts schemas containing structs and maps to variants. We will call it `to_variant_object`.

Also, schema_of_variant and schema_of_variant_agg expressions currently print STRUCT when Variant Objects are observed. We should also correct that to OBJECT.