av9ash/GitBugs huggingface dataset:
Bug: Spark SQL should do further validation on join condition fields
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 20/Feb/23 15:52
Affects Version/s: 3.3.2

In Spark SQL, the conditions for the join use fields that are allowed to be fields from a non-left table or a non-right table. In this case, the join will degenerate into a cross join. 

Suppose you have two tables, test1 and test2, which have the same table schema:
{code:java}
```
CREATE TABLE `default`.`test1` (
  `id` INT,
  `name` STRING,
  `age` INT,
  `dt` STRING)
USING parquet
PARTITIONED BY (dt)
```{code}
The following SQL has three joins, but in the last left join, the conditions is `t1.name=t2.name`, and t3.name is not used. So the last left join will be cross join. 
{code:java}
```
select *
from 
(select * from test1 where dt="20230215"  and age=1 ) t1 
left join
(select * from test1 where dt=="20230215"  and age=2) t2
on t1.name=t2.name
left join
(select * from test2 where dt="20230215") t3
on
t1.name=t2.name;
```{code}
So i think Spark SQL should do further validation on join condition, the fields of join condition must be a left table or right table field , otherwise it is thrown `AnalysisException`.
Bug: Support query inserted partitions after insert data into table when hive.exec.dynamic.partition=true
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Feb/23 07:27
Affects Version/s: 3.5.0

When hive.exec.dynamic.partition=true and hive.exec.dynamic.partition.mode=nonstrict, we can insert table by sql like 'insert overwrite table aaa partition(dt) select xxxx',  of course we can know the partitions inserted into the table by the sql itself,  but if we want do something for common use, we need some common way to get the inserted partitions,  for example:

    spark.sql("insert overwrite table aaa partition(dt) select xxxx")  //insert table

    val partitions = getInsertedPartitions()   //need some way to get inserted partitions

    monitorInsertedPartitions(partitions)    //do something for common use

Since insert statement should not return any data, this ticket propose to introduce spark.hive.exec.dynamic.partition.savePartitions=true (default false) spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix=hive_dynamic_inserted_partitions

when spark.hive.exec.dynamic.partition.savePartitions=true we save the partitions to the 

temporary view $spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix_$dbName_$tableName

we will allow user to do this

scala> spark.conf.set("hive.exec.dynamic.partition", true)

scala> spark.conf.set("hive.exec.dynamic.partition.mode", "nonstrict")

scala> spark.conf.set("spark.hive.exec.dynamic.partition.savePartitions", true)

scala> spark.sql("insert overwrite table db1.test_partition_table partition (dt) select 1, '2023-02-22'").show(false)

++                                                                              

||

++

++

scala> spark.sql("select * from hive_dynamic_inserted_partitions_db1_test_partition_table").show(false)

+----------+                                                                    

|dt        |

+----------+

|2023-02-22|

+----------+
Bug: SPARK-42045 is incomplete in supporting ANSI_MODE fro round() and bround()
Status: Open | Priority: Major | Resolution: 
Created: 24/Feb/23 03:01
Affects Version/s: 3.3.2

under ANSI mode SPARK-42045 added error conditions insetad of silent overflows for edge cases in round() and bround().
However it appears this fix works only for the INT data type. Trying it on a e.g. SMALLINT the function still returns wrong results:
{code:java}
spark-sql> select round(2147483647, -1);
[ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.{code}
{code:java}
spark-sql> select round(127y, -1);
-126 {code}
Bug: Add Broadcast to functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 18:23
Affects Version/s: 3.4.0

Add the {{broadcast}} function to functions.scala. Please check if we can get the same semantics as the current implementation using unresolved hints.

https://github.com/apache/spark/blame/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L1246-L1261
Bug: Push down topK through join
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Feb/23 14:27
Affects Version/s: 3.4.0


{code:scala}
    spark.range(100000000).selectExpr("id % 10000 as a", "id as b").write.saveAsTable("t1")
    spark.range(100000000).selectExpr("id % 10000 as x", "id as y").write.saveAsTable("t2")

    sql("select * from t1 left join t2 on a = x order by b limit 5").collect()
    spark.sql("set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.LimitPushDown")
    sql("select * from t1 left join t2 on a = x order by b limit 5").collect()
{code}
Bug: parameter markers not blocked in DDL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 17:34
Affects Version/s: 3.4.0

The parameterized query code does not block DDL statements from referencing parameter markers.
E.g. a 

 
{code:java}
scala> spark.sql(sqlText = "CREATE VIEW v1 AS SELECT current_timestamp() + :later as stamp, :x * :x AS square", args = Map("later" -> "INTERVAL'3' HOUR", "x" -> "15.0")).show()
++
||
++
++
{code}
It appears we have some protection that fails us when the view is invoked:

 
{code:java}
scala> spark.sql(sqlText = "SELECT * FROM v1", args = Map("later" -> "INTERVAL'3' HOUR", "x" -> "15.0")).show()
org.apache.spark.sql.AnalysisException: [UNBOUND_SQL_PARAMETER] Found the unbound parameter: `later`. Please, fix `args` and provide a mapping of the parameter to a SQL literal.; line 1 pos 29
{code}

Right now I think affected are:
* DEFAULT definition
* VIEW definition

but any other future standard expression popping up is at risk, such as SQL Functions, or GENERATED COLUMN.

CREATE TABLE AS is debatable, since it it executes the query at definition only.
For simplicity I propose to block the feature from ANY DDL statement (CREATE, ALTER).
Bug: A large stage could run indefinitely due to executor lost
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/23 08:43
Affects Version/s: 3.0.3

When a stage is extremely large and Spark runs on spot instances or problematic clusters with frequent worker/executor loss,  the stage could run indefinitely due to task rerun caused by the executor loss. This happens, when the external shuffle service is on, and the large stages runs hours to complete, when spark tries to submit a child stage, it will find the parent stage - the large one, has missed some partitions, so the large stage has to rerun. When it completes again, it finds new missing partitions due to the same reason.

We should add a attempt limitation for this kind of scenario.
Bug: Upgrade ZooKeeper from 3.6.3 to 3.6.4
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Feb/23 15:51
Affects Version/s: 3.5.0

[ZooKeeper 3.6 is EoL since 30th December, 2022|https://zookeeper.apache.org/releases.html]


[Release notes|https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html]
Bug: spark-hadoop-cloud is not provided in the default Spark distribution
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 13/Feb/23 21:46
Affects Version/s: 3.3.1

The library spark-hadoop-cloud is absent in the default Spark distribution (as well as its dependencies like hadoop-aws). Therefore the dependency management section described in [Integration with Cloud Infrastructures|https://spark.apache.org/docs/3.3.1/cloud-integration.html#installation] is invalid. Actually the libraries for cloud integration are not provided.

A naive workaround would be to add the spark-hadoop-cloud library as a compile-scope dependency. However, this does not work due to Spark classpath hierarchy. Spark system classloader does not see classes loaded by the application classloader.

Therefore a proper fix would be to enable the hadoop-cloud build profile by default: -Phadoop-cloud
Bug: Log4j2 doesn't works with Spark 3.3.0
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 17/Feb/23 12:31
Affects Version/s: 3.2.1

Hi All,
Was trying to run spark application on the cluster mode using log4j2 and Spark 3.3.0.

When I run the below spark-submit command, only one worker (out of 3) starts executing the job.
{code:java}
// code placeholder
spark-submit --master spark://spark-master-svc:7077 \
    --conf spark.cores.max=4 \
    --conf spark.sql.broadcastTimeout=3600 \
    --conf spark.executor.cores=1 \
    --jars /opt/spark/work-dir/<main_jar_name>.jar \
    --deploy-mode cluster \
    --class <class_name> \
    --properties-file /opt/spark/conf/spark-defaults.conf \
    --conf spark.driver.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
    --conf spark.executor.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
    --files "/opt/spark/work-dir/<log4j2>.properties" \
    /opt/spark/work-dir/<jar_name>.jar /opt/spark/work-dir/application.properties >> /var/log/containers/hourly.log 2>&1 {code}
It means, on only one worker, I can see the driver logs and the other workers are idle and there are no app or executors logs created on other workers.

Below is the log4j2.properties file being used.
{code:java}
// code placeholder
rootLogger.level = INFO
rootLogger.appenderRef.rolling.ref = loggerId

appender.rolling.type = RollingFile
appender.rolling.name = loggerId
appender.rolling.fileName=/var/log/containers/hourly.log
appender.rolling.filePattern=hourly-.%d{yyyyMMdd}.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern=%d [%t] %-5p (%F:%L) - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = TimeBasedTriggeringPolicy
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 5


logger.spark.name = org.apache.spark
logger.spark.level = WARN
logger.spark.additivity = false

logger.spark.repl.SparkIMain$exprTyper.level = INFO
logger.spark.repl.SparkILoop$SparkILoopInterpreter.level = INFO

# Settings to quiet third party logs that are too verbose
logger.jetty.name = org.eclipse.jetty
logger.jetty.level = WARN
logger.jetty.util.component.AbstractLifeCycle.level = ERROR
logger.parquet.name = org.apache.parquet
logger.parquet.level = ERROR

logger.kafka.name = org.apache.kafka
logger.kafka.level = WARN
logger.kafka.clients.consumer.internals.Fetcher.level=WARN {code}
All log4j2 jars are included in the Spark home classpath under the jars directory.
 * log4j-1.2-api-2.17.2.jar
 * log4j-api-2.17.2.jar
 * log4j-api-scala_2.12-12.0.jar
 * log4j-core-2.17.2.jar
 * log4j-slf4j-impl-2.17.2.jar

Can you please check and let me know whether I need to add or update anything to start the job in cluster mode supporting log4j2

Note : Things work fine with log4j1
Bug: [MESOS] OMP_NUM_THREADS not set to number of executor cores by default
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 27/Feb/23 19:26
Affects Version/s: 3.3.2

We could have similar issue to SPARK-42596 (YARN) in Mesos.

Could someone verify? Unfortunately I am not able to due to lack of infra.
Bug: Document SS guide doc for introducing watermark propagation among operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 04:43
Affects Version/s: 3.5.0

Once SPARK-42376 has merged, we would want to also provide the example of using stream-stream time interval join followed by streaming aggregation. Just adding the feature without proper document may lead to the case no one even knows this is supported.
Bug: Cannot resolve orderby attributes in DISTINCT 
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Feb/23 08:45
Affects Version/s: 3.3.1

query:
{code:java}
CREATE TABLE students (name VARCHAR(64), address VARCHAR(64)) USING PARQUET PARTITIONED BY (student_id INT);
SELECT DISTINCT trim(name) AS new_name FROM students order by name desc;
{code}
error:
{code:java}
spark-sql> SELECT DISTINCT trim(name) AS new_name FROM students order by name desc;
Error in query: Column 'name' does not exist. Did you mean one of the following? [new_name]; line 1 pos 62;
'Sort ['name DESC NULLS LAST], true
+- Distinct
   +- Project trim(name#1, None) AS new_name#0
      +- SubqueryAlias spark_catalog.default.students
         +- Relation default.studentsname#1,address#2,student_id#3 parquet {code}
Bug: Introduce watermark propagation among operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 13:51
Affects Version/s: 3.5.0

With introduction of SPARK-40925, we enabled workloads containing multiple stateful operators in a single streaming query.

The JIRA ticket clearly described out-of-scope, "Here we propose fixing the late record filtering in stateful operators to allow chaining of stateful operators {*}which do not produce delayed records (like time-interval join or potentially flatMapGroupsWithState){*}".

We identified production use case for stream-stream time-interval join followed by stateful operator (e.g. window aggregation), and propose to address such use case via this ticket.

The design will be described in the PR, but the sketched idea is introducing simulation of watermark propagation among operators. As of now, Spark considers all stateful operators to have same input watermark and output watermark, which introduced the limitation. With this ticket, we construct the logic to simulate watermark propagation so that each operator can have its own (input watermark, output watermark). Operators introducing delayed records will produce delayed output watermark, and downstream operator can take the delay into account as input watermark will be adjusted.
Bug: Make a serializable jobTrackerId instead of a non-serializable JobID in FileWriterFactory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 11:53
Affects Version/s: 3.3.2

https://issues.apache.org/jira/browse/SPARK-41448 make consistent MR job IDs in FileBatchWriter and FileFormatWriter, but it breaks a serializable issue, JobId is non-serializable
Bug: Dataset.colregex should link a plan_id when it only matches a single column.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 18:20
Affects Version/s: 3.4.0

When colregex returns a single column it should link the plans plan_id. For reference here is the non-connect Dataset code that does this:

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L1512]

This also needs to be fixed for the Python client.
Bug: Implement SparkSession.newSession
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Feb/23 18:44
Affects Version/s: 3.4.0

Implement SparkSession.newSession for Connect.
{code:java}
/**
 * Start a new session with isolated SQL configurations, temporary tables, registered
 * functions are isolated, but sharing the underlying `SparkContext` and cached data.
 *
 * @note Other than the `SparkContext`, all shared state is initialized lazily.
 * This method will force the initialization of the shared state to ensure that parent
 * and child sessions are set up with the same shared state. If the underlying catalog
 * implementation is Hive, this will initialize the metastore, which may take some time.
 *
 * @since 2.0.0
 */
def newSession(): SparkSession {code}
Bug: An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 06:27
Affects Version/s: 3.3.1

*when 'union all' and one select statement use* *Literal as column value , the other* *select statement  has computed expression at the same column , then the whole statement will compile failed. A explicit cast will be needed.*

for example:

{color:#4c9aff}explain{color}

{color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}

{color:#4c9aff}*select* *null* *as* amt1, {*}cast{*}('256.99' *as* {*}decimal{*}(20,8)) *as* amt2{color}

{color:#4c9aff}*union* *all*{color}

{color:#4c9aff}*select* {*}cast{*}('200.99' *as* {*}decimal{*}(20,8)){*}/{*}100 *as* amt1,{*}cast{*}('256.99' *as* {*}decimal{*}(20,8)) *as* amt2;{color}

*will got error :* 

org.apache.spark.{*}sql{*}.catalyst.expressions.Literal cannot be *cast* *to* org.apache.spark.{*}sql{*}.catalyst.expressions.AnsiCast

The SQL will need to change to : 

{color:#4c9aff}explain{color}

{color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}

{color:#4c9aff}*select* *null* *as* amt1,{*}cast{*}('256.99' *as* {*}decimal{*}(20,8)) *as* amt2{color}

{color:#4c9aff}*union* *all*{color}

{color:#4c9aff}*select* {color:#de350b}{*}cast{*}({color}{*}cast{*}('200.99' *as* {*}decimal{*}(20,8)){*}/{*}100 *as* {*}decimal{*}(20,8){color:#de350b}){color} *as* amt1,{*}cast{*}('256.99' *as* {*}decimal{*}(20,8)) *as* amt2;{color}

 

*but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ?*
Bug: Make `CompatibilitySuite` as a tool like `dev/mima`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 09:28
Affects Version/s: 3.4.0

Using maven to test `CompatibilitySuite` requires some pre-work(need maven build sql & 
connect-client-jvm module before test), so when we run `mvn package test`, there will be following errors:
 
{code:java}
CompatibilitySuite:
- compatibility MiMa tests *** FAILED ***
  java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  ...
- compatibility API tests: Dataset *** FAILED ***
  java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$7(CompatibilitySuite.scala:110)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22) {code}
Bug: StackOverflowError reading json that does not conform to schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 15:11
Affects Version/s: 3.4.0

Databricks runtime 12.1 uses a pre-release version of spark 3.4.x we encountered the following problem

 

!https://user-images.githubusercontent.com/133639/221866500-99f187a0-8db3-42a7-85ca-b027fdec160d.png!
Bug: PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 02:03
Affects Version/s: 3.3.0

Follow up from [https://github.com/apache/spark/pull/40199#discussion_r1119453996]

If OMP_NUM_THREADS is not set explicitly, we should set it to `spark.task.cpus` instead of `spark.executor.cores` as described in [PR #38699|https://github.com/apache/spark/pull/38699].
Bug: createDataFrame should support DDL string as schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 00:50
Affects Version/s: 3.4.0

{code:python}
File "/.../python/pyspark/sql/connect/readwriter.py", line 393, in pyspark.sql.connect.readwriter.DataFrameWriter.option
Failed example:
    with tempfile.TemporaryDirectory() as d:
        # Write a DataFrame into a CSV file with 'nullValue' option set to 'Hyukjin Kwon'.
        df = spark.createDataFrame([(100, None)], "age INT, name STRING")
        df.write.option("nullValue", "Hyukjin Kwon").mode("overwrite").format("csv").save(d)

        # Read the CSV file as a DataFrame.
        spark.read.schema(df.schema).format('csv').load(d).show()
Exception raised:
    Traceback (most recent call last):
      File "/.../lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.readwriter.DataFrameWriter.option[2]>", line 3, in <module>
        df = spark.createDataFrame([(100, None)], "age INT, name STRING")
      File "/.../python/pyspark/sql/connect/session.py", line 312, in createDataFrame
        raise ValueError(
    ValueError: Some of types cannot be determined after inferring, a StructType Schema is required in this case
{code}
Bug: Support Scala 2.12/2.13 tests in connect module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/23 22:58
Affects Version/s: 3.4.0

{code}
$ build/sbt -Dscala.version=2.13.8 -Pscala-2.13 -Phadoop-3 assembly/package "connect/test"
{code}
Bug: Specify protocol for UDF artifact transfer in JVM/Scala client 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/23 21:28
Affects Version/s: 3.4.0

An "artifact" is any file that may be used during the execution of a UDF.

In the decoupled client-server architecture of Spark Connect, a remote client may use a local JAR or a new class in their UDF that may not be present on the server. To handle these cases of missing "artifacts", a protocol for artifact transfer is needed to move the required artifacts from the client side over to the server side.