av9ash/GitBugs huggingface dataset:
Bug: Install matplotlib in Python 3.9 and PyPy 3 in GitHub Actions image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/21 03:05
Affects Version/s: 3.3.0

Same as SPARK-36361,

SPARK-37756 requires to install `matplotlib` in both Python 3.9 and PyPy.

It would be great if we have them installed in the image we use to save installation time.
Bug: Close SessonState when each hive request done
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 28/Dec/21 03:15
Affects Version/s: 3.2.0

In current code in HiveClientImpl, we only call 

{code}

    shim.setCurrentSessionState(state)
    val ret = try {
      f
    } catch {
      case e: NoClassDefFoundError if e.getMessage.contains("apache/hadoop/hive/serde2/SerDe") =>
        throw QueryExecutionErrors.serDeInterfaceNotFoundError(e)
    } finally {
      state.getConf.setClassLoader(originalConfLoader)
      Thread.currentThread().setContextClassLoader(original)
    }
    ret
{code}

This will call SessionState.start() and won't  call SessionState.stop()
Bug: Allow LateralJoin node to host non-deterministic expressions when the outer query is a single row relation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/21 18:28
Affects Version/s: 3.2.1

After https://issues.apache.org/jira/browse/SPARK-37199, Analyzer will block LateralJoin that has non-deterministic lateral subqueries. But when the outer query is a single row relation, this should be allowed

For example:
Query:
{code:java}
SELECT * FROM VALUES(0) t(x) JOIN LATERAL (SELECT rand(0) + x AS y); {code}
Result:
{code:java}
org.apache.spark.sql.AnalysisException: nondeterministic expressions are only allowed in
Project, Filter, Aggregate or Window{code}
Bug: ANSI mode: allow casting between numeric type and timestamp type 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/21 13:02
Affects Version/s: 3.3.0

h3. What changes were proposed?
 * By default, allow casting between numeric type and timestamp type under ANSI mode
 * Remove the user-facing configuration {{spark.sql.ansi.allowCastBetweenDatetimeAndNumeric}}

h3. Why are the changes needed?

Same reason as mentioned in [#34459|https://github.com/apache/spark/pull/34459]. It is for better adoption of ANSI SQL mode since users are relying on it:
 * As we did some data science, we found that many Spark SQL users are actually using {{Cast(Timestamp as Numeric)}} and {{{}Cast(Numeric as Timestamp){}}}.
 * The Spark SQL connector for Tableau is using this feature for DateTime math. e.g.
{{CAST(FROM_UNIXTIME(CAST(CAST(%1 AS BIGINT) + (%2 * 86400) AS BIGINT)) AS TIMESTAMP)}}
Bug: Create a plan for top & frequency for pandas-on-Spark optimization
Status: Open | Priority: Major | Resolution: 
Created: 22/Dec/21 07:26
Affects Version/s: 3.3.0

When invoking the DataFrame.describe() in pandas API on Spark, the multiple Spark job is run as much as number of columns to retrieve the `top` and `freq` status.

Top is the most common value in column, and the freq is the count of the most common value.

We should write a util in Scala side, and make it return key value (count) in one Spark job. e.g.) Dataset.mapPartitions and calculate the summation of the key and value (count). Such APIs are missing in PySpark so we would have to write one in Scala side.

See the [https://github.com/apache/spark/pull/34931#discussion_r772220260] for more detail.
Bug: Upgrade SBT to 1.5.8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 08:45
Affects Version/s: 3.2.1

SBT 1.5.8 was released a few hours ago, which includes a fix for CVE-2021-45105{color:#172b4d}.{color}

https://github.com/sbt/sbt/releases/tag/v1.5.8
Bug: Update mypy in tests to 0.920
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 12:25
Affects Version/s: 3.3.0

As far as I can tell, there has been no official release yet, but 0.920 is already on PyPI ‒ https://pypi.org/project/mypy/0.920/.

We should consider updating out CI pipeline.

I've done some testing, and we need only a small fix in {{pyspark.pandas}}.
Bug: Incorrect annotations in SeriesGroupBy._cleanup_and_return 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/21 18:33
Affects Version/s: 3.3.0

[{{SeriesGroupBy._cleanup_and_return}}|https://github.com/apache/spark/blob/02ee1ae10b938eaa1621c3e878d07c39b9887c2e/python/pyspark/pandas/groupby.py#L2997-L2998] annotations

{code:python}
    def _cleanup_and_return(self, pdf: pd.DataFrame) -> Series:
        return first_series(pdf).rename().rename(self._psser.name)
{code}

are inconsistent:

- If {{pdf}} is {{pd.DataFrame}} then output should be {{pd.Series}}.
- If output is {{ps.Series}} then {{pdf}} should be {{ps.DataFrame}}.

Doesn't seem like the method is used (it is possible that my search skills and PyCharm inspection failed), so I am not sure which of these options was intended.
Bug: Fix ChildProcAppHandleSuite failed in spark-master-test-maven-hadoop-3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/21 09:00
Affects Version/s: 3.3.0

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-3.2/3522/#showFailuresLink
Bug: Expand should be supported PropagateEmptyRelation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/21 06:30
Affects Version/s: 3.2.0

Empty LocalScan still trigger HashAggregate execute. Expand should be supported in PropagateEmptyRelation too.

!image-2021-12-20-14-30-34-276.png|width=347,height=336!
Bug: Use more precise types for SparkContext Optional fields  (i.e. _gateway, _jvm)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/21 21:45
Affects Version/s: 3.3.0

As a part of SPARK-37152 [we agreed|https://github.com/apache/spark/pull/34466/files#r762609181] to keep these typed as not {{Optional}} for simplicity, but this just a temporary solution to move things forward, until we decide on best approach to handle such cases.
Bug: Reduce the output partition of output stage to avoid producing small files.
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Dec/21 08:13
Affects Version/s: 3.0.0

The partition size of the finalStage with `DataWritingCommand` or
 `V2TableWriteExec` may use the ADVISORY_PARTITION_SIZE_IN_BYTES which is smaller one and  produce some small files, it may bad for production, we should use a new partition size for the finalStage with `DataWritingCommand` or `V2TableWriteExec` to avoid small files.
Bug: Fix MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/21 17:45
Affects Version/s: 3.3.0

The failure happens on Java 17 native Apple Silicon version on Python 3.9/3.10.
{code}
$ java -version
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu17.30+15-CA (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu17.30+15-CA (build 17.0.1+12-LTS, mixed mode, sharing)
{code}

{code}
======================================================================
FAIL: test_raw_and_probability_prediction (pyspark.ml.tests.test_algorithms.MultilayerPerceptronClassifierTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dongjoon/APACHE/spark-merge/python/pyspark/ml/tests/test_algorithms.py", line 104, in test_raw_and_probability_prediction
    self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, rtol=0.102))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 7.385s

FAILED (failures=1)

Had test failures in pyspark.ml.tests.test_algorithms MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction with python3; see logs.
{code}
Bug: exception when handling late data with watermarking and window
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/21 09:01
Affects Version/s: 3.2.0

when i use watermark to block late data, meanwhile window for state de-duplication, the order will cause unexpected behavior.

a）below code will cause exception state that {color:#172b4d}"Couldn't find {color:#de350b}timestamp#58-T5000ms{color} in [{color:#4c9aff}window#550-T5000ms{color},raid#132L,app#528]"{color}
{code:java}
// code placeholder
withWatermark("timestamp", "5 seconds").
withColumn("window", window($"timestamp", "1 hours")).
dropDuplicates("window", "raid", "app"). {code}
b) but when i switch the order of watermark and window config as below, it work without any exception 
{code:java}
// code placeholder
withColumn("window", window($"timestamp", "1 hours")). 
withWatermark("timestamp", "5 seconds").
dropDuplicates("window", "raid", "app").  {code}
pls. note ,  this issue does not exist on spark v3.1.2
Bug: Flexible ways of launching executors
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/21 02:10
Affects Version/s: 3.3.0

Currently Spark launches executor processes by constructing and running commands [1], for example:
{code:java}
/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java -cp /opt/spark-3.2.0-bin-hadoop3.2/conf/:/opt/spark-3.2.0-bin-hadoop3.2/jars/* -Xmx1024M -Dspark.driver.port=35729 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@example.host:35729 --executor-id 0 --hostname 100.116.124.193 --cores 6 --app-id app-20211207131146-0002 --worker-url spark://Worker@100.116.124.193:45287 {code}
But there are use cases which require more flexible ways of launching executors. In particular, our use case is that we run Spark in standalone mode, Spark master and workers are running in VMs. We want to allow Spark app developers to provide custom container images to customize the job runtime environment (typically Java and Python dependencies), so executors (which run the job code) need to run in Docker containers.

After reading the source code, we found that the concept of Spark Command Runner might be a good solution. Basically, we want to introduce an optional Spark command runner in Spark, so that instead of running the command to launch executor directly, it passes the command to the runner, the runner then runs the command with its own strategy which could be running in Docker, or by default running the command directly.

The runner is specified through an env variable `SPARK_COMMAND_RUNNER`, which by default could be a simple script like:
{code:java}
#!/bin/bash
exec "$@" {code}
or in the case of Docker container:
{code:java}
#!/bin/bash
docker run ... – "$@" {code}
 

I already have a patch for this feature and have tested in our environment.

 

[1]: [https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala#L52]
Bug: spark history server clean event log directory with out check status file
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/21 06:54
Affects Version/s: 3.1.1

i foud a problem, the thrift server create event log file(.inprogress file create at init), and history server clean the application event log file according size and modtime. so there is a potential problem under this situation

*if the thrift server accept no quest long time(longer than time config by spark.history.fs.cleaner.maxAge), the history server will clean  the applicaiton log [directory] with the inprogress file; after clean  the thrift server accept a lot of request ,and will generate new event log directory without inprogress status file, and the director will never be clean by history server because it not contain status file. this will leads spack leak*

i think whenever create new log file , need to check wether the status file is exist, if not create it

last i think extra function need add, like log4j the compact file stii need to be clean after a period(config by user)，so ，long run spark service like thrift server‘s event log file space can be limit in a config size
Bug: Tell spark-env.sh the python interpreter
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Dec/21 00:48
Affects Version/s: 3.2.0

When loading config defaults via spark-env.sh, it can be useful to know
the current pyspark python interpreter to allow the configuration to set
values properly. Pass this value in the environment as
_PYSPARK_DRIVER_SYS_EXECUTABLE to the environment script.
h3. What changes were proposed in this pull request?

It's currently possible to set sensible site-wide spark configuration defaults by using {{{}$SPARK_CONF_DIR/spark-env.sh{}}}. In the case where a user is using pyspark, however, there are a number of things that aren't discoverable by that script, due to the way that it's called. There is a chain of calls (java_gateway.py -> shell script -> java -> shell script) that ends up obliterating any bit of the python context.

This change proposes to add en environment variable {{_PYSPARK_DRIVER_SYS_EXECUTABLE}} which points to the filename of the top-level python executable within pyspark's {{java_gateway.py}} bootstrapping process. With that, spark-env.sh will be able to infer enough information about the python environment to set the appropriate configuration variables.
h3. Why are the changes needed?

Right now, there a number of config options useful to pyspark that can't be reliably set by {{spark-env.sh}} because it is unaware of the python context that spawning the executor. To give the most trivial example, it is currently possible to set {{spark.kubernetes.container.image}} or {{spark.driver.host}} based on information readily available from the environment (e.g. the k8s downward API). However, {{spark.pyspark.python}} and family cannot be set because when {{spark-env.sh}} executes it's lost all of the python context. We can instruct users to add the appropriate config variables, but this form of cargo-culting is error-prone and not scalable. It would be much better to expose important python variables so that pyspark can not be a second-class citizen.
h3. Does this PR introduce _any_ user-facing change?

Yes. With this change, if python spawns the JVM, {{spark-env.sh}} will receive an environment variable {{_PYSPARK_DRIVER_SYS_EXECUTABLE}} pointing to the python executor.
h3. How was this patch tested?

To be perfectly honest, I don't know where this fits into the testing infrastructure. I monkey-patched a binary 3.2.0 install to add the lines to java_gateway.py and that works, but in terms of adding this to the CI ... I'm at a loss. I'm more than willing to add the additional info, if needed.
Bug: reuse Exchange failed 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Dec/21 09:44
Affects Version/s: 3.1.1

PR available：https://github.com/apache/spark/pull/34858

use case：query2 in TPC-DS.There are three exchange subquery will scan the same table "store_sales" in logical plan,these subqueries meet exchange reuse rule.I confirm that the exchange use rule  work in physical plan.But when spark execute the physical plan,I find out 

exchange reuse failed,reused exchange has been executed twice.

physical plan：

!physical plan-query2.png!

 

execution stages：

!execution stage-query2.png!

 

!execution stage(1)-query2.png!
Bug: View Analysis incorrectly marks nested fields as nullable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/21 00:16
Affects Version/s: 3.2.0

Consider a view as follows with all fields non-nullable (required)
{code:java}
spark.sql("""
    CREATE OR REPLACE VIEW v AS 
    SELECT id, named_struct('a', id) AS nested
    FROM RANGE(10)
""")
{code}
we can see that the view schema has been correctly stored as non-nullable
{code:java}
scala> System.out.println(spark.sessionState.catalog.externalCatalog.getTable("default", "v2"))
CatalogTable(
Database: default
Table: v2
Owner: smahadik
Created Time: Tue Dec 07 09:00:42 PST 2021
Last Access: UNKNOWN
Created By: Spark 3.3.0-SNAPSHOT
Type: VIEW
View Text: SELECT id, named_struct('a', id) AS nested
    FROM RANGE(10)
View Original Text: SELECT id, named_struct('a', id) AS nested
    FROM RANGE(10)
View Catalog and Namespace: spark_catalog.default
View Query Output Columns: [id, nested]
Table Properties: [transient_lastDdlTime=1638896442]
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Storage Properties: [serialization.format=1]
Schema: root
 |-- id: long (nullable = false)
 |-- nested: struct (nullable = false)
 |    |-- a: long (nullable = false)
)
{code}
However, when trying to read this view, it incorrectly marks nested column {{a}} as nullable
{code:java}
scala> spark.table("v2").printSchema
root
 |-- id: long (nullable = false)
 |-- nested: struct (nullable = false)
 |    |-- a: long (nullable = true)
{code}
This is caused by [this line|https://github.com/apache/spark/blob/fb40c0e19f84f2de9a3d69d809e9e4031f76ef90/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3546] in Analyzer.scala. Going through the history of changes for this block of code, it seems like {{asNullable}} is a remnant of a time before we added [checks|https://github.com/apache/spark/blob/fb40c0e19f84f2de9a3d69d809e9e4031f76ef90/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3543] to ensure that the from and to types of the cast were compatible. As nullability is already checked, it should be safe to add a cast without converting the target datatype to nullable.
Bug: Upgrade SBT to 1.5.6
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/21 04:00
Affects Version/s: 3.2.1

This issue aims to upgrade SBT to 1.5.6.

- https://github.com/sbt/sbt/releases/tag/v1.5.6

SBT 1.5.6 updates log4j 2 to 2.15.0, which fixes remote code execution vulnerability (CVE-2021-44228)

This only affects build servers.
Bug: Support window frame ORDER BY i ROWS BETWEEN (('foo' < 'foobar')::integer) PRECEDING AND CURRENT ROW
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/21 03:31
Affects Version/s: 3.3.0

SELECT array_agg(i) OVER w
FROM range(1,6) i
WINDOW w AS (ORDER BY i ROWS BETWEEN (('foo' < 'foobar')::integer) PRECEDING AND CURRENT ROW);
Bug: Add the support for struct type column in the DropDuplicate in spark
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Dec/21 11:05
Affects Version/s: 3.2.0

Add the support for struct type coulmn in the DropDuplicate in spark 


Currently on using the struct col in the DropDuplicate we will get the below exception

case class StructDropDup(c1: Int, c2: Int)

val df = Seq(("d1", StructDropDup(1, 2)),
      ("d1", StructDropDup(1, 2))).toDF("a", "b")

df.dropDuplicates("b.c1")
{code:java}
org.apache.spark.sql.AnalysisException: Cannot resolve column name "b.c1" among (a, b)
  at org.apache.spark.sql.Dataset.$anonfun$dropDuplicates$1(Dataset.scala:2576)
  at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
  at scala.collection.Iterator.foreach(Iterator.scala:941)
  at scala.collection.Iterator.foreach$(Iterator.scala:941)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1429){code}
As workAround inorder to find the the duplicate using the struct column

df1.withColumn("b.c1", col("b.c1")).dropDuplicates("b.c1").drop("b.c1").collect
{code:java}
df.withColumn("b.c1", col("b.c1")).dropDuplicates("b.c1").drop("b.c1").collect
res25: Array[org.apache.spark.sql.Row] = Array([d1,[1,2]]){code}
There is need to add the support for the dropDuplicates
Bug: DatasourceV2 `exists ... select *` column push down
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/21 09:42
Affects Version/s: 3.1.2

The datasourcev2 table is very slow when executing TPCDS, because `exists ... select *` will not push down the cropped columns to the data source

 

Add test in `org.apache.spark.sql.connector.DataSourceV2SQLSuite`
{code:java}
test("datasourcev2 exists") {
    val t1 = s"${catalogAndNamespace}t1"
    withTable(t1) {
      sql(s"CREATE TABLE $t1 (col1 string, col2 string) USING $v2Format")
      val t2 = s"${catalogAndNamespace}t2"
      withTable(t2) {
        sql(s"CREATE TABLE $t2 (col1 string, col2 string) USING $v2Format")
        val query = sql(s"select * from $t1 where not exists" +
            s"(select * from $t2 where t1.col1=t2.col1)").queryExecution
        // scalastyle:off println
        println(query.executedPlan)
      }
    }
  }


AdaptiveSparkPlan isFinalPlan=false
+- BroadcastHashJoin [col1#17], [col1#19], LeftSemi, BuildRight, false
   :- Project [col1#17, col2#18]
   :  +- BatchScan[col1#17, col2#18] class org.apache.spark.sql.connector.catalog.InMemoryTable$InMemoryBatchScan RuntimeFilters: []
   +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]),false), [id=#28]
      +- Project [col1#19]
         +- BatchScan[col1#19, col2#20] class org.apache.spark.sql.connector.catalog.InMemoryTable$InMemoryBatchScan RuntimeFilters: []


Expectation is `BatchScan[col1#19] class org.apache.spark.sql.connector.catalog.InMemoryTable$InMemoryBatchScan RuntimeFilters: []` {code}
Reason `Batch("Early Filter and Projection Push-Down" V2ScanRelationPushDown` is executed before `Batch("RewriteSubquery"`, parallel datasourceV2 does not support `FileSourceStrategy`
Bug: Deser void class fail with Java serialization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/21 06:40
Affects Version/s: 3.2.0

Spark code contains Java void type cannot be serialized with JavaSerializer. For example:
{code:java}
class Foo extends Serializable {
  Class k = Void.TYPE;
}
{code}
Spark will throw error:
{code:java}
java.lang.Void; local class name incompatible with stream class name "void"
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:703)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
	at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1771)
        ......
{code}
All other primitive type works. The reason is when SPARK-8730 try to fix the primitive type deserialization, it introduces the [following code|https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80]:
{code:java}
private object JavaDeserializationStream {
  val primitiveMappings = Map[String, Class[_]](
    "boolean" -> classOf[Boolean],
    "byte" -> classOf[Byte],
    "char" -> classOf[Char],
    "short" -> classOf[Short],
    "int" -> classOf[Int],
    "long" -> classOf[Long],
    "float" -> classOf[Float],
    "double" -> classOf[Double],
    "void" -> classOf[Void]
  )
}
{code}
However, classOf[Void] is not the equivalence of other types. It's point to Void.class not Void.Type. The equivalence for void should be classOf[Unit]:
{code:java}
scala> classOf[Long]
val res0: Class[Long] = long

scala> classOf[Double]
val res1: Class[Double] = double

scala> classOf[Byte]
val res2: Class[Byte] = byte

scala> classOf[Void]
val res3: Class[Void] = class java.lang.Void  <--- this is wrong

scala> classOf[Unit]
val res4: Class[Unit] = void <---- this is right
{code}
Bug: minRatePerPartition works as "max messages per partition per a batch" (it should be per seconds)
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Dec/21 00:35
Affects Version/s: 2.4.8

{{maxRatePerPartition}} means "max messages per partition per second".
But minRatePerPartition does not. ("max messages per partition per a batch"). This is a bug.

 

minRatePerPartition should be works as "min messages per partition per second".
 * [https://github.com/apache/spark/blob/f97de309792f382ae823894e978f7e54f34f1a29/docs/configuration.md?plain=1#L2878-L2886]