av9ash/GitBugs huggingface dataset:
Bug: BroadcastHashJoinExe's outputPartitioning can utilize the build side
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/May/20 03:41
Affects Version/s: 3.1.0

Currently, the BroadcastHashJoinExec's outputPartitioning only uses the streamed side's outputPartitioning. Thus, if the join key is from the build side for the join where one side is BroadcastHashJoinExec:
{code:java}
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "500")
val t1 = (0 until 100).map(i => (i % 5, i % 13)).toDF("i1", "j1")
val t2 = (0 until 100).map(i => (i % 5, i % 13)).toDF("i2", "j2")
val t3 = (0 until 20).map(i => (i % 7, i % 11)).toDF("i3", "j3")
val t4 = (0 until 100).map(i => (i % 5, i % 13)).toDF("i4", "j4")

// join1 is a sort merge join.
val join1 = t1.join(t2, t1("i1") === t2("i2"))

// join2 is a broadcast join where t3 is broadcasted.
val join2 = join1.join(t3, join1("i1") === t3("i3"))

// Join on the column from the broadcasted side (i3).
val join3 = join2.join(t4, join2("i3") === t4("i4"))

join3.explain
{code}
it produces Exchange hashpartitioning(i2#103, 200):
{code:java}
== Physical Plan ==
*(6) SortMergeJoin [i3#29], [i4#40], Inner
:- *(4) Sort [i3#29 ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(i3#29, 200), true, [id=#55]
:     +- *(3) BroadcastHashJoin [i1#7], [i3#29], Inner, BuildRight
:        :- *(3) SortMergeJoin [i1#7], [i2#18], Inner
:        :  :- *(1) Sort [i1#7 ASC NULLS FIRST], false, 0
:        :  :  +- Exchange hashpartitioning(i1#7, 200), true, [id=#28]
:        :  :     +- LocalTableScan [i1#7, j1#8]
:        :  +- *(2) Sort [i2#18 ASC NULLS FIRST], false, 0
:        :     +- Exchange hashpartitioning(i2#18, 200), true, [id=#29]
:        :        +- LocalTableScan [i2#18, j2#19]
:        +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))), [id=#34]
:           +- LocalTableScan [i3#29, j3#30]
+- *(5) Sort [i4#40 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(i4#40, 200), true, [id=#39]
      +- LocalTableScan [i4#40, j4#41]
{code}
 But, since BroadcastHashJoinExec is only for equi-join, if the streamed side has HashPartitioning, BroadcastHashJoinExec can utilize the info to eliminate the exchange.
Bug: Improve the current Spark Scheduler test framework
Status: Open | Priority: Minor | Resolution: 
Created: 26/May/20 09:29
Affects Version/s: 3.1.0

The major source of Spark Scheduler unit test cases are DAGSchedulerSuite、TaskSchedulerImplSuite、TaskSetManagerSuite. These test suites have played an important role to ensure the Spark Scheduler behaves as we expected, however, we should significantly improve these suites to provide better organized and more extendable test framework now, to further support the evolution of the Spark Scheduler.

The major limitations of the current Spark Scheduler test framework:
* The test framework was designed at the very early stage of Spark, so it doesn’t integrate well with the features introduced later, e.g. barrier execution, indeterminate stage, zombie taskset, resource profile.
* Many test cases are added in a hacky way, don’t fully utilize or expend the original test framework (while they could have been), this leads to a heavy maintenance burden.
* The test cases are not organized well, many test cases are appended case by case, each test file consists of thousands of LOCs.
* Frequently introducing flaky test cases because there is no standard way to generate test data and verify the result.
Bug: Make Spark's guava version configurable from the maven command line.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/May/20 18:03
Affects Version/s: 3.1.0

All future releases of hadoop are going to ship with 27.0 or later, *including point releases of the 3.1 branch, which is a mixed blessing.

Pro: 

* it's up to date
* no active CVEs

Con:

 * code which uses things the guava team removed won't compile
 * code built with the later release won't link to older versions due to signature overloading, with Preconditions.checkArgument a key example

Making the guava.version spark pulls in an overrideable value makes it possible to choose the version to build with alongside the -Dhadoop.version selection of hadoop version.
Bug: AdaptiveQueryExecSuite: "Do not optimize skew join if introduce additional shuffle" test has no skew join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/May/20 10:35
Affects Version/s: 3.0.0

Due to incorrect configurations of 

- spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes

- spark.sql.adaptive.advisoryPartitionSizeInBytes
Bug: Dataset.repartition leverage adaptive execution
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 27/May/20 21:15
Affects Version/s: 3.0.0

hello,

we are very happy users of adaptive query execution. its a great feature to now have to think about and tune the number of partitions anymore in a shuffle.

i noticed that Dataset.groupBy consistently uses adaptive execution when its enabled (e.g. i don't see the default 200 partitions) but when i do Dataset.repartition it seems i am back to a hardcoded number of partitions.

Should adaptive execution also be used for repartition? It would be nice to be able to repartition without having to think about optimal number of partitions.

An example:
{code:java}
$ spark-shell --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=100000
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_252)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val x = (1 to 1000000).toDF
x: org.apache.spark.sql.DataFrame = [value: int]
scala> x.rdd.getNumPartitions
res0: Int = 2scala> x.repartition($"value").rdd.getNumPartitions
res1: Int = 200
scala> x.groupBy("value").count.rdd.getNumPartitions
res2: Int = 67
{code}
Bug: Wrong error message in VectorAssembler  when column lengths can not be inferred
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/20 17:33
Affects Version/s: 2.4.4

In VectorAssembler when input column lengths can not be inferred and handleInvalid = "keep", it will throw a runtime exception with message like below

_Can not infer column lengths with handleInvalid = "keep". *Consider using VectorSizeHint*_
 *_|to add metadata for columns: [column1, column2]_*

However, even if you set vector size hint for *column1*, the message remains, and will not change to  *[column2]* only. This is not consistent with the description in the error message.

This introduce difficulties when I try to resolve this exception, for I do not know which column required vectorSizeHint. This is especially troublesome when you have a large number of columns to deal with.

Here is a simple example:

 
{code:java}
// create a df without vector size
val df = Seq(
  (Vectors.dense(1.0), Vectors.dense(2.0))
).toDF("n1", "n2")

// only set vector size hint for n1 column
val hintedDf = new VectorSizeHint()
  .setInputCol("n1")
  .setSize(1)
  .transform(df)

// assemble n1, n2
val output = new VectorAssembler()
  .setInputCols(Array("n1", "n2"))
  .setOutputCol("features")
  .setHandleInvalid("keep")
  .transform(hintedDf)

// because only n1 has vector size, the error message should tell us to set vector size for n2 too
output.show()
{code}
Expected error message:

 
{code:java}
Can not infer column lengths with handleInvalid = "keep". Consider using VectorSizeHint to add metadata for columns: [n2].
{code}
Actual error message:
{code:java}
Can not infer column lengths with handleInvalid = "keep". Consider using VectorSizeHint to add metadata for columns: [n1, n2].
{code}
I change one line in VectorAssembler.scala, so that it can work properly as expected.
Bug: Python multiclass logistic regression evaluate should return LogisticRegressionSummary
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/20 22:09
Affects Version/s: 3.0.0

{code:java}
    def evaluate(self, dataset):
        ......
        java_blr_summary = self._call_java("evaluate", dataset)
        return BinaryLogisticRegressionSummary(java_blr_summary)
{code}

We should return LogisticRegressionSummary instead of BinaryLogisticRegressionSummary for multiclass LogisticRegression
Bug: PandasUDFType.GROUPED_AGG with Java 11
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 13/May/20 13:28
Affects Version/s: 3.0.0

Running the example from the [docs|https://spark.apache.org/docs/3.0.0-preview2/api/python/pyspark.sql.html#module-pyspark.sql.functions] gives an error with java 11. It works with java 8.


{code:python}
import findspark
findspark.init('/usr/local/lib/spark-3.0.0-preview2-bin-hadoop2.7')
from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql import Window
from pyspark.sql import SparkSession

if __name__ == '__main__':
    spark = SparkSession \
        .builder \
        .appName('test') \
        .getOrCreate()

    df = spark.createDataFrame(
        [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
        ("id", "v"))

    @pandas_udf("double", PandasUDFType.GROUPED_AGG)
    def mean_udf(v):
        return v.mean()

    w = (Window.partitionBy('id')
         .orderBy('v')
         .rowsBetween(-1, 0))
    df.withColumn('mean_v', mean_udf(df['v']).over(w)).show()
{code}


{noformat}
File "/usr/local/lib/spark-3.0.0-preview2-bin-hadoop2.7/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o81.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 44 in stage 7.0 failed 1 times, most recent failure: Lost task 44.0 in stage 7.0 (TID 37, 131.130.32.15, executor driver): java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:473)
	at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
	at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
	at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
	at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:240)
	at org.apache.arrow.vector.ipc.ArrowWriter.writeRecordBatch(ArrowWriter.java:132)
	at org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:120)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.$anonfun$writeIteratorToStream$1(ArrowPythonRunner.scala:94)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.writeIteratorToStream(ArrowPythonRunner.scala:101)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:373)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1932)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:213)
{noformat}
Bug: HistoryServer should set Content-Type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/20 06:59
Affects Version/s: 3.1.0

I noticed that we will get html as plain text when we access to wrong URLs on HistoryServer.
{code:java}
<html>
      <head>
        <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/static/bootstrap.min.css" type="text/css"/><link rel="stylesheet" href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link rel="stylesheet" href="/static/webui.css" type="text/css"/><link rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script src="/static/sorttable.js"></script><script src="/static/jquery-3.4.1.min.js"></script><script src="/static/vis-timeline-graph2d.min.js"></script><script src="/static/bootstrap.bundle.min.js"></script><script src="/static/initialize-tooltips.js"></script><script src="/static/table.js"></script><script src="/static/timeline-view.js"></script><script src="/static/log-view.js"></script><script src="/static/webui.js"></script><script>setUIRoot('')</script>
        
        <link rel="shortcut icon" href="/static/spark-logo-77x50px-hd.png"></link>
        <title>Not Found</title>
      </head>
      <body>
        <div class="container-fluid">
          <div class="row">
            <div class="col-12">
              <h3 style="vertical-align: middle; display: inline-block;">
                <a style="text-decoration: none" href="/">
                  <img src="/static/spark-logo-77x50px-hd.png"/>
                  <span class="version" style="margin-right: 15px;">3.1.0-SNAPSHOT</span>
                </a>
                Not Found
              </h3>
            </div>
          </div>
          <div class="row">
            <div class="col-12">
              <div class="row">Application local-1589239 not found.</div>
            </div>
          </div>
        </div>
      </body>
    </html> {code}
 

The reason is Content-Type not set.
{code:java}
HTTP/1.1 404 Not Found
Date: Wed, 13 May 2020 06:59:29 GMT
Cache-Control: no-cache, no-store, must-revalidate
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Length: 1778
Server: Jetty(9.4.18.v20190429) {code}
Bug: PrintStackTrace for Spark SQL CLI when error occurs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/20 11:08
Affects Version/s: 2.4.5

When I was finding the root cause of https://issues.apache.org/jira/browse/SPARK-31675, I noticed that it is very difficult for me to see what was actually going on, since it output nothing else but


{code:java}
Error in query: java.lang.IllegalArgumentException: Wrong FS: hdfs://cluster2/user/warehouse/bdms_hzyaoqin_test.db/abc/.hive-staging_hive_2020-05-11_17-10-27_123_6306294638950056285-1/-ext-10000/part-00000-badf2a31-ab36-4b60-82a1-0848774e4af5-c000, expected: hdfs://cluster1
{code}

It is really hard for us to find causes through such a simple error message.
Bug: Python side flatten the result dataframe of ANOVATest/ChisqTest/FValueTest
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/May/20 21:10
Affects Version/s: 3.1.0

Add Python version of

{code:java}
@Since("3.1.0")
def test(
    dataset: DataFrame,
    featuresCol: String,
    labelCol: String,
    flatten: Boolean): DataFrame 
{code}
Bug: Make Prometheus-format metric endpoints experimental
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/20 04:27
Affects Version/s: 3.0.0

Although the new metrics are disabled by default, we had better make it experimental in Apache Spark 3.0.0 since the output format is still not fixed. We will finalize it in Apache Spark 3.1.0.
Bug: RowEncoderSuite.encode/decode fails on 1000-02-29
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/20 09:14
Affects Version/s: 3.0.0

Here is the failure https://github.com/apache/spark/pull/28481#issuecomment-626034381:
{code}
org.scalatest.exceptions.TestFailedException:  schema: struct<null:null,boolean:boolean,byte:tinyint,short:smallint,int:int,long:bigint,float:float,double:double,decimal:decimal(38,18),string:string,binary:binary,date:date,timestamp:timestamp,udt:examplepoint> input: [null,true,127,-32768,684257610,3148440411416190456,Infinity,null,2.036236359763072870,ം뵡所碆ᚯ᧳ꒁ밯झ᧱휽⑲岫遳翁㎊륣䓵읹씶읽Â␣⪸붵끂꩖⟭䶄裻乌⇇깍뵙࿻偁뷩셙녶퐾귘嫫䍧쩔ꆁ䠾ՠ訣췐つ亙⚓깠긄蚣꿞묌泓㘡ⵆ橾櫻膋뿽⮎㖍杘䊣臼穇붘켑镅抎灕쿿ァ쏍㤰酀旬槳鑻槸놛턌춅ꉪ陪⡉法耸郄篍㹏吡ط汢측䱣 莶婚ⳟ슿쓻̷흖〦湶ဎ銓霁叹롄ᯕ珅䅃卩慗銁묠쯟ሄ啕澻矌軈憃䑋餤I쒚ᡭ⪩⚋湐蒒ジ䝱綅媪㍉芸礮猱耳藁笲⽽壶젅溜穸⫾룚྿뇳Ѩ䍢넪谦⎠줊넳楼橨䖊ꪗꚔ鬜⋍羯ሾ삦毜਷뢍෍⛛᭟莽糸꟤픣좖뮋撜혍牭ӎ뢂験ꆪᩉ跙㌌ᔸꦐ〷旽k텁ଘ쩧媉❛뛽뷺㱂᪭挃ቿ셾⁞邞郰홋쀘ᜍ뉿ഁ꓌迭梽ዳ硟崤쑼놱뎬蓬覄挗뾱뉍枈懂⼞ܭ갸첟ᢍ燃Ò䦛∫㦿ᶡ랗ђ䓸쑾퀷ၓ鍖霃솄⨗얔嫚ꨵ캁큰࿩ߢ䌵Ⓡ扛郾ꌟ䫀㑈瓺냾෍厌ᇗ玹띏푏㻛䰁ᤠ邰굇뷉恃ᦜ⶞쾖戀諕돚裹聼鬽劙Ἱ䏐烗䢭뉁ꏼⴾ欆⛺坶磩̿꽦⾩綬跩玉谩嶂퇗떾心鵈짘쒸봐傱䦂殏┗ח듅宥㣠ꘙ㽟忽ଟ겚鄀梧ж䋁癫剫㠉繮੫ݽ櫌非剖䤖噹뫒圏쬧罍氒ញ梶印䶋蝗杨윇鬑䰡笤㜇梀큦먚碈蝠⒊쩔蹂ૡወ쵩襒ᇳ擴ꓙ踘짧㤫倍趯鱘剨궐ঔ⇮ᶄಂ꼉⨛插柬ᒠ뇯뉒Ⱛ돝ヘ枀冗ꈑ筚綃놪㞴滅䷀ቿ䋃絚孝⏍ɞԃ灚诔懠卮쬸υ뇺闭䆲葞颫頴渋皒夂Ⲧ蟹폊綘ꥄ悈匢觏奴둇⺮웧쭑析윘ⴉ㯒罧䔫妬滢顂⺀ᶠ洷㈋祵鼲꿓阤煳⪧耒襠Ὀ蒣尥鴴⿥涜⭕넳ⶬꑷ㢭憾휦蘀暫줔䐳Ӏ膲뜊꾓휔⤻染肽㉟Ὲ돋⏦턝໫⨋噴䡧☘蟾違숶籩헺Ꮼ͵ळ⣣ੲ憋ጴ癤Ś泣ࢨ뎜뚗꽳텭⽊ꦞ⍝臬슯챑捒ᐑ薯闌巡猰恝ᘱ퇨倶掫ύ矞㹿䱟᭵ජꞥ푥✠儦慮௚齵ệ艝傫⠤⾿챔លͬ츂궄裐편ἵ핗곐촂Ѷ鋟ໄ櫷諩艄掽᧡輎ଇ颁굺㒔企鲺脞稯흂휾ꆲ駊㲹恾暤ź沥咺ଅᖣ嶀㱎쐢꼕㮚ⴞĒ䯭튔㹶ꯜꇙ廦㏚颿垌빫ࠣ悰흥꧆괱鈋暶ᭇ燙㐇뿜閆䩋쾽䉄ៈᵅ칇ચ厑济갺캜㤩봫껫衴㎱롺藪夞䃡㮛픳餣୅최껐ꮾꃼ友Ῡ磗༩ꡐ흏崋䰖牀㨊䞋ᓊ㺧೏ᔣꥱ룛ᚁ἟爥呯ᩮၥ㴳᭿㗀籧鮶噿浦ٰ癝牻⬬䷗㽂醙ꨞੇ굾鏬⑰酚곥ℰ菁εⓤ嶐媒帊녲湙犉ܒ啹⾧孨䜸錸ஊ쐡ᾫ㊮夒䇏繍힂ᡗ奄輽섚肫쀺왗隬㨖ⲝⵙ껽狇貥෫孒톶鄜趿滃逅ꨎ䫻⁡箚美뮣湾梠贉遚줐㞻䴳떛齿楂ᣀ䟯再ꨬ驂䉭ꇜ,[B@3f1a4861,1970-01-01,1000-02-29 10:11:12.123,null]
{code}
Bug: Mention removal of params mixins setter in migration guide
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 28/May/20 08:12
Affects Version/s: 3.0.0

In SPARK-29093, all setters have been removed from `Params` mixins in `pyspark.ml.param.shared`. Those setters had been part of the public pyspark ML API, hence this is a breaking change.
Bug: Make (SQL|ThriftServer)QueryTestSuite pass except Pandas UDF tests in Java 21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/22 14:14
Affects Version/s: 3.4.0

Some case in SQLQueryTestSuite(sql/core) and ThriftServerQueryTestSuite(sql/hive-thriftserver) failed for this reason:

for example:

 
{code:java}
SQLQueryTestSuite- 

try_aggregates.sql *** FAILED ***
  try_aggregates.sql
  Expected "4.61168601842738[79]E18", but got "4.61168601842738[8]E18" Result did not match for query #20
  SELECT try_avg(col) FROM VALUES (9223372036854775807L), (1L) AS tab(col) (SQLQueryTestSuite.scala:495) 

{code}
{code:java}
ThriftServerQueryTestSuite- try_aggregates.sql *** FAILED ***
  Expected "4.61168601842738[79]E18", but got "4.61168601842738[8]E18" Result did not match for query #20
  SELECT try_avg(col) FROM VALUES (9223372036854775807L), (1L) AS tab(col) (ThriftServerQueryTestSuite.scala:222)- try_arithmetic.sql *** FAILED ***
  Expected "-4.65661287307739[26]E-10", but got "-4.65661287307739[3]E-10" Result did not match for query #26
  SELECT try_divide(1, (2147483647 + 1)) (ThriftServerQueryTestSuite.scala:222)- datetime-{code}
Bug: Fix late record filtering to support chaining of steteful operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/22 17:46
Affects Version/s: 3.4.0

This is followup ticket on https://issues.apache.org/jira/browse/SPARK-40821.

Here we propose fixing the late record filtering in stateful operators to allow chaining of stateful operators which do not produce delayed records (like time-interval join of potentially flatMapGroupsWithState) - e.g. time-equality streaming join followed by aggregations or chaining of window aggregations.

 

There are 2 issues which need to be addressed:
 # Stateful operators filter input late records based on the current watermark. If e.g. chaining window aggregations, the records produced by the first window aggregation will be behind the current watermark by semantics (the watermark closes all past windows and emits the corresponding aggregates) and therefore these records will by definition appear late relative to the current watermark in the second stateful operator. The proposed fix for this issue is to use the previous batch watermark for late record filtering and the current batch watermark for state eviction - effectively each stateful operator should be initialized with 2 watermark values instead of 1.
 # The second issue with chaining window aggregators is that the records produced by the first aggregator do not have explicit event time column and thus can not be directly fed into a subsequent stateful operator which needs that column. This is partially handled by [https://github.com/apache/spark/pull/38288] so the user can explicitly introduce a new event time column by extracting the event time from the window column. This is slightly cumbersome. We propose changing the window function to handle the window column transparently - e.g.

input
  .withWatermark("eventTime", "1 seconds")
  .groupBy(window($"eventTime", "5 seconds") as 'window)
  .agg(count("*") as 'count)
  .groupBy({_}*window($"window", "10 seconds")*{_})
  .agg(count("*") as 'count, sum("count") as 'sum)
  .select($"count".as[Long], $"sum".as[Long])
Bug: Auto update table statistics based on write metrics
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Oct/22 02:18
Affects Version/s: 3.4.0

{code:scala}
  // Get write statistics
  def getWriteStats(mode: SaveMode, metrics: Map[String, SQLMetric]): Option[WriteStats] = {
    val numBytes = metrics.get(NUM_OUTPUT_BYTES_KEY).map(_.value).map(BigInt(_))
    val numRows = metrics.get(NUM_OUTPUT_ROWS_KEY).map(_.value).map(BigInt(_))
    numBytes.map(WriteStats(mode, _, numRows))
  }

// Update table statistics
      val stat = wroteStats.get
      stat.mode match {
        case SaveMode.Overwrite | SaveMode.ErrorIfExists =>
          catalog.alterTableStats(table.identifier,
            Some(CatalogStatistics(stat.numBytes, stat.numRows)))
        case _ if table.stats.nonEmpty => // SaveMode.Append
          catalog.alterTableStats(table.identifier, None)
        case _ => // SaveMode.Ignore Do nothing
{code}
Bug: Mismatch between ParquetFileFormat and FileSourceScanExec in # columns for WSCG.isTooManyFields when using _metadata
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/22 11:46
Affects Version/s: 3.3.0

_metadata.columns are taken into account in FileSourceScanExec.supportColumnar, but not when the parquet reader is created. This can result in Parquet reader outputting columnar (because it has less columns than WSCG.isTooManyFields), whereas FileSourceScanExec wants row output (because with the extra metadata columns it hits the isTooManyFields limit).

I have a fix forthcoming.
Bug: Introduce error sub-classes of the operation not allowed parse error
Status: Open | Priority: Major | Resolution: 
Created: 24/Oct/22 05:38
Affects Version/s: 3.4.0

Add an error class for operationNotAllowed() and sub-classes instead of passing error text. For instance,
https://github.com/apache/spark/blob/933dc0c42f0caf74aaa077fd4f2c2e7208452b9b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L3042
pass an error sub-class + message parameters instead of the text "Column ordering must be ASC, was ..."
Bug: Release a shaded version of Apache Spark / shade jars on main jar
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/22 18:26
Affects Version/s: 3.4.0

I suggest shading in Apache Spark, to resolve the dependency hell that may occur when building / deploying Apache Spark. This mainly occurs on Java projects and on Hadoop environments, but shading will help for using Spark with Scala & even Python either.
Flink has a similar solution, delivering [flink-shaded|https://github.com/apache/flink-shaded/blob/master/README.md].

The dependencies I think that are relevant for shading are Jackson, Guava, Netty & any of the Hadoop ecosystems if possible.

As for releasing sources for the shaded version, I think the [issue that has been raised in Flink|https://github.com/apache/flink-shaded/issues/25] is relevant and unanswered here too, hence I don't think that's an option currently (personally I don't see any value for it either).
Bug: Reuse the broadcast exchange for bloom filter
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Oct/22 11:08
Affects Version/s: 3.4.0

Currently, if the creation side of bloom filter could be broadcasted, Spark cannot inject a bloom filter or InSunquery filter into the application side.

In fact, we can inject bloom filter which could reuse the broadcast exchange and improve performance.