av9ash/GitBugs huggingface dataset:
Bug: Migrate spark-sql Java library from log4j to slf4j
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 18/Feb/22 18:10
Affects Version/s: 3.2.1

As there are numerous vulnerabilities in log4j and the project is no longer actively supported, Can we upgrade *spark-sql* Java library from log4j to slf4j.

This will also enable to easily integrate with log4j, logback and log4j2 without a breaking change.

 

+*Maven Dependency*+
{quote}{{<dependency>}}
{{  <groupId>org.apache.spark</groupId>}}
{{  <artifactId>spark-sql_2.13</artifactId>}}
{{  <version>3.2.1</version>}}
{{</dependency>}}
{quote}
 

+*Vulnerabilities*+
{quote}{{---------------------------------------------------------------}}
{{| SEVERITY  |  LIBRARY                      |  ID             |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2019-17571 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2020-9493  |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2021-4104  |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23302 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23305 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23307 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| LOW       |  log4j-1.2.17.jar             |  CVE-2020-9488  |}}
{{---------------------------------------------------------------}}
{quote}
Bug: [SQL] Spark JDBC Savemode Supports Upsert
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/22 08:02
Affects Version/s: 3.3.0

upsert sql for different databases， Most databases support merge sql：

sqlserver merge into sql : [https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/sqlserver/SqlServerDialect.java]

mysql: [https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/mysql/MysqlDialect.java]

oracle merge into sql : [https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oracle/OracleDialect.java]

postgres: [https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/psql/PostgresDialect.java]

postgres merg into sql : [https://www.postgresql.org/docs/current/sql-merge.html]

db2 merge into sql : [https://www.ibm.com/docs/en/db2-for-zos/12?topic=statements-merge]

derby merge into sql: [https://db.apache.org/derby/docs/10.14/ref/rrefsqljmerge.html]

he merg into sql : [https://www.tutorialspoint.com/h2_database/h2_database_merge.htm]

 

[~yao] 

 

https://github.com/melin/datatunnel/tree/master/plugins/jdbc/src/main/scala/com/superior/datatunnel/plugin/jdbc/support/dialect
Bug: InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Feb/22 10:23
Affects Version/s: 3.0.2

In `org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand`, `sparkSession.sessionState.catalog.listPartitions` will call method `org.apache.hadoop.hive.metastore.listPartitionsPsWithAuth` of hive metastore client, this method will produce multiple queries per partition on hive metastore db. So when you insert into a table which has too many partitions(ie: 10k), it will produce too many queries on hive metastore db(ie: n * 10k = 10nk), it puts a lot of strain on the database.

In fact, it calls method `listPartitions` in order to get locations of partitions and get `customPartitionLocations`. But in most cases, we do not have custom partitions, we can just get partition names, so we can call method listPartitionNames.
Bug: CSV parsing error when using escape='"' 
Status: Open | Priority: Major | Resolution: 
Created: 09/Feb/22 13:45
Affects Version/s: 3.2.1

hi all,

When reading CSV files with Spark, I ran into a parsing bug.

{*}The summary{*}:

When
 # reading a comma separated, double-quote quoted CSV file using the csv reader options _escape='"'_ and {_}header=True{_},
 # with a row containing a quoted empty field
 # followed by a quoted field starting with a comma and followed by one or more characters

selecting columns from the dataframe at or after the field described in 3) gives incorrect and inconsistent results

{*}In detail{*}:

When I instruct Spark to read this CSV file:

 
{code:java}
col1,col2
"",",a"
{code}
 

using the CSV reader options escape='"' (unnecessary for the example, necessary for the files I'm processing) and header=True, I expect the following result:

 
{code:java}
spark.read.csv(path, escape='"', header=True).show()
 
+----+----+
|col1|col2|
+----+----+
|null|  ,a|
+----+----+   {code}
 
 Spark does yield this result, so far so good. However, when I select col2 from the dataframe, Spark yields an incorrect result:

 
{code:java}
spark.read.csv(path, escape='"', header=True).select('col2').show()
 
+----+
|col2|
+----+
|  a"|
+----+{code}
 
If you run this example with more columns in the file, and more commas in the field, e.g. ",,,,,,,a", the problem compounds, as Spark shifts many values to the right, causing unexpected and incorrect results. The inconsistency between both methods surprised me, as it implies the parsing is evaluated differently between both methods. 

I expect the bug to be located in the quote-balancing and un-escaping methods of the csv parser, but I can't find where that code is located in the code base. I'd be happy to take a look at it if anyone can point me where it is.
Bug: Spark sql: 3.2.1 - Function of add_ Months returns an incorrect date
Status: Reopened | Priority: Major | Resolution: 
Created: 28/Feb/22 01:25
Affects Version/s: 3.2.1

Step to reproduce:

Version of spark sql: 3.2.1(latest  version in maven repository)

Run sql:

spark.sql("""SELECT ADD_MONTHS(last_day('2020-06-30'), -1)""").toPandas()

expect: 2020-05-31

actual: 2020-05-30 (x)

 

Version of spark sql: 2.4.3 

spark.sql("""SELECT ADD_MONTHS(last_day('2020-06-30'), -1)""").toPandas()

expect: 2020-05-31 

actual: 2020-05-31 (/)
Bug: Optimize one row plan in normal and AQE Optimizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 10:18
Affects Version/s: 3.3.0

Optimize the plan if its max row is equal to or less than 1 in these cases:

- if the child of sort max rows less than or equal to 1, remove the sort
- if the child of local sort max rows per partition less than or equal to 1, remove the local sort
- if the child of aggregate max rows less than or equal to 1 and it's grouping only (include the rewritten distinct plan), remove the aggregate
- if the child of aggregate max rows less than or equal to 1, set distinct to false in all aggregate expression
Bug: Add a config to control decoding of datetime as Java 8 classes
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 24/Feb/22 12:17
Affects Version/s: 3.3.0

Add new config that should control collect(), in particular, and allow to enable/disable to Java 8 types in the Thrift server. The config should solve the following issue:

When an user connects to the Thrift Server and a query involve a datasource connect which doesn't handle Java8 types, the user observes the following exception:

{code:java}
ERROR SparkExecuteStatementOperation: Error executing query with ac61b10a-486e-463b-8726-3b61da58582e, currentState RUNNING,  
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 8) (10.157.1.194 executor 0): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for schema of timestamp  
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, TimestampType, instantToMicros, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, loan_perf_date), TimestampType), true, false) AS loan_perf_date#1125  
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:239)  
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:210)  
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)  
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)  
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)  
{code}
Bug: regression when replacing a dataset view
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/22 17:09
Affects Version/s: 3.2.0

The below use case works well in 3.1 but failed in 3.2 and master.
{code:java}
sql("select 1").createOrReplaceTempView("v")
sql("select * from v").createOrReplaceTempView("v")
// in 3.1 it works well, and select will output 1
// in 3.2 it failed with error: "AnalysisException: Recursive view v detected (cycle: v -> v)"{code}
Bug: Implement support for DEFAULT values for columns in tables 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/22 00:22
Affects Version/s: 3.4.0

This story tracks the implementation of DEFAULT values for columns in tables.

CREATE TABLE and ALTER TABLE invocations will support setting column default values for future operations. Following INSERT, UPDATE, MERGE statements may then reference the value using the DEFAULT keyword as needed.

Examples:
{code:sql}
CREATE TABLE T(a INT, b INT NOT NULL);

-- The default default is NULL
INSERT INTO T VALUES (DEFAULT, 0);
INSERT INTO T(b)  VALUES (1);
SELECT * FROM T;
(NULL, 0)
(NULL, 1)

-- Adding a default to a table with rows, sets the values for the
-- existing rows (exist default) and new rows (current default).
ALTER TABLE T ADD COLUMN c INT DEFAULT 5;
INSERT INTO T VALUES (1, 2, DEFAULT);
SELECT * FROM T;
(NULL, 0, 5)
(NULL, 1, 5)
(1, 2, 5) {code}
Bug: Spark build fails in Windows
Status: Open | Priority: Major | Resolution: 
Created: 04/Feb/22 19:28
Affects Version/s: 3.3.0

java.lang.NoSuchMethodError: org.fusesource.jansi.AnsiConsole.wrapOutputStream(Ljava/io/OutputStream;)Ljava/io/OutputStream;
jline.AnsiWindowsTerminal.detectAnsiSupport(AnsiWindowsTerminal.java:57)
jline.AnsiWindowsTerminal.<init>(AnsiWindowsTerminal.java:27)

 

A similar issue is being faced by the quarkus project with latest Maven. 

[https://github.com/quarkusio/quarkus/issues/19491]

 

Upgrading the scala-maven-plugin seems to resolve the issue but this ticket can be a blocker

https://issues.apache.org/jira/browse/SPARK-36547
Bug: Upgrade Google guava to version 30.0-jre
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Feb/22 17:31
Affects Version/s: 3.3.0

This is duplicated many times like in [SPARK-32502|https://issues.apache.org/jira/browse/SPARK-32502]  

Apache Spark is using com.google.guava:guava version 14.0.1 which has two security issues.

[CVE-2018-10237|https://nvd.nist.gov/vuln/detail/CVE-2018-10237] 

[CVE-2020-8908|https://nvd.nist.gov/vuln/detail/CVE-2020-8908] 

We should upgrade to [version 30.0|https://mvnrepository.com/artifact/com.google.guava/guava/30.0-jre] 


I will add some links to what I have found about this issue 


[HIVE-25617:fix bug introduced by CVE-2020-8908|https://github.com/apache/hive/pull/2725]

[Upgrade Guava to 27|https://github.com/apache/druid/pull/10683]     

[HIVE-21961: Upgrade Hadoop to 3.1.4, Guava to 27.0-jre and Jetty to 9.4.20.v20190813|https://github.com/apache/hive/pull/1821]   

[Shade Guava manually|https://github.com/apache/druid/issues/6942] 


[[DISCUSS] Hadoop 3, dropping support for Hadoop 2.x|https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4]
Bug: Upgrade postgresql from 42.3.0 to 42.3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 16:45
Affects Version/s: 3.3.0

postgresql 42.3.0 has two security issues.  

[CVE-2022-21724|https://nvd.nist.gov/vuln/detail/CVE-2022-21724] 

and 

[Arbitrary File Write Vulnerability|https://github.com/advisories/GHSA-673j-qm5f-xpv8]

By upgrading postgresql from 42.3.0 to 42.3.3 we will resolve these issues.
Bug: Upgrade ansi-regex from 5.0.0 to 5.0.1 in /dev
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/22 10:40
Affects Version/s: 3.2.1

[CVE-2021-3807|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3807]
  
[releases notes at github|https://github.com/chalk/ansi-regex/releases]
Bug: The second range is not [0, 59] in the day time ANSI interval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 04:21
Affects Version/s: 3.3.0

[https://spark.apache.org/docs/latest/sql-ref-datatypes.html]
 * SECOND, seconds within minutes and possibly fractions of a second [0..59.999999]{{{{}}{}}}

{{Doc shows SECOND is seconds within minutes, it's range should be [0, 59]}}

 

But testing shows 99 second is valid:

{{>>> spark.sql("select INTERVAL '10 01:01:99' DAY TO SECOND")}}
{{{}DataFrame[INTERVAL '10 01:02:39' DAY TO SECOND: interval day to second]{}}}{{{{}}{}}}

 

Meanwhile, minute range check is ok, see below:

>>> spark.sql("select INTERVAL '10 01:60:01' DAY TO SECOND")
requirement failed: {color:#de350b}*minute 60 outside range [0, 59]*{color}(line 1, pos 16)

== SQL ==
select INTERVAL '10 01:60:01' DAY TO SECOND
----------------^^^
Bug: Encoding of java.time.Period always results in "INTERVAL '0-0' YEAR TO MONTH"
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 24/Feb/22 15:26
Affects Version/s: 3.2.0

{code}
val dates = Seq(
    Period.ZERO,
    Period.ofWeeks(2),
).toDS()
dates.show(false)
{code}

Results in:

{code}
+----------------------------+
|value                       |
+----------------------------+
|INTERVAL '0-0' YEAR TO MONTH|
|INTERVAL '0-0' YEAR TO MONTH|
+----------------------------+
{code}
Bug: Support the hidden file metadata in Streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 04:17
Affects Version/s: 3.2.1

Currently, querying the hidden file metadata struct `_metadata` will fail with `readStream`, `writeStream` APIs.
{code:java}
spark
  .readStream
  ...
  .select("_metadata")
  .writeStream
  ...
  .start(){code}
Need to expose the metadata output to `StreamingRelation` as well.
Bug: Fix mypy failure on DataFrame.to_numpy in pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/22 04:48
Affects Version/s: 3.3.0

{code}
starting mypy annotations test...
annotations failed mypy checks:
python/pyspark/pandas/generic.py:585: error: Incompatible return value type (got "Union[ndarray[Any, Any], ExtensionArray]", expected "ndarray[Any, Any]")  [return-value]
Found 1 error in 1 file (checked 324 source files)
1
{code}

https://github.com/apache/spark/runs/5298261168?check_suite_focus=true
Bug: Aggregate push down doesnt work using Spark SQL jdbc datasource with postgresql
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/22 12:30
Affects Version/s: 3.2.1

I am establishing a connection with postgresql using the Spark SQL jdbc datasource. I have started the spark shell including the postgres driver and I can connect and execute queries without problems. I am using this statement:
{code:java}
val df = spark.read.format("jdbc").option("url", "jdbc:postgresql://host:port/").option("driver", "org.postgresql.Driver").option("dbtable", "test").option("user", "postgres").option("password", "*******").option("pushDownAggregate",true).load()
{code}
I am adding the pushDownAggregate option because I would like the aggregations are delegated to the source. But for some reason this is not happening.

Reviewing this pull request, it seems that this feature should be merged into 3.2. [https://github.com/apache/spark/pull/29695]

I am making the aggregations considering the mentioned limitations. An example case where I don't see pushdown being done would be this one:
{code:java}
df.groupBy("name").max("age").show()
{code}

The results of the queryExecution are shown below:

{code:java}
scala> df.groupBy("name").max("age").queryExecution.executedPlan
res19: org.apache.spark.sql.execution.SparkPlan =
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[name#274], functions=[max(age#246)], output=[name#274, max(age)#544])
   +- Exchange hashpartitioning(name#274, 200), ENSURE_REQUIREMENTS, [id=#205]
      +- HashAggregate(keys=[name#274], functions=[partial_max(age#246)], output=[name#274, max#548])
         +- Scan JDBCRelation(test) [numPartitions=1] [age#246,name#274] PushedAggregates: [], PushedFilters: [], PushedGroupby: [], ReadSchema: struct<age:int,name:string>

scala> dfp.groupBy("name").max("age").queryExecution.toString
res20: String =
"== Parsed Logical Plan ==
Aggregate [name#274], [name#274, max(age#246) AS max(age)#581]
+- Relation [age#246] JDBCRelation(test) [numPartitions=1]

== Analyzed Logical Plan ==
name: string, max(age): int
Aggregate [name#274], [name#274, max(age#246) AS max(age)#581]
+- Relation [age#24...
{code}

What could be the problem? Should pushDownAggregate work in this case?
Bug: Select of a stream of window expressions fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/22 20:43
Affects Version/s: 3.1.3

The following query fails:
{noformat}
val df = spark.range(0, 20).map { x =>
  (x % 4, x + 1, x + 2)
}.toDF("a", "b", "c")

import org.apache.spark.sql.expressions._

val w = Window.partitionBy("a").orderBy("b")
val selectExprs = Stream(
  sum("c").over(w.rowsBetween(Window.unboundedPreceding, Window.currentRow)).as("sumc"),
  avg("c").over(w.rowsBetween(Window.unboundedPreceding, Window.currentRow)).as("avgc")
)

df.select(selectExprs: _*).show(false)
{noformat}
It fails with the following error:
{noformat}
org.apache.spark.sql.AnalysisException: Resolved attribute(s) avgc#23 missing from c#16L,a#14L,b#15L,sumc#21L in operator !Project [c#16L, a#14L, b#15L, sumc#21L, sumc#21L, avgc#23].;
{noformat}
If you change the Stream of window expressions to a Vector or List, the query succeeds.
Bug: Pin markupsafe to 2.0.1 fix linter failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 05:36
Affects Version/s: 3.3.0

The CI is failing at branch-3.2 due to "ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.9/dist-packages/markupsafe/_{_}init{_}_.py)". [https://github.com/apache/spark/runs/5262268902?check_suite_focus=true] For more detail.

We should downgrade the Python package "markupsafe" to 2.0.1 for fixing the CI. Referred to https://github.com/aws/aws-sam-cli/issues/3661#issuecomment-1044340547.
Bug: decodeUnsafeRows's iterators should close underlying input streams
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 10:46
Affects Version/s: 3.2.0

SPARK-34647 replaced the ZstdInputStream with ZstdInputStreamNoFinalizer. This meant that all usages of `CompressionCodec.compressedInputStream` would need to manually close the stream as this would no longer be handled by GC finaliser mechanism.

In SparkPlan, the result of `CompressionCodec.compressedInputStream` is wrapped in an Iterator which never calls close. This implementation needs to make use of NextIterator which allows for the closing of underlying streams.
Bug: Looks like the wrong package is available on the spark downloads page. The name reads pre built for hadoop3.3 but the tgz file is marked as hadoop3.2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Feb/22 10:45
Affects Version/s: 3.2.1

!https://files.slack.com/files-pri/T4S1WH2J3-F032FA551U7/screenshot_20220214-013156.jpg!

!https://files.slack.com/files-pri/T4S1WH2J3-F032FA551U7/screenshot_20220214-013156.jpg!

Does the tgz have Hadoop 3.3 but it was written wrong or it is 3.2 Hadoop version only?
if yes is hadoop comes with the S3 magic commitor support.
Bug: Group by a stream of complex expressions fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/22 21:02
Affects Version/s: 3.2.1

This query fails:
{noformat}
scala> Seq(1).toDF("id").groupBy(Stream($"id" + 1, $"id" + 2): _*).sum("id").show(false)
java.lang.IllegalStateException: Couldn't find _groupingexpression#24 in [id#4,_groupingexpression#23]
  at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
  at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:83)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:425)
  at org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:73)
  at org.apache.spark.sql.catalyst.expressions.BindReferences$.$anonfun$bindReferences$1(BoundAttribute.scala:94)
  at scala.collection.immutable.Stream.$anonfun$map$1(Stream.scala:418)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.$anonfun$map$1(Stream.scala:418)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
  at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
  at scala.collection.immutable.Stream.foreach(Stream.scala:534)
  at scala.collection.TraversableOnce.count(TraversableOnce.scala:152)
  at scala.collection.TraversableOnce.count$(TraversableOnce.scala:145)
  at scala.collection.AbstractTraversable.count(Traversable.scala:108)
  at org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection$.createCode(GenerateUnsafeProjection.scala:293)
  at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doConsumeWithKeys(HashAggregateExec.scala:623)
{noformat}
However, replace {{Stream}} with {{Seq}} and it works:
{noformat}
scala> Seq(1).toDF("id").groupBy(Seq($"id" + 1, $"id" + 2): _*).sum("id").show(false)
+--------+--------+-------+
|(id + 1)|(id + 2)|sum(id)|
+--------+--------+-------+
|2       |3       |1      |
+--------+--------+-------+

scala> 
{noformat}
Bug: Address the 'pyspark' tagged tests when "spark.sql.ansi.enabled" is True
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 01:29
Affects Version/s: 3.3.0

The default value of `spark.sql.ansi.enabled` is False in Apache Spark, but several tests are failed when the `spark.sql.ansi.enabled` is set to True.

For stability of project, it's always good to test passed regardless of the option value.