av9ash/GitBugs huggingface dataset:
Bug: V2 write for type struct fails to handle case sensitivity on field names during resolution of V2 write command
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Jun/22 19:34
Affects Version/s: 3.1.1

Summary:

When a V2 write uses an input with a {{struct}} type which contains differences in the casing of field names, the {{caseSensitive}} config is not being honored, always doing a strict case sensitive comparison.

Repro:
{code:java}
CREATE TABLE tmp.test_table_to (key int, object struct<shardId:int>) USING ICEBERG;
CREATE TABLE tmp.test_table_from (key int, object struct<shardid:int>) USING HIVE;
INSERT OVERWRITE tmp.test_table_to SELECT 1 as key, object FROM tmp.test_table_from;{code}
The above results in Exception:
{code:java}
Error in query: unresolved operator 'OverwriteByExpression RelationV2[key#3, object#4] spark_catalog.tmp.test_table_to, true, false;
'OverwriteByExpression RelationV2[key#3, object#4] spark_catalog.tmp.test_table_to, true, false
+- Project [1 AS key#0, object#2]
   +- SubqueryAlias spark_catalog.tmp.test_table_from
      +- HiveTableRelation [`tmp`.`test_table_from`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, Data Cols: [key#1, object#2], Partition Cols: []]{code}
 

If the casing matches in the struct field names, the v2 write works as expected.
Bug: Improve the analysis exception of missing map key column
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jun/22 00:41
Affects Version/s: 3.4.0

Sometimes users forgot to add single quotes on the map key string literal, for example `map_col[a]`. In such a case, the Analyzer will throw an exception:
{code:java}
[MISSING_COLUMN] Column 'struct.a' does not exist. Did you mean one of the following? ...{code}
We can improve this message by saying that the user should append single quotes if the map key is a string literal.
Bug: Pandas UDF executed twice if used in projection followed by filter
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Jun/22 16:12
Affects Version/s: 3.2.1

In this scenario, a Pandas UDF will be executed twice:
 # Projection that applies a Pandas UDF
 # Filter

In the {{explain}} output of the example below, the Optimized Logical Plan and Physical Plan contain {{ArrowEvalPython}} twice:
{code:python}
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder.master('local[1]').getOrCreate()

df = spark.createDataFrame(
    [
        [1, 'one'],
        [2, 'two'],
        [3, 'three'],
    ],
    'int_col int, string_col string',
)

@F.pandas_udf('int')
def copy_int_col(s):
    return s

df = df.withColumn('int_col_copy', copy_int_col(df['int_col']))
df = df.filter(F.col('int_col_copy') >= 3)

df.explain(True)
{code}
{code:java}
== Parsed Logical Plan ==
'Filter ('int_col_copy >= 3)
+- Project [int_col#322, string_col#323, copy_int_col(int_col#322) AS int_col_copy#327]
   +- LogicalRDD [int_col#322, string_col#323], false

== Analyzed Logical Plan ==
int_col: int, string_col: string, int_col_copy: int
Filter (int_col_copy#327 >= 3)
+- Project [int_col#322, string_col#323, copy_int_col(int_col#322) AS int_col_copy#327]
   +- LogicalRDD [int_col#322, string_col#323], false

== Optimized Logical Plan ==
Project [int_col#322, string_col#323, pythonUDF0#332 AS int_col_copy#327]
+- ArrowEvalPython [copy_int_col(int_col#322)], [pythonUDF0#332], 200
   +- Project [int_col#322, string_col#323]
      +- Filter (pythonUDF0#331 >= 3)
         +- ArrowEvalPython [copy_int_col(int_col#322)], [pythonUDF0#331], 200
            +- LogicalRDD [int_col#322, string_col#323], false

== Physical Plan ==
*(3) Project [int_col#322, string_col#323, pythonUDF0#332 AS int_col_copy#327]
+- ArrowEvalPython [copy_int_col(int_col#322)], [pythonUDF0#332], 200
   +- *(2) Project [int_col#322, string_col#323]
      +- *(2) Filter (pythonUDF0#331 >= 3)
         +- ArrowEvalPython [copy_int_col(int_col#322)], [pythonUDF0#331], 200
            +- *(1) Scan ExistingRDD[int_col#322,string_col#323]
{code}
If the Pandas UDF is marked as non-deterministic (e.g. {{{}copy_int_col = copy_int_col.asNondeterministic(){}}}), then it is not executed twice.
Bug: Remove no-null conditional statements in SparkSubmitCommandBuilder#isThriftServer
Status: Open | Priority: Minor | Resolution: 
Created: 20/Jun/22 03:45
Affects Version/s: 3.3.0

{code:java}
private boolean isThriftServer(String mainClass) {
  return (mainClass != null &&
   mainClass.equals("org.apache.spark.sql.hive.thriftserver.HiveThriftServer2")); 
} {code}
No scenario that *mainClass* is null, because already have defensive code.
{code:java}
if (isExample && !isSpecialCommand) {
  checkArgument(mainClass != null, "Missing example class name.");
}

if (mainClass != null) {
  args.add(parser.CLASS);
  args.add(mainClass);
} {code}
Bug: Clean `core/temp-secrets/` after executing `SecurityManagerSuite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/22 09:38
Affects Version/s: 3.4.0

run the following commands:

 
{code:java}
mvn clean install -DskipTests -pl core -am 
mvn clean test -Dtest=none -DwildcardSuites=org.apache.spark.SecurityManagerSuite -pl core  
git status{code}
The `core/temp-secrets/` directory is not cleaned up
Bug: to_date function returns a null for the first week of the year
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 09/Jun/22 23:45
Affects Version/s: 3.1.2

When I use week of year in the to_date function, the first week of the year returns a null for many years.

```
df=pyrasa.sparkSession.createDataFrame([["2013-01"],["2013-02"],["2017-01"],["2018-01"]],["input"])
df.select(func.col("input"),func.to_date(func.col("input"),"yyyy-ww").alias("date")) \
  .show()
```

```
+-------+----------+
|  input|      date|
+-------+----------+
|2013-01|      null|
|2013-02|2013-01-06|
|2017-01|2017-01-01|
|2018-01|      null|
+-------+----------+


```


Why is this? Is it a bug in the to_date function?
Bug: Add option to skip initial message in Pregel API
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jun/22 15:22
Affects Version/s: 3.2.1

The current (3.2.1) [Pregel API|https://github.com/apache/spark/blob/5a3ba9b0b301a3b0c43f8d0d88e2b6bdce57d0e6/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala#L117] takes a parameter {{initialMsg: A}} where {{A : scala.reflect.ClassTag}} is the message type for the Pregel iterations. At the start of the iterative process, the user-supplied vertex update method {{vprog}} is called with the initial message.

However, in some cases, the start point for a message passing scheme is best described by starting with the {{message}} phase rather than the {{vprog}} phase, and in many cases the first message depends on individual vertex data (instead of a static initial message). In these cases, users are forced to add boilerplate to their {{vprog}} function to check if the message received is the {{initialMessage}} and ignore the message (leave the node state unchanged) if it is. This leads to less efficient (due to extra iteration and check) and less readable code.
 
My proposed solution is to change {{initialMsg}} to a parameter of type {{Option[A]}} with default {{{}None{}}}, and then inside {{Pregel.apply}} function, set:
{code:scala}
var g = initialMsg match {
  case Some(msg) => graph.mapVertices((vid, vdata) => vprog(vid, vdata, msg))
  case _ => graph
}
{code}
This way, the user chooses whether to start the iteration from the {{message}} or {{vprog}} phase. I believe this small change could improve user code readability and efficiency.

Note: The signature of {{GraphOps.pregel}} would have to be changed to match
Bug: Convert Inner Join With Aggregate To Semi Join
Status: In Progress | Priority: Minor | Resolution: 
Created: 09/Jun/22 11:03
Affects Version/s: 3.2.1

Inner Join with aggregation on one of its child it equivalent to a Semi Join with the aggregation eliminated If the aggregation is just doing grouping and all the grouping columns appears in the join condition.

For example, Inner Join like below

 
{code:java}
select * from table1 inner join (select key2 from table2 group by key2) subquery on table1.key1 = subquery.key2{code}
can be transformed to a semi join and eliminate the aggregate in the subquery

 

 
{code:java}
select * from table1 semi join (select key2 from table2) subquery on table1.key1 = subquery.key2
{code}
 

I think this can be added as a optimize rule to save the unnecessary aggregation in the subquery.
Bug: Local mode supports HadoopDelegationTokenManager
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 08/Jun/22 13:35
Affects Version/s: 3.2.1

Now in the kerberos environment, using spark-submit --master=local --proxy-user xxx cannot access Hive Meta Store, and using --keytab will not automatically relogin.


{code:java}
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1743)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:483)
{code}
Bug: Hide and optimize `viewAcls`/`viewAclsGroups`/`modifyAcls`/`modifyAclsGroups` fron INFO log
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/22 11:57
Affects Version/s: 3.3.0

This issue aims to hide and optimize `viewAcls`/`viewAclsGroups`/`modifyAcls`/`modifyAclsGroups` fron INFO log.
{code:java}
2022-06-02 22:02:48.328 - stderr> 22/06/03 05:02:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set(){code}
Bug: Single column uses quoted to construct UnresolvedAttribute
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Jun/22 15:19
Affects Version/s: 3.2.0

 
{code:java}
select * from (select '2022-06-01' as c1 ) a where c1 in (select date_add('2022-06-01',0)); {code}
{code:java}
Error in query:
mismatched input '(' expecting {<EOF>, '.', '-'}(line 1, pos 8)
== SQL ==
date_add(2022-06-01, 0)
--------^^^ {code}
Bug: Have the arguments passed via spark.executor.extraJavaOptions placed before the classpath when composing the executors commandline
Status: Open | Priority: Trivial | Resolution: 
Created: 01/Jun/22 17:27
Affects Version/s: 2.4.8

When submitting a job it's possible to pass java options to be applied to the executor processes via the configuration parameter {{spark.executor.extraJavaOptions}} as in 
{noformat}
--conf 'spark.driver.extraJavaOptions=-Dmycompany.app.name=some_name'{noformat}
The command line for the executors is composed as
{noformat}
java <$SPARK_CONF_DIF/java.opts options>  -cp <classpath> -Xmx <extra_java_options> ...{noformat}
Since the classpath can be particularly long, it would be helpful for the purpose of visibility  when systems are monitored with tools that show the first part of the command line of a process (e.g nmon), to get the extra java options moved before classpath, as in
{noformat}
java <$SPARK_CONF_DIR/java.opts options> <extra_java_options> -cp <classpath> -Xmx  ...{noformat}
The type of extra options that would be passed in this way would be application and submission dependent and so it would not be appropriate or even possible to place them inside {{{}$SPARK_CONF_DIR/java-opts{}}}.

---

For reference, the code that builds the command line is [buildSparkSubmitCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L263], where, one of the first actions is calling  [buildJavaCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L92] that prepare the first part of the string from the "java" command to "-cp classpath" included.
Bug: Avro reader drops rows
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Nov/20 00:48
Affects Version/s: 3.1.0

Under certain circumstances, the V1 Avro reader drops rows. For example:
{noformat}
scala> val df = spark.range(0, 25).toDF("index")
df: org.apache.spark.sql.DataFrame = [index: bigint]

scala> df.write.mode("overwrite").format("avro").save("index_avro")

scala> val loaded = spark.read.format("avro").load("index_avro")
loaded: org.apache.spark.sql.DataFrame = [index: bigint]

scala> loaded.collect.size
res1: Int = 25

scala> loaded.orderBy("index").collect.size
res2: Int = 17   <== expected 25

scala> loaded.orderBy("index").write.mode("overwrite").format("parquet").save("index_as_parquet")

scala> spark.read.parquet("index_as_parquet").count
res4: Long = 17

scala>
{noformat}
SPARK-32346 slightly refactored the AvroFileFormat and AvroPartitionReaderFactory to use a new iterator-like trait called AvroUtils#RowReader. RowReader#hasNextRow consumes a raw input record and stores the deserialized row for the next call to RowReader#nextRow. Unfortunately, sometimes hasNextRow is called twice before nextRow is called, resulting in a lost row (see [BypassMergeSortShuffleWriter#write|https://github.com/apache/spark/blob/69c27f49acf2fe6fbc8335bde2aac4afd4188678/core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java#L132], which calls records.hasNext once before calling it again [here|https://github.com/apache/spark/blob/69c27f49acf2fe6fbc8335bde2aac4afd4188678/core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java#L155]).

RowReader consumes the Avro record in hasNextRow, rather than nextRow, because AvroDeserializer#deserialize potentially filters out the record.

Two possible fixes that I thought of:

1) keep state in RowReader such that multiple calls to RowReader#hasNextRow with no intervening call to RowReader#nextRow avoids consuming more than 1 Avro record. This requires no changes to any code that extends RowReader, just RowReader itself.
 2) Move record consumption to RowReader#nextRow (such that RowReader#nextRow could potentially return None) and wrap any iterator that extends RowReader with a new iterator created by flatMap. This last iterator will filter out the Nones and extract rows from the Somes. This requires changes to AvroFileFormat and AvroPartitionReaderFactory as well as RowReader.

The first one seems simplest and most straightfoward, and doesn't require changes to AvroFileFormat and AvroPartitionReaderFactory, only to AvroUtils#RowReader. So I propose this.
Bug: Vector reader got incorrect data with binary partition value
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Nov/20 03:06
Affects Version/s: 2.0.2

{code:java}
test("Parquet vector reader incorrect with binary partition value") {
  Seq(false, true).foreach(tag => {
    withSQLConf("spark.sql.parquet.enableVectorizedReader" -> tag.toString) {
      withTable("t1") {
        sql(
          """CREATE TABLE t1(name STRING, id BINARY, part BINARY)
            | USING PARQUET PARTITIONED BY (part)""".stripMargin)
        sql(s"INSERT INTO t1 PARTITION(part = 'Spark SQL') VALUES('a', X'537061726B2053514C')")
        if (tag) {
          checkAnswer(sql("SELECT name, cast(id as string), cast(part as string) FROM t1"),
            Row("a", "Spark SQL", ""))
        } else {
          checkAnswer(sql("SELECT name, cast(id as string), cast(part as string) FROM t1"),
            Row("a", "Spark SQL", "Spark SQL"))
        }
      }
    }
  })
}
{code}
Bug: SimplifyConditionals simplifies non-deterministic expressions
Status: Resolved | Priority: Blocker | Resolution: Not A Bug
Created: 06/Nov/20 05:12
Affects Version/s: 2.4.7

It seems we simplified non-deterministic expressions with aliases. for example:
{code:sql}
CREATE TABLE t(a int, b int, c int) using parquet
{code}

{code:sql}
sql
SELECT CASE 
 WHEN rand(100) > 1 THEN 1 
 WHEN rand(100) + 1 > 1000 THEN 1 
 WHEN rand(100) + 2 < 100 THEN 1 
 ELSE 1 
END AS x 
FROM t 
{code}

The plan is:
{noformat}
== Physical Plan ==
*(1) Project [CASE WHEN (rand(100) > 1.0) THEN 1 WHEN ((rand(100) + 1.0) > 1000.0) THEN 1 WHEN ((rand(100) + 2.0) < 100.0) THEN 1 ELSE 1 END AS x#6]
+- *(1) ColumnarToRow
 +- FileScan parquet default.t[] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
{noformat}


{code:sql}
SELECT CASE 
 WHEN rd > 1 THEN 1 
 WHEN rd + 1 > 1000 THEN 1 
 WHEN rd + 2 < 100 THEN 1 
 ELSE 1 
END AS x 
FROM (SELECT *, rand(100) as rd FROM t) t1 
{code}

The plan is:
{noformat}
== Physical Plan ==
*(1) Project [1 AS x#1]
+- *(1) ColumnarToRow
 +- FileScan parquet default.t[] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
{noformat}
Bug: Improve and fix cache behavior in v1 and v2
Status: Open | Priority: Critical | Resolution: 
Created: 20/Nov/20 23:15
Affects Version/s: 3.1.0

This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark datasource v1 and v2, which includes:
  - fix existing cache behavior in v1 and v2.
  - fix inconsistent cache behavior between v1 and v2
  - implement missing features in v2 to align with those in v1.
Bug: ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed
Status: In Progress | Priority: Critical | Resolution: 
Created: 04/Nov/20 17:48
Affects Version/s: 3.0.1

I launch my spark application with the [spark-on-kubernetes-operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] with the following yaml file:

{code:yaml}
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
   name: spark-kafka-streamer-test
   namespace: kafka2hdfs
spec: 
   type: Scala
   mode: cluster
   image: <my-repo>/spark:3.0.2-SNAPSHOT-2.12-0.1.0
   imagePullPolicy: Always
   timeToLiveSeconds: 259200
   mainClass: path.to.my.class.KafkaStreamer
   mainApplicationFile: spark-kafka-streamer_2.12-spark300-assembly.jar
   sparkVersion: 3.0.1
   restartPolicy:
     type: Always
   sparkConf:
     "spark.kafka.consumer.cache.capacity": "8192"
     "spark.kubernetes.memoryOverheadFactor": "0.3"
   deps:
   jars:
     - my
     - jar
     - list
   hadoopConfigMap: hdfs-config

   driver:
     cores: 4
     memory: 12g
     labels:
       version: 3.0.1
     serviceAccount: default
     javaOptions: "-Dlog4j.configuration=file:///opt/spark/log4j/log4j.properties"

  executor:
     instances: 4
    cores: 4
    memory: 16g
    labels:
      version: 3.0.1
    javaOptions: "-Dlog4j.configuration=file:///opt/spark/log4j/log4j.properties"
{code}

 I have tried with both Spark `3.0.1` and `3.0.2-SNAPSHOT` with the ["Restart the watcher when we receive a version changed from k8s"|https://github.com/apache/spark/pull/29533] patch.

This is the driver log:

{code}
20/11/04 12:16:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

... // my app log, it's a structured streaming app reading from kafka and writing to hdfs

20/11/04 13:12:12 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 1574101276 (1574213896)
 at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
 at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
 at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.base/java.lang.Thread.run(Unknown Source)
{code}

The error above appears after roughly 50 minutes.

After the exception above, no more logs are produced and the app hangs.
Bug: Spark SQL CLI command processing loop can't exit while one comand fail
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Nov/20 09:57
Affects Version/s: 3.0.0

 

When submit a multiple statements sql script through bin/spark-sql, if one of the prior command fail, the processing loop will not exit and continuing executing the following statements, and finally makes the whole program success. 

 
{code:java}
for (oneCmd <- commands) {
  if (StringUtils.endsWith(oneCmd, "\\")) {
    command += StringUtils.chop(oneCmd) + ";"
  } else {
    command += oneCmd
    if (!StringUtils.isBlank(command)) {
      val ret = processCmd(command)
      command = ""
      lastRet = ret
      val ignoreErrors = HiveConf.getBoolVar(conf, HiveConf.ConfVars.CLIIGNOREERRORS)
      if (ret != 0 && !ignoreErrors) {
        CommandProcessorFactory.clean(conf.asInstanceOf[HiveConf])
        ret // for loop will not return if one of the commands fail
      }
    }
  }

{code}