av9ash/GitBugs huggingface dataset:
Bug: Send ExecutorMetricsUpdate EventLog appropriately
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 08:55
Affects Version/s: 3.2.0

In current EventLoggingListener, we won't write SparkListenerExecutorMetricsUpdate message at all
{code:java}
override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
  if (shouldLogStageExecutorMetrics) {
    event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
      liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor) =>
        // If the update came from the driver, stageKey1 will be the dummy key (-1, -1),
        // so record those peaks for all active stages.
        // Otherwise, record the peaks for the matching stage.
        if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
          val metrics = metricsPerExecutor.getOrElseUpdate(
            event.execId, new ExecutorMetrics())
          metrics.compareAndUpdatePeakValues(newPeaks)
        }
      }
    }
  }
}
{code}
It causes this effect that we can't get driver peakMemoryMetrics in SHS.  We can get executor's since it will update with TaskEnd events.
Bug: Linear Models standardization optimization
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 17/Mar/21 01:09
Affects Version/s: 3.1.1

Existing impl of standardization in linear models does *NOT* center the vectors by removing the means, for the purpose of keep the dataset sparsity.

However, this will cause feature values with small var be scaled to large values, and underlying solver like LBFGS can not efficiently handle this case. see SPARK-34448 for details.

If internal vectors are centers (like other famous impl, i.e. GLMNET/Scikit-Learn), the convergence ratio will be better. In the case in SPARK-34448, the number of iteration to convergence will be reduced from 93 to 6. Moreover, the final solution is much more close to the one in GLMNET.

luckily, we find a new way to 'virtually' center the vectors without densifying the dataset, iff:

1, fitIntercept is true;
 2, no penalty on the intercept, it seem this is always true in existing impls;
 3, no bounds on the intercept;

 

We will also need to check whether this new methods work in all other linear models (i.e, mlor/svc/lir/aft, etc.) as we expected , and introduce it into those models if possible.
Bug: conv() does not convert negative inputs to unsigned correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/21 17:38
Affects Version/s: 3.1.0

{noformat}
scala> spark.sql("select conv('-10', 11, 7)").show(20, 150)
+-----------------------+
|       conv(-10, 11, 7)|
+-----------------------+
|4501202152252313413456|
+-----------------------+
scala> spark.sql("select hex(conv('-10', 11, 7))").show(20, 150)
+----------------------------------------------+
|                         hex(conv(-10, 11, 7))|
+----------------------------------------------+
|3435303132303231353232353233313334313334353600|
+----------------------------------------------+
{noformat}

The correct result is 45012021522523134134555. The above output has an incorrect second-to-last digit (6 instead of 5) and the last digit is a non-printing character the null byte.

I tracked the bug down to NumberConverter.unsignedLongDiv returning incorrect results. I tried replacing with java.lang.Long.divideUnsigned and that fixed it.
Bug: Fix cases of corruption in merged shuffle blocks that are pushed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/21 16:06
Affects Version/s: 3.1.0

The {{RemoteBlockPushResolver}} which handles the shuffle push blocks and merges them was introduced in [#30062|https://github.com/apache/spark/pull/30062]. We have identified 2 scenarios where the merged blocks get corrupted:
 # {{StreamCallback.onFailure()}} is called more than once. Initially we assumed that the onFailure callback will be called just once per stream. However, we observed that this is called twice when a client connection is reset. When the client connection is reset then there are 2 events that get triggered in this order.

 * {{exceptionCaught}}. This event is propagated to {{StreamInterceptor}}. {{StreamInterceptor.exceptionCaught()}} invokes {{callback.onFailure(streamId, cause)}}. This is the first time StreamCallback.onFailure() will be invoked.
 * {{channelInactive}}. Since the channel closes, the {{channelInactive}} event gets triggered which again is propagated to {{StreamInterceptor}}. {{StreamInterceptor.channelInactive()}} invokes {{callback.onFailure(streamId, new ClosedChannelException())}}. This is the second time StreamCallback.onFailure() will be invoked.

 # The flag {{isWriting}} is set prematurely to true. This introduces an edge case where a stream that is trying to merge a duplicate block (created because of a speculative task) may interfere with an active stream if the duplicate stream fails.

Also adding additional changes that improve the code.
 # Using positional writes all the time because this simplifies the code and with microbenchmarking haven't seen any performance impact.
 # Additional minor changes.
Bug: Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/21 08:17
Affects Version/s: 3.2.0

[~shaneknapp] as we discussed [on the mailing list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html] Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).

[Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new method to configure the kubernetes client. Thanks in advance to use it for testing on the Jenkins after the Minikube version is updated.

 

Added by Shane:

we also need to move from the kvm2 virtualization layer to docker.  docker is a recommended driver w/the latest versions of minikube, and this will allow devs to more easily recreate the minikube/k8s env on their local workstations and run the integration tests in an identical environment as jenkins.

the TL;DR is that upgrading to docker works, except that the PV integration tests are failing due to a couple of possible reasons:

1) the 'spark-kubernetes-driver' isn't properly being loaded (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)

2) during the PV test run, the error message 'Given path (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in the logs.  however, the mk cluster *does* mount successfully to the local bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount and read/write successfully to it (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312548&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312548)

i could really use some help, and if it's useful, i can create some local accounts manually and allow ssh access for a couple of people to assist me.
Bug: Parquet with invalid chars on column name reads double as null when a clean schema is applied
Status: Open | Priority: Major | Resolution: 
Created: 15/Mar/21 19:09
Affects Version/s: 2.4.3

I have a parquet file that has data with invalid column names on it. [#Reference](https://issues.apache.org/jira/browse/SPARK-27442)  Here is the file [Invalid Header Parquet|https://drive.google.com/file/d/101WNWXnPwhjocSMVjkhn5jo85Ri_NydP/view?usp=sharing].

I tried to load this file with 

{{df = glue_context.read.parquet('invalid_columns_double.parquet')}}

{{df = df.withColumnRenamed('COL 1', 'COL_1')}}

{{df = df.withColumnRenamed('COL,2', 'COL_2')}}

{{df = df.withColumnRenamed('COL;3', 'COL_3') }}

and so on.

Now if i call

{{df.show()}}

it throws this exception that is still pointing to the old column name.

 {{pyspark.sql.utils.AnalysisException: 'Attribute name "COL 1" contains invalid character(s) among " ,;{}()\\n\\t=". Please use alias to rename it.;'}}

 

When i read about it in some blogs, there was suggestion to re-read the same parquet with new schema applied. So i did 

{{df = glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}}{{}}

 

and it works, but all the data in the dataframe are null. The same works for Strings
Bug: Incorrect star expansion logic MERGE INSERT * / UPDATE *
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/21 17:27
Affects Version/s: 3.0.2

The natural expectation of INSERT * or UPDATE * in MERGE is to assign source columns to target columns of the same name. However, the current logic here generates the assignment by position. https://github.com/apache/spark/commit/7cfd589868b8430bc79e28e4d547008b222781a5#diff-ed19f376a63eba52eea59ca71f3355d4495fad4fad4db9a3324aade0d4986a47R1214

This can very easily lead to incorrect results without the user realizing this odd behavior.
Bug: Nested higher-order functions broken in DSL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 21:55
Affects Version/s: 3.1.1

In Spark 3, if I have:

{code:java}
val df = Seq(
    (Seq(1,2,3), Seq("a", "b", "c"))
).toDF("numbers", "letters")
{code}
and I want to take the cross product of these two arrays, I can do the following in SQL:

{code:java}
df.selectExpr("""
    FLATTEN(
        TRANSFORM(
            numbers,
            number -> TRANSFORM(
                letters,
                letter -> (number AS number, letter AS letter)
            )
        )
    ) AS zipped
""").show(false)
+------------------------------------------------------------------------+
|zipped                                                                  |
+------------------------------------------------------------------------+
|[{1, a}, {1, b}, {1, c}, {2, a}, {2, b}, {2, c}, {3, a}, {3, b}, {3, c}]|
+------------------------------------------------------------------------+
{code}

This works fine. But when I try the equivalent using the scala DSL, the result is wrong:

{code:java}
df.select(
    f.flatten(
        f.transform(
            $"numbers",
            (number: Column) => { f.transform(
                $"letters",
                (letter: Column) => { f.struct(
                    number.as("number"),
                    letter.as("letter")
                ) }
            ) }
        )
    ).as("zipped")
).show(10, false)
+------------------------------------------------------------------------+
|zipped                                                                  |
+------------------------------------------------------------------------+
|[{a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}]|
+------------------------------------------------------------------------+
{code}

Note that the numbers are not included in the output. The explain for this second version is:

{code:java}
== Parsed Logical Plan ==
'Project [flatten(transform('numbers, lambdafunction(transform('letters, lambdafunction(struct(NamePlaceholder, lambda 'x AS number#442, NamePlaceholder, lambda 'x AS letter#443), lambda 'x, false)), lambda 'x, false))) AS zipped#444]
+- Project [_1#303 AS numbers#308, _2#304 AS letters#309]
   +- LocalRelation [_1#303, _2#304]

== Analyzed Logical Plan ==
zipped: array<struct<number:string,letter:string>>
Project [flatten(transform(numbers#308, lambdafunction(transform(letters#309, lambdafunction(struct(number, lambda x#446, letter, lambda x#446), lambda x#446, false)), lambda x#445, false))) AS zipped#444]
+- Project [_1#303 AS numbers#308, _2#304 AS letters#309]
   +- LocalRelation [_1#303, _2#304]

== Optimized Logical Plan ==
LocalRelation [zipped#444]

== Physical Plan ==
LocalTableScan [zipped#444]
{code}

Seems like variable name x is hardcoded. And sure enough: https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3647
Bug: Kubernetes and Minikube version upgrade for integration tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/21 05:23
Affects Version/s: 3.2.0

As [discussed in the mailing list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]: upgrading Minikube version from v0.34.1 to v1.7.3 and kubernetes version from v1.15.12 to v1.17.3.

Moreover Minikube version will be checked.

By making this upgrade we can simplify how the kubernetes client is configured for Minikube.
Bug: Adds a new job in GitHub Actions to check the output of TPC-DS queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 22:48
Affects Version/s: 3.2.0

This ticket aims at adding a new job in GitHub Actions to check the output of TPC-DS queries. There are some cases where we noticed runtime-realted bugs after merging commits (e.g. .SPARK-33822). Therefore, I think it is worth adding a new job in GitHub Actions to check query output of TPC-DS (sf=1).
Bug: Push down limit through window when partitionSpec is not empty
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Mar/21 10:00
Affects Version/s: 3.2.0

For example:
{code:sql}
SELECT *, ROW_NUMBER() OVER(PARTITION BY a ORDER BY b) AS rn FROM t LIMIT 10 ==>
SELECT *, ROW_NUMBER() OVER(PARTITION BY a ORDER BY b) AS rn FROM (SELECT * FROM t ORDER BY a, b LIMIT 10) tmp
{code}
Bug: Remove analyzing temp view again in CreateViewCommand
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/21 20:42
Affects Version/s: 3.2.0

Remove analyzing temp view again in CreateViewCommand. This can be done once all the caller passes analyzed plan to CreateViewCommand.

Reference:
https://github.com/apache/spark/pull/31652/files#r589544449
https://github.com/apache/spark/pull/31273/files#r581592786
Bug: Report metrics for streaming source through progress reporter with Kafka source use-case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/21 07:36
Affects Version/s: 3.2.0

Report Metrics for streaming source through progress reporter to users.

Add kafka micro batch streaming use-case to report stats of # of offsets for the current offset falling behind the latest.

 

SPARK-34366 and SPARK-34297 report metrics in spark ui, but this issue reports metrics through progress report, making it available to users via listener.
Bug: BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/21 10:21
Affects Version/s: 3.0.2

BoundAttribute issue will occur after optimization by BooleanSimplification and PushFoldableIntoBranches. How to reproduce this issue:
{code:scala}
spark.sql("CREATE TABLE t1 (a INT, b INT) USING parquet")
spark.sql("CREATE TABLE t2 (a INT, b INT) USING parquet")

      spark.sql(
        """
          |SELECT cnt,
          |       NOT ( buyer_id ) AS buyer_id2
          |FROM   (SELECT t1.a IS NOT NULL AS buyer_id,
          |               Count(*)         AS cnt
          |        FROM   t1
          |               INNER JOIN t2
          |                       ON t1.a = t2.a
          |        GROUP  BY 1) t 
          |""".stripMargin).collect()
{code}
{noformat}
Couldn't find a#4 in [CASE WHEN isnotnull(a#4) THEN 1 ELSE 2 END#10,count(1)#3L]
java.lang.IllegalStateException: Couldn't find a#4 in [CASE WHEN isnotnull(a#4) THEN 1 ELSE 2 END#10,count(1)#3L]
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:316)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:316)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:321)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:406)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:242)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:404)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:357)
{noformat}
Another case:
{code:scala}
spark.sql(
  """
    |SELECT cnt,
    |       CASE WHEN ( buyer_id = 2 AND cnt > 3 ) THEN 2 ELSE 3 END AS buyer_id2
    |FROM   (SELECT CASE WHEN t1.a IS NOT NULL THEN 1 ELSE 2 END AS buyer_id, Count(*) AS cnt
    |        FROM   t1 INNER JOIN t2 ON t1.a = t2.a
    |        GROUP  BY 1) t
    |""".stripMargin).collect()
{code}
Bug: Test actual size of year-month and day-time intervals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/21 19:58
Affects Version/s: 3.2.0

Add tests to https://github.com/apache/spark/blob/3a299aa6480ac22501512cd0310d31a441d7dfdc/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ColumnTypeSuite.scala#L52
Bug: Parquet with invalid chars on column name reads double as null when a clean schema is applied
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Mar/21 19:10
Affects Version/s: 2.4.3

I have a parquet file that has data with invalid column names on it. [#Reference](https://issues.apache.org/jira/browse/SPARK-27442)  Here is the file attached with this ticket.

I tried to load this file with 

{{df = glue_context.read.parquet('invalid_columns_double.parquet')}}

{{df = df.withColumnRenamed('COL 1', 'COL_1')}}

{{df = df.withColumnRenamed('COL,2', 'COL_2')}}

{{df = df.withColumnRenamed('COL;3', 'COL_3') }}

and so on.

Now if i call

{{df.show()}}

it throws this exception that is still pointing to the old column name.

 {{pyspark.sql.utils.AnalysisException: 'Attribute name "COL 1" contains invalid character(s) among " ,;{}()}}
{{n}}
 {{t=". Please use alias to rename it.;'}}

 

When i read about it in some blogs, there was suggestion to re-read the same parquet with new schema applied. So i did 

{{df = glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}}

 

and it works, but all the data in the dataframe are null. The same works for String datatypes
Bug: Support reconcile schemas based on index after nested column pruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 08:51
Affects Version/s: 3.0.2

How to reproduce this issue:
{code:scala}
spark.sql(
  """
    |CREATE TABLE `t1` (
    |  `_col0` INT,
    |  `_col1` STRING,
    |  `_col2` STRUCT<`c1`: STRING, `c2`: STRING, `c3`: STRING, `c4`: BIGINT>,
    |  `_col3` STRING)
    |USING orc
    |PARTITIONED BY (_col3)
    |""".stripMargin)

spark.sql("INSERT INTO `t1` values(1, '2', null, '2021-02-01')")

spark.sql("SELECT _col2.c1, _col0 FROM `t1` WHERE _col3 = '2021-02-01'").show
{code}


Error message:
{noformat}
java.lang.AssertionError: assertion failed: The given data schema struct<_col0:int,_col2:struct<c1:string>> has less fields than the actual ORC physical schema, no idea which columns were dropped, fail to read. Try to disable 
	at scala.Predef$.assert(Predef.scala:223)
	at org.apache.spark.sql.execution.datasources.orc.OrcUtils$.requestedColumnIds(OrcUtils.scala:159)
	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$3(OrcFileFormat.scala:180)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2620)
	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$1(OrcFileFormat.scala:178)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:117)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:165)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:94)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:756)
{noformat}
Bug: New SQL Function: TRY_CAST
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/21 14:08
Affects Version/s: 3.2.0

Add a new SQL function try_cast. try_cast is identical to CAST with `spark.sql.ansi.enabled` as true, except it returns NULL instead of raising an error. This expression has one major difference from `cast` with `spark.sql.ansi.enabled` as true: when the source value can't be stored in the target integral(Byte/Short/Int/Long) type, `try_cast` returns null instead of returning the low order bytes of the source value.

This is learned from Google BigQuery and Snowflake:
https://docs.snowflake.com/en/sql-reference/functions/try_cast.html
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#safe_casting
Bug: Add round trip tests for period <-> month and duration <-> micros
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/21 09:34
Affects Version/s: 3.2.0

Similarly to the test from the PR https://github.com/apache/spark/pull/31799, add tests:
1. Months -> Period -> Months
2. Period -> Months -> Period
3. Duration -> micros -> Duration
Bug: Temporary views should be analyzed during the analysis phase for all applicable commands
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/21 19:09
Affects Version/s: 3.2.0

There are commands such as "CREATE TEMP VIEW", "ALTER VIEW AS", "CREATE TEMP VIEW USING", etc. where the temporary view is resolved when the command runs. Instead, the analysis should be moved to the analyzer.

This is an umbrella JIRA to track subtasks to move analysis of temp views to the analyzer.
Bug: Jars downloaded with the --packages argument are not added to the classpath for executors.
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/21 15:47
Affects Version/s: 3.0.0

When Spark is run in local mode, it works as expected. However, when Spark is run in client mode, it copies the jars to the executor ($SPARK_HOME/work/<app id>/<executor id>), but never adds them to the classpath.

It might be worth noting that `spark.jars` does add the jars to the classpath, but unlike `spark.jars.packages` it doesn't automatically download the jar's compile dependencies.

 

```

spark = SparkSession.builder\
 .master(SPARK_MASTER)\
 .appName(APP_NAME)\
 ...
 .config("spark.jars.packages", DEPENDENCY_PACKAGES) \

...
 .getOrCreate()
 ```