av9ash/GitBugs huggingface dataset:
Bug: `running-on-kubernetes` page render bad in v3.2.1(latest) website
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/22 07:32
Affects Version/s: 3.2.1

Looks like the `running-on-kubernetes` page encounterd some problems when published.

[1] https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties

(You can see bad format after #spark-properties)
- I also check the master branch (setup local env) and also v3.2.0 (https://spark.apache.org/docs/3.2.0/running-on-kubernetes.html) it works well, .- But for v3.2.1 tag, I couldn't install doc deps due to deps conflict.
I'm not very familiar with doc infra tool. Can anyone help to take a look?
Bug: Unify get preferred locations for shuffle in AQE
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Mar/22 07:12
Affects Version/s: 3.3.0

It has several issues with method `ShuffledRowRDD#getPreferredLocations`.
 * it does not respect the config `spark.shuffle.reduceLocality.enabled`, so we can not disable it.
 * it does not respect `REDUCER_PREF_LOCS_FRACTION`, so it has no effect if DAG schedule task to an executor who has less data. In worse, driver will take more memory to store the useless locations.
Bug: StackOverflowError with OR(data filter, partition filter)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 04:29
Affects Version/s: 3.2.1

If the filter has OR and contains both data filter and partition filter, 
e.g. p is partition col and id is data col

{code:java}
SELECT * FROM tmp WHERE (p = 0 AND id > 0) OR (p = 1 AND id = 2) 
{code}

throws StackOverflowError
Bug: Avoid runtime error in Dataset.summary() when ANSI mode is on
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 13:54
Affects Version/s: 3.3.0

When executing df.summary(), Spark SQL converts String columns as Double for the 

percentiles/mean/stddev metrics. 

This can cause runtime errors with ANSI mode on. 

Since this API is for getting a quick summary of the Dataframe, I suggest using "TryCast" for the problematic stats so that the API still works under ANSI mode.
Bug: Test the error class: CONCURRENT_QUERY
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 16:50
Affects Version/s: 3.4.0

Add at least one test for the error class *CONCURRENT_QUERY* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def concurrentQueryInstanceError(): Throwable = {
    new SparkConcurrentModificationException("CONCURRENT_QUERY", Array.empty)
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: INTERNAL_ERROR
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 18:00
Affects Version/s: 3.4.0

Add tests for the error class *INTERNAL_ERROR* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def logicalHintOperatorNotRemovedDuringAnalysisError(): Throwable = {
    new SparkIllegalStateException(errorClass = "INTERNAL_ERROR",
      messageParameters = Array(
        "Internal error: logical hint operator should have been removed during analysis"))
  }

  def cannotEvaluateExpressionError(expression: Expression): Throwable = {
    new SparkUnsupportedOperationException(errorClass = "INTERNAL_ERROR",
      messageParameters = Array(s"Cannot evaluate expression: $expression"))
  }

  def cannotGenerateCodeForExpressionError(expression: Expression): Throwable = {
    new SparkUnsupportedOperationException(errorClass = "INTERNAL_ERROR",
      messageParameters = Array(s"Cannot generate code for expression: $expression"))
  }

  def cannotTerminateGeneratorError(generator: UnresolvedGenerator): Throwable = {
    new SparkUnsupportedOperationException(errorClass = "INTERNAL_ERROR",
      messageParameters = Array(s"Cannot terminate expression: $generator"))
  }
  def methodNotDeclaredError(name: String): Throwable = {
    new SparkNoSuchMethodException(errorClass = "INTERNAL_ERROR",
      messageParameters = Array(
        s"""A method named "$name" is not declared in any enclosing class nor any supertype"""))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: MISSING_STATIC_PARTITION_COLUMN
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Mar/22 18:22
Affects Version/s: 3.4.0

Add a test for the error classes *MISSING_STATIC_PARTITION_COLUMN* to QueryCompilationErrorsSuite. The test should cover the exception throw in QueryCompilationErrors:

{code:scala}
  def missingStaticPartitionColumn(staticName: String): Throwable = {
    new AnalysisException(
      errorClass = "MISSING_STATIC_PARTITION_COLUMN",
      messageParameters = Array(staticName))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: CANNOT_CAST_DATATYPE
Status: Open | Priority: Minor | Resolution: 
Created: 31/Mar/22 16:42
Affects Version/s: 3.4.0

Add at least one test for the error class *CANNOT_CAST_DATATYPE* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def cannotCastFromNullTypeError(to: DataType): Throwable = {
    new SparkException(errorClass = "CANNOT_CAST_DATATYPE",
      messageParameters = Array(NullType.typeName, to.typeName), null)
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: "Prune unrequired child index" branch of ColumnPruning has wrong condition
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Mar/22 01:43
Affects Version/s: 3.2.1

The "prune unrequired references" branch has the condition:
{code:java}
case p @ Project(_, g: Generate) if p.references != g.outputSet => {code}
This is wrong as generators like Inline will always enter this branch as long as it does not use all the generator output.

 

Example:

 

input: <col1: array<struct<a: struct<a: int, b: int>, b: int>>>

 

Project(a.a as x)

- Generate(Inline(col1), ..., a, b)

 

p.references is [a]

g.outputSet is [a, b]

 

This bug makes us never enter the GeneratorNestedColumnAliasing branch below thus miss some optimization opportunities. The condition should be
{code:java}
g.requiredChildOutput.contains(!p.references.contains(_)) {code}
Bug: Disable Pandas usage logging for method calls inside @contextmanager functions
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Mar/22 15:48
Affects Version/s: 3.2.1

Currently, calls inside @contextmanager functions are treated as external for *with* statements.

For example, the below code records config.set_option calls inside ps.option_context(...)
{code:java}
with ps.option_context("compute.ops_on_diff_frames", True):
    pass {code}
We should disable usage logging for calls inside @contextmanager functions to improve accuracy of the usage data
Bug: Add PYSPARK_PANDAS_USAGE_LOGGER environment variable as an alias of KOALAS_USAGE_LOGGER
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/22 02:28
Affects Version/s: 3.3.0

There's one place missing about the renaming Koalas to pandas API on Spark (POS). We should add an alias for this.
Bug: Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Mar/22 01:25
Affects Version/s: 3.2.1

The Project(_, g: Generate) branch in GeneratorNestedColumnAliasing is only supposed to work for ExplodeBase generators but we do not explicitly return for other types like Inline. Currently the bug is not trigger because there is another bug in the "prune unrequired child" branch in the ColumnPruning which makes other generators like Inline always go to that branch even if it is not applicable.

 

An easy example to show the bug:

Input: <col1: int, col2: array<struct<field1 struct<field1: int, field2: int>, field2 int>>>

Project(field1.field1 as ...)

- Generate(Inline(col2), ..., field1, field2)

 

We will try to incorrectly push the .field1 on field1 into the input of the Inline (col2).
Bug: Column name or alias as an attribute of the PySpark Column class
Status: Open | Priority: Minor | Resolution: 
Created: 09/Mar/22 17:12
Affects Version/s: 3.2.1

Having the name of a column as an attribute of PySpark {{Column}} class instances can enable some convenient patterns, for example:

Applying a function to a column and aliasing with the original name:
{code:java}
values = F.col("values")
# repeating the column name as an alias
distinct_values = F.array_distinct(values).alias("values")
# re-using the existing column name
distinct_values = F.array_distinct(values).alias(values._name){code}
Checking the column name inside a custom function and applying conditional logic on the name:
{code:java}
def custom_function(col: Column) -> Column:
    if col._name == "my_column":
        return col.astype("int")
    return col.astype("string"){code}
The proposal in this issue is to add a property {{Column.\_name}} that obtains the name or alias of a column in a similar way as currently done in the {{Column.\_\_repr\_\_}} method: [https://github.com/apache/spark/blob/master/python/pyspark/sql/column.py#L1062.] The choice of {{_name}} intentionally avoids collision with the existing {{Column.name}} method, which is an alias for {{{}Column.alias{}}}.
Bug: Why doesn't shuffle hash join support build left table for left-outer-join but full-outer-join?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 03/Mar/22 06:00
Affects Version/s: 3.2.0

Why doesn't shuffle hash join support building left table for left-outer-join, but it supports building right table for full-outer-join?

!image-2022-03-03-16-34-17-909.png!

 

IMO, if left table is the build table, similar to full-outer-table,  we can first create a BitSet to record any mismatch of next joins, and iterate all rows from stream table iterator and look up the hash key in the hash relation.  

If no one from stream table can join with the built hash relation, then we iterate the hash relation and get relative value from the BitSet, so we can get the left-outer rows.

 

Does anyone help me? Thx~
Bug: Move eclipse.m2e Maven plugin config in its own profile
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Mar/22 11:40
Affects Version/s: 3.3.0

Today I had a weird issue with org.eclipse.m2e:lifecycle-mapping Maven fake-plugin:
{code:java}
WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available
[WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced {code}
 

It was weird because I didn't do any changes to my setup since yesterday when the maven build was working fine.

{*}T{*}he actual problem was that ./dev/make-distribution was failing to read the version from pom.xml. The warnings above was the only thing printed by "mvn help:evaluate -Dexpression=project.version" so I thought it was related and spent time investigating it. There is no need other developers to waste time on Eclipse M2E warnings!

 

org.eclipse.m2e:lifecycle-mapping is a hack that is used by Eclipse to map Maven plugins' lifecycle with Eclipse lifecycle. It does not affect plain Maven usage on the command line! There is no Maven artifact at [https://repo.maven.apache.org/maven2/org/eclipse/m2e] !

 

As explained at [https://stackoverflow.com/a/23707050/497381] the best way to setup Maven+m2e is by using a custom Maven profile that is auto-activated only by Eclipse when M2E plugin is being used:
{code:java}
<profile>
  <id>only-eclipse</id>
  <activation>
    <property>
      <name>m2e.version</name>
    </property>
  </activation>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.eclipse.m2e</groupId>
          <artifactId>lifecycle-mapping</artifactId>
          <version>1.0.0</version>
          <configuration>
            ...
          </configuration>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</profile> {code}
Bug: Test the error class: FAILED_RENAME_PATH
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 17:41
Affects Version/s: 3.4.0

Add at least one test for the error class *FAILED_RENAME_PATH* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def renamePathAsExistsPathError(srcPath: Path, dstPath: Path): Throwable = {
    new SparkFileAlreadyExistsException(errorClass = "FAILED_RENAME_PATH",
      Array(srcPath.toString, dstPath.toString))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: PIVOT_VALUE_DATA_TYPE_MISMATCH
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 20:11
Affects Version/s: 3.4.0

Add a test for the error classes *PIVOT_VALUE_DATA_TYPE_MISMATCH* to QueryCompilationErrorsSuite. The test should cover the exception throw in QueryCompilationErrors:

{code:scala}
  def pivotValDataTypeMismatchError(pivotVal: Expression, pivotCol: Expression): Throwable = {
    new AnalysisException(
      errorClass = "PIVOT_VALUE_DATA_TYPE_MISMATCH",
      messageParameters = Array(
        pivotVal.toString, pivotVal.dataType.simpleString, pivotCol.dataType.catalogString))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: INDEX_OUT_OF_BOUNDS
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 17:57
Affects Version/s: 3.4.0

Add at least one test for the error class *INDEX_OUT_OF_BOUNDS* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def indexOutOfBoundsOfArrayDataError(idx: Int): Throwable = {
    new SparkIndexOutOfBoundsException(errorClass = "INDEX_OUT_OF_BOUNDS", Array(idx.toString))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Spark does not find CTE inside nested CTE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/22 07:48
Affects Version/s: 3.2.0

Hello! 

Seems that when defining CTEs and using them inside another CTE in Spark SQL, Spark thinks the inner call for the CTE is a table or view, which is not found and then it errors with `Table or view not found: <CTE name>`
h3. Steps to reproduce
 # `pip install pyspark==3.2.0` (also happens with 3.2.1)
 # start pyspark console by typing `pyspark` in the terminal
 # Try to run the following SQL with `spark.sql(sql)`

 
{code:java}
  WITH mock_cte__users    AS (
           SELECT 1 AS id
       ),
       model_under_test          AS (
             WITH users    AS (
                      SELECT *
                        FROM mock_cte__users
                  )
           SELECT *
             FROM users
       )
SELECT *
  FROM model_under_test;{code}
Spark will fail with 

 
{code:java}
pyspark.sql.utils.AnalysisException: Table or view not found: mock_cte__users; line 8 pos 29; {code}
I don't know if this is a regression or an expected behavior of the new 3.2.* versions. This fix introduced in 3.2.0 might be related: https://issues.apache.org/jira/browse/SPARK-36447
Bug: Qualified star selection produces duplicated common columns after join then alias
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 19/Mar/22 00:43
Affects Version/s: 3.2.0

When joining two DataFrames and then aliasing the result, selecting columns from the resulting Dataset by a qualified star produces duplicates of the joined columns.
{code:scala}
scala> val df1 = Seq((1, 10), (2, 20)).toDF("a", "x")
df1: org.apache.spark.sql.DataFrame = [a: int, x: int]

scala> val df2 = Seq((2, 200), (3, 300)).toDF("a", "y")
df2: org.apache.spark.sql.DataFrame = [a: int, y: int]

scala> val joined = df1.join(df2, "a").alias("joined")
joined: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [a: int, x: int ... 1 more field]

scala> joined.select("*").show()
+---+---+---+
|  a|  x|  y|
+---+---+---+
|  2| 20|200|
+---+---+---+

scala> joined.select("joined.*").show()
+---+---+---+---+
|  a|  a|  x|  y|
+---+---+---+---+
|  2|  2| 20|200|
+---+---+---+---+

scala> joined.select("*").select("joined.*").show()
+---+---+---+
|  a|  x|  y|
+---+---+---+
|  2| 20|200|
+---+---+---+ {code}
This appears to be introduced by SPARK-34527, leading to some surprising behaviour. Using an earlier version, such as Spark 3.0.2, produces the same output for all three {{{}show(){}}}s.
Bug: SPIP: Simplified API for DL Inferencing
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 24/Mar/22 16:06
Affects Version/s: 3.0.0

h1. Background and Motivation

The deployment of deep learning (DL) models to Spark clusters can be a point of friction today.  DL practitioners often aren't well-versed with Spark, and Spark experts often aren't well-versed with the fast-changing DL frameworks.  Currently, the deployment of trained DL models is done in a fairly ad-hoc manner, with each model integration usually requiring significant effort.

To simplify this process, we propose adding an integration layer for each major DL framework that can introspect their respective saved models to more-easily integrate these models into Spark applications.  You can find a detailed proposal here: [https://docs.google.com/document/d/1n7QPHVZfmQknvebZEXxzndHPV2T71aBsDnP4COQa_v0]
h1. Goals
 - Simplify the deployment of pre-trained single-node DL models to Spark inference applications.
 - Follow pandas_udf for simple inference use-cases.
 - Follow Spark ML Pipelines APIs for transfer-learning use-cases.
 - Enable integrations with popular third-party DL frameworks like TensorFlow, PyTorch, and Huggingface.
 - Focus on PySpark, since most of the DL frameworks use Python.
 - Take advantage of built-in Spark features like GPU scheduling and Arrow integration.
 - Enable inference on both CPU and GPU.

h1. Non-goals
 - DL model training.
 - Inference w/ distributed models, i.e. "model parallel" inference.

h1. Target Personas
 - Data scientists who need to deploy DL models on Spark.
 - Developers who need to deploy DL models on Spark.
Bug: Test the error class: RENAME_SRC_PATH_NOT_FOUND
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 20:13
Affects Version/s: 3.4.0

Add a test for the error classes *RENAME_SRC_PATH_NOT_FOUND* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def renameSrcPathNotFoundError(srcPath: Path): Throwable = {
    new SparkFileNotFoundException(errorClass = "RENAME_SRC_PATH_NOT_FOUND",
      Array(srcPath.toString))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Test the error class: UNSUPPORTED_DATATYPE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 20:17
Affects Version/s: 3.4.0

Add a test for the error classes *UNSUPPORTED_DATATYPE* to QueryExecutionErrorsSuite. The test should cover the exception throw in QueryExecutionErrors:

{code:scala}
  def dataTypeUnsupportedError(dataType: String, failure: String): Throwable = {
    new SparkIllegalArgumentException(errorClass = "UNSUPPORTED_DATATYPE",
      messageParameters = Array(dataType + failure))
  }
{code}

For example, here is a test for the error class *UNSUPPORTED_FEATURE*: https://github.com/apache/spark/blob/34e3029a43d2a8241f70f2343be8285cb7f231b9/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala#L151-L170

+The test must have a check of:+
# the entire error message
# sqlState if it is defined in the error-classes.json file
# the error class
Bug: Add KubernetesUtils.buildEnvVars(WithFieldRef)? utility functions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/22 06:29
Affects Version/s: 3.4.0

There are many duplicate code patterns in Spark Code:
{code:java}
new EnvVarBuilder()
  .withName(key)
  .withValue(value)
  .build() {code}
{code:java}
new EnvVarBuilder()
   .withName(name)
     .withValueFrom(new EnvVarSourceBuilder()
       .withNewFieldRef(version, field)
       .build())
   .build()





{code}
 

[The assignment statement for executor envVar | https://github.com/apache/spark/blob/branch-3.3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L123-L185] has 63 lines.  We could introduce _buildEnvVarsWithKV_ and _buildEnvVarsWithFieldRef_ function to simplify the above code patterns.