av9ash/GitBugs huggingface dataset:
Bug: access apiserver by pod env
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/23 06:28
Affects Version/s: 3.1.2

When start spark on k8s，driver pod  use spark.kubernetes.driver.master to get apiserver address. This config  us  [https://kubernetes.default.svc|https://kubernetes.default.svc/] as default and do not care about the apiserver port.

In our case, apiserver port is not 443 will driver will throw connectException. As k8s doc mentioned （https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#directly-accessing-the-rest-api）, we can get master url by getting {{KUBERNETES_SERVICE_HOST}} and {{KUBERNETES_SERVICE_PORT_HTTPS}} environment variables from pod. So we add a new conf spark.kubernetes.driver.master.from.pod.env to allow driver get master url from env in cluster mode on k8s
Bug: `MapOutputTracker#getMapLocation` should respect  `spark.shuffle.reduceLocality.enabled`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/23 12:44
Affects Version/s: 3.5.0

Discuss as [https://github.com/apache/spark/pull/40307]

{{getPreferredLocations}} in {{ShuffledRowRDD}} should return {{Nil}} at the very beginning in case {{spark.shuffle.reduceLocality.enabled = false}} (conceptually).

This logic is pushed into MapOutputTracker though - and {{getPreferredLocationsForShuffle}} honors {{spark.shuffle.reduceLocality.enabled}} - but {{getMapLocation}} does not.

So the fix would be to fix {{getMapLocation}} to honor the parameter.
Bug: Allow ShuffleDriverComponent to declare if shuffle data is reliably stored
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 19:24
Affects Version/s: 3.5.0

Currently, if there is an executor node loss, we assume the shuffle data on that node is also lost. This is not necessarily the case if there is a shuffle component managing the shuffle data and reliably maintaining it (for example, in distributed filesystem or in a disaggregated shuffle cluster).

Downstream projects have patches to Apache Spark in order to workaround this issue, for example Apache Celeborn has [this|https://github.com/apache/incubator-celeborn/blob/main/assets/spark-patch/RSS_RDA_spark3.patch].
Bug: Implement CSV/JSON parsing funcions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 19:28
Affects Version/s: 3.4.0

Implement the following two methods in DataFrameReader:

 

 
{code:java}
/**
* Loads a `Dataset[String]` storing JSON objects (<a href="http://jsonlines.org/">JSON Lines
* text format or newline-delimited JSON</a>) and returns the result as a `DataFrame`.
*
* Unless the schema is specified using `schema` function, this function goes through the
* input once to determine the input schema.
*
* @param jsonDataset input Dataset with one JSON object per record
* @since 3.4.0
*/
def json(jsonDataset: Dataset[String]): DataFrame
/**
* Loads an `Dataset[String]` storing CSV rows and returns the result as a `DataFrame`.
*
* If the schema is not specified using `schema` function and `inferSchema` option is enabled,
* this function goes through the input once to determine the input schema.
*
* If the schema is not specified using `schema` function and `inferSchema` option is disabled,
* it determines the columns as string types and it reads only the first line to determine the
* names and the number of fields.
*
* If the enforceSchema is set to `false`, only the CSV header in the first line is checked
* to conform specified or inferred schema.
*
* @note if `header` option is set to `true` when calling this API, all lines same with
* the header will be removed if exists.
*
* @param csvDataset input Dataset with one CSV row per record
* @since 3.4.0
*/
def csv(csvDataset: Dataset[String]): DataFrame
{code}
 

For this we need a new message. We cannot use project because we don't know the schema upfront.

 
{code:java}
message Parse {
  // (Required) Input relation to Parse. The input is expected to have single text column.
  Relation input = 1;
  // (Required) The expected format of the text.
  ParseFormat format = 2;
  enum ParseFormat {
    PARSE_FORMAT_UNSPECIFIED = 0;
    PARSE_FORMAT_CSV = 1;
    PARSE_FORMAT_JSON = 2;
  }
}
{code}
Bug: Add the try_aes_decrypt() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/23 13:05
Affects Version/s: 3.4.0

Add new function try_aes_decrypt(). The function aes_decrypt() fails w/ an exception when it faces to a column value which it cannot decrypt. So, if a column contains bad and good input, it is impossible to decrypt even good input.
Bug: How to use Fair Scheduler Pools
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 07/Mar/23 15:27
Affects Version/s: 3.2.3

I have two questions to ask:
 #   I wrote a demo referring to the official website, but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1 always performs tasks before pool2
 #    What is the relationship between "spark.scheduler.mode" and "{{{}schedulingMode{}}}" in fairscheduler.xml？

 

 
{code:java}
object MultiJobTest {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf()
    conf.setAppName("test-pool").setMaster("local[1]")
    conf.set("spark.scheduler.mode", "FAIR")
    conf.set("spark.scheduler.allocation.file", "file:///D:/tmp/input/fairscheduler.xml")
    val sparkContext = new SparkContext(conf)
    val data: RDD[String] = sparkContext.textFile("file:///D:/tmp/input/input.txt")

    val rdd = data.flatMap(_.split(","))
      .map(x => (x(0), x(0)))

    new Thread(() => {


      sparkContext.setLocalProperty("spark.scheduler.pool", "pool1")
      rdd.foreachAsync(x => {
        println("1======start==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
        Thread.sleep(10000)
        println("1======end==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
      })

    }).start()

    new Thread(() => {

      sparkContext.setLocalProperty("spark.scheduler.pool", "pool2")
      rdd.foreachAsync(x => {
        println("2======start==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
        Thread.sleep(10000)
        println("2======end==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
      })
    }).start()


    TimeUnit.MINUTES.sleep(2)
    sparkContext.stop()

  }
} {code}
 

fairscheduler.xml

 
{code:java}
<?xml version="1.0"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at       http://www.apache.org/licenses/LICENSE-2.0   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
--><allocations>
  <pool name="pool1">
    <schedulingMode>FAIR</schedulingMode>
    <weight>100</weight>
    <minShare>0</minShare>
  </pool>
  <pool name="pool2">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>0</minShare>
  </pool>
</allocations> {code}
 

 

input.txt

 
{code:java}
aa bb {code}
Bug: Remove experimental warning in developer documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/23 03:17
Affects Version/s: 3.4.0

{code}
diff --git a/connector/connect/README.md b/connector/connect/README.md
index 6567daf5504..dfe49cea3df 100644
--- a/connector/connect/README.md
+++ b/connector/connect/README.md
@@ -1,8 +1,5 @@
 # Spark Connect

-**Spark Connect is a strictly experimental feature and under heavy development.
-All APIs should be considered volatile and should not be used in production.**
-
 This module contains the implementation of Spark Connect which is a logical plan
 facade for the implementation in Spark. Spark Connect is directly integrated into the build
 of Spark.
diff --git a/python/pyspark/sql/connect/__init__.py b/python/pyspark/sql/connect/__init__.py
index 9bd4513db22..8b5d30e214c 100644
--- a/python/pyspark/sql/connect/__init__.py
+++ b/python/pyspark/sql/connect/__init__.py
@@ -15,5 +15,4 @@
 # limitations under the License.
 #

-"""Currently Spark Connect is very experimental and the APIs to interact with
-Spark through this API are can be changed at any time without warning."""
+"""Spark Connect cleint"""
{code}
Bug: Add h2 as test dependency of connect-server module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/23 12:40
Affects Version/s: 3.4.0

run 
 # mvn clean install -DskipTests -pl connector/connect/server -am
 # mvn test -pl connector/connect/server

{code:java}
*** RUN ABORTED ***
  java.lang.ClassNotFoundException: org.h2.Driver
  at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
  at java.base/java.lang.Class.forName0(Native Method)
  at java.base/java.lang.Class.forName(Class.java:398)
  at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
  at org.apache.spark.sql.connect.ProtoToParsedPlanTestSuite.beforeAll(ProtoToParsedPlanTestSuite.scala:68)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
  ...
 {code}
Bug: Relax ordering constraint for ALTER TABLE ADD|REPLACE column options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 08:36
Affects Version/s: 3.5.0

Currently the grammar for ALTER TABLE ADD|REPLACE column is:

qualifiedColTypeWithPosition
    : name=multipartIdentifier dataType (NOT NULL)? defaultExpression? commentSpec? colPosition?
    ;

This enforces a constraint on the order of: (NOT NULL, DEFAULT value, COMMENT value FIRST|AFTER value). We can update the grammar to allow these options in any order instead, to improve usability.
Bug: insert overwrite table will casue table location lost if java.lang.ArithmeticException is thrown
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/23 09:29
Affects Version/s: 3.2.3

When use [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/] and when a `insert overwrite` statment meet exception ，a no partion table's home directory will lost ,partion table will lost partion directory.
 
my spark-defaults.conf config : 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550
Bug: SubqueryAlias should propagate metadata columns its child already selects 
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Mar/23 17:43
Affects Version/s: 3.3.2

The `AddMetadataColumns` analyzer rule intends to make resolve available metadata columns, even if the plan already contains projections that did not explicitly mention the metadata column.

The `SubqueryAlias` plan node intentionally does not propagate metadata columns automatically from a non-leaf/non-subquery child node, because the following should _not_ work:

 
{code:java}
spark.read.table("t").select("a", "b").as("s").select("_metadata"){code}
However, today it is too strict in breaks the metadata chain, in case the child node's output already includes the metadata column:

 
{code:java}
// expected to work (and does)
spark.read.table("t")
  .select("a", "b").select("_metadata")

// by extension, should also work (but does not)
spark.read.table("t").select("a", "b", "_metadata").as("s")
  .select("a", "b").select("_metadata"){code}
The solution is for `SubqueryAlias` to always propagate metadata columns that are already in the child's output, thus preserving the `metadataOutput` chain for that column.
Bug: Implement Dataset.toJson
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Mar/23 19:39
Affects Version/s: 3.4.0

Implement Dataset.toJSON:

 
{code:java}
/**
* Returns the content of the Dataset as a Dataset of JSON strings.
* @since 3.4.0
*/
def toJSON: Dataset[String]{code}
 

Please see if we can implement this using {{{}project(to_json(struct(*))).as(StringEncoder){}}}.
Bug: Speed up parquet reading with Java Vector API
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/23 08:55
Affects Version/s: 3.5.0

Parquet has supported use Java 17 Vector API to perform bit-unpacking to enjoy 4x ~ 8x performance gain in microbenchmark.

I have finished the TPC-H(SF100) benchmark with spark integrated parquet optimization, each SQL has a different performance gain,  Q6 can reach up 11%

 

Please assign it to me, I will summit a PR, thanks!
Bug: Client mode submit task client should keep same exitcode with AM
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Mar/23 10:51
Affects Version/s: 3.5.0

```
    try {
      app.start(childArgs.toArray, sparkConf)
    } catch {
      case t: Throwable =>
        throw findCause(t)
    } finally {
      if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
        !isThriftServer(args.mainClass)) {
        try {
          SparkContext.getActive.foreach(_.stop())
        } catch {
          case e: Throwable => logError(s"Failed to close SparkContext: $e")
        }
      }
    }
  }
```
Bug: `simple udf` test failed using Maven 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/23 17:41
Affects Version/s: 3.4.0

{code:java}
simple udf *** FAILED ***
  io.grpc.StatusRuntimeException: INTERNAL: org.apache.spark.sql.ClientE2ETestSuite
  at io.grpc.Status.asRuntimeException(Status.java:535)
  at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  at org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:61)
  at org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:106)
  at org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:123)
  at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2426)
  at org.apache.spark.sql.Dataset.withResult(Dataset.scala:2747)
  at org.apache.spark.sql.Dataset.collect(Dataset.scala:2425)
  at org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$8(ClientE2ETestSuite.scala:85)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) {code}
Bug: Write temp checkpoints for streaming queries to local filesystem even if default FS is set differently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 03:29
Affects Version/s: 3.4.0

Write temp checkpoints for streaming queries to local filesystem even if default FS is set differently

 

We have seen cases where the default FS could be a remote file system and since the path for streaming checkpoints is not specified explcitily, this could cause pileup under 2 cases:
 * query exits with exception and the flag to force checkpoint removal is not set
 * driver/cluster terminates without query being terminated gracefully
Bug: current_user() is blocked from VALUES, but current_timestamp() is not
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/23 20:30
Affects Version/s: 3.3.0

VALUES(current_user());
returns:

cannot evaluate expression current_user() in inline table definition.; line 1 pos 8

 

The same with current_timestamp() works.

It appears current_user() is recognized as non-deterministic. But it is constant within the statement, just like current_timestanmp().

PS: It's not clear why we block non-deterministic functions to begin with....
Bug: Handle timeouts and CRC failures during artifact transfer
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/23 22:46
Affects Version/s: 3.4.0

We would need a retry mechanism on the client side to handle CRC failures during artifact transfer because the server would discard data that fails CRC and hence, may lead to missing artifacts during UDF execution. 

We also require a timeout policy to prevent indefinitely waiting for the server reply.
Bug: Introduce feature to allow for function caching across input rows.
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/23 04:57
Affects Version/s: 3.3.2

Introduce the ability to make functions cachable across input rows. I'm imagining this function to work similarly to python's [functools.cache|https://docs.python.org/3/library/functools.html#functools.cache] where you could add a decorator to certain expensive functions that you know will regularly encounter repeated values as you read the input data.

 

With this new feature you would be able to significantly speed up many real world jobs that use expensive functions on data that naturally has repeated column values. An example of this would be parsing user agent fields from internet traffic logs partitioned by user id. Even though the data is not sorted by user agent, in a sample of 10k continuous rows there would be much less than 10k unique values because popular user agents exist on a large fraction of traffic and the user agent of the first event from a user is likely to be shared among all subsequent events from that user. Currently there is a way to hack an approximation of this in a python implementation of this via pandas_udfs. This works because pandas_udfs by default read in batches of 10k input rows, so you can used a caching UDF that empties every 10k rows. At my current job I have noticed that some applications of this trick can significantly speed up queries where custom UDFs are the bottleneck in a query. An example of this is

 
{code:java}
@F.pandas_udf(T.StringType())
def parse_user_agent_field(user_agent_series):
    @functools.cache
    def parse_user_agent_field_helper(user_agent):
        # parse the user agent and return the relevant field
        return None
    return user_agent_series.apply(parse_user_agent_field_helper){code}
 

 

It would be nice if there was some official support for this behavior for both built in functions and UDFs. If there was official support for this I'd imagine it to look something like

 
{code:java}
# using pyspark dataframe API
df = df.withColumn(output_col, F.cache(F.function)(input_col)){code}
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2000
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:08
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2000* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2011
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:50
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2011* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]