av9ash/GitBugs huggingface dataset:
Bug: Mima complains with Scala 2.13 in the master branch
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jul/22 01:16
Affects Version/s: 3.4.0

{code}
[error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.2.0! Found 6 potential problems (filtered 933)
[error]  * the type hierarchy of object org.apache.spark.deploy.DeployMessages#LaunchExecutor is different in current version. Missing types {scala.runtime.AbstractFunction7}
[error]    filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.deploy.DeployMessages$LaunchExecutor$")
[error]  * method requestedTotal()Int in class org.apache.spark.deploy.DeployMessages#RequestExecutors does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.requestedTotal")
[error]  * method copy(java.lang.String,Int)org.apache.spark.deploy.DeployMessages#RequestExecutors in class org.apache.spark.deploy.DeployMessages#RequestExecutors's type is different in current version, where it is (java.lang.String,scala.collection.immutable.Map)org.apache.spark.deploy.DeployMessages#RequestExecutors instead of (java.lang.String,Int)org.apache.spark.deploy.DeployMessages#RequestExecutors
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy")
[error]  * synthetic method copy$default$2()Int in class org.apache.spark.deploy.DeployMessages#RequestExecutors has a different result type in current version, where it is scala.collection.immutable.Map rather than Int
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2")
[error]  * method this(java.lang.String,Int)Unit in class org.apache.spark.deploy.DeployMessages#RequestExecutors's type is different in current version, where it is (java.lang.String,scala.collection.immutable.Map)Unit instead of (java.lang.String,Int)Unit
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this")
[error]  * method apply(java.lang.String,Int)org.apache.spark.deploy.DeployMessages#RequestExecutors in object org.apache.spark.deploy.DeployMessages#RequestExecutors in current version does not have a correspondent with same parameter signature among (java.lang.String,scala.collection.immutable.Map)org.apache.spark.deploy.DeployMessages#RequestExecutors, (java.lang.Object,java.lang.Object)java.lang.Object
[error]    filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
{code}

https://github.com/apache/spark/runs/7221231391?check_suite_focus=true
Bug: Switch 'spark.sql.codegen.factoryMode' configuration from testing purpose to internal purpose
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jul/22 00:09
Affects Version/s: 3.4.0

{{spark.sql.codegen.factoryMode}} is currently only effective when SPARK_TESTING environment variable is set.

We can actually leverage this configuration in other testing bad that mimics production environment without setting SPARK_TESTING too.

This JIRA aims to make this configuration working without SPARK_TESTING environment variable.
Bug: pyspark drop doesn't accept *cols 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/22 09:49
Affects Version/s: 3.0.3

Pyspark dataframe drop has following signature:

{color:#4c9aff}{{def drop(self, *cols: "ColumnOrName") -> "DataFrame":}}{color}

However when we try to pass multiple Column types to drop function it raises TypeError

{{each col in the param list should be a string}}

*Minimal reproducible example:*
{color:#4c9aff}values = [("id_1", 5, 9), ("id_2", 5, 1), ("id_3", 4, 3), ("id_1", 3, 3), ("id_2", 4, 3)]{color}
{color:#4c9aff}df = spark.createDataFrame(values, "id string, point int, count int"){color}
|– id: string (nullable = true)|
|– point: integer (nullable = true)|
|– count: integer (nullable = true)|

{color:#4c9aff}{{df.drop(df.point, df.count)}}{color}
{quote}{color:#505f79}/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py in drop(self, *cols){color}
{color:#505f79}2537 for col in cols:{color}
{color:#505f79}2538 if not isinstance(col, str):{color}
{color:#505f79}-> 2539 raise TypeError("each col in the param list should be a string"){color}
{color:#505f79}2540 jdf = self._jdf.drop(self._jseq(cols)){color}
{color:#505f79}2541{color}

{color:#505f79}TypeError: each col in the param list should be a string{color}
{quote}
Bug: Upgrade HtmlUnit and its related artifacts from 2.50.0 to 2.62.0.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jul/22 02:41
Affects Version/s: 3.4.0

* We currently uses 2.50 but 2.51+ seem to include bunch of bug fixes and improvements.

           release notes: https://htmlunit.sourceforge.io/changes-report.html#a2.62.0

 
 * The last upgrade occurred 1 year and 1 month ago.
Bug: Issue with querying dataframe produced by 'binaryFile' format using 'not' operator
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/22 20:19
Affects Version/s: 3.2.1

When creating a dataframe using the binaryFile format I am encountering weird result when filtering/query with the 'not' operator.

 

Here's a repo that will help describe and reproduce the issue.

[https://github.com/cccs-br/spark-binaryfile-issue]
{code:java}
git@github.com:cccs-br/spark-binaryfile-issue.git {code}
 

Here's a very simple test case that illustrate what's going on:

[https://github.com/cccs-br/spark-binaryfile-issue/blob/main/src/test/scala/BinaryFileSuite.scala]

TLDR;
{code:java}
   test("binary file dataframe") {
    // load files in directly into df using 'binaryFile' format.
    //     
    // - src/test/resources/files/
    //  - test1.csv
    //  - test2.json
    //  - test3.txt
    val df = spark
      .read
      .format("binaryFile")
      .load("src/test/resources/files")

    df.createOrReplaceTempView("files")

    // This works as expected.
    val like_count = spark.sql("select * from files where path like '%.csv'").count()
    assert(like_count === 1)

    // This does not work as expected.
    val not_like_count = spark.sql("select * from files where path not like '%.csv'").count()
    assert(not_like_count === 2)

    // This used to work in 3.2.1
    // df.filter(col("path").endsWith(".csv") === false).show()
  }{code}
Bug: Add array_sort(column, comparator) overload to DataFrame operations
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jul/22 16:10
Affects Version/s: 3.3.0

The ability to use {{array_sort with a comparator was added in SPARK-29020; however, the new signature wasn't made available to the DataFrame operations API.}}

 

Proposed new signature:
{code:java}
package org.apache.spark.sql

object functions {
  ...

  def array_sort(e: Column, comparator: (Column, Column) => Column): Column

  ...
} {code}
Bug: Add flag to run-tests.py to retain the test output.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jul/22 11:21
Affects Version/s: 3.3.0

Today, when tests are executed using the `run-tests.py` script the output of the individual tests is deleted when the test successfully finishes. It would be great to have an additional option to retain the output for investigations.
Bug: In Spark 3.0 standalone cluster mode, unable to customize driver JVM path
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Jul/22 06:09
Affects Version/s: 3.0.0

In Spark 3.0 standalone mode, unable to customize driver JVM path, instead the JAVA_HOME of the spark-submit submission machine is used, but the JVM paths of my submission machine and the cluster machine are different
{code:java}
launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java

List<String> buildJavaCommand(String extraClassPath) throws IOException {
  List<String> cmd = new ArrayList<>();

  String firstJavaHome = firstNonEmpty(javaHome,
    childEnv.get("JAVA_HOME"),
    System.getenv("JAVA_HOME"),
    System.getProperty("java.home")); {code}
Bug: Add an additional usage example for the map_zip_with function
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/22 12:10
Affects Version/s: 3.3.0

The function map_zip_with in Spark SQL is roughly described in the docs. Of course, it can do much more.
 
Adding an additional usage example can better demonstrate how it handles maps with different keys and lead to a more general solution.
Bug: Test for parity of SQL functions between Python and JVM DataFrame API's
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jul/22 20:54
Affects Version/s: 3.3.0

Add a unit test that compares the available list of Python DataFrame functions in pyspark.sql.functions with those available in the Scala/Java DataFrame API in org.apache.spark.sql.functions.
Bug: Executor memory footprint substantially increases while reading zstd compressed parquet files
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jul/22 11:50
Affects Version/s: 3.2.0

Hi all,

 

While transitioning from the default snappy compression to zstd, we noticed a substantial increase in executor memory whilst *reading* and applying transformations on *zstd* compressed parquet files.

Memory footprint increased increased 3 fold in some cases, compared to reading and applying the same transformations on a parquet file compressed with snappy.

This behaviour only occurs when reading zstd compressed parquet files. Writing a zstd parquet file does not result in this behaviour.

To reproduce:
 # Set "spark.sql.parquet.compression.codec" to zstd
 # Write some parquet files, the compression will default to zstd after setting the option above
 # Read the compressed zstd file and run some transformations. Compare the memory usage of the executor vs running the same transformation on a parquet file with snappy compression.
Bug: Add a regexp_extract variant which returns an array of all the matched capture groups
Status: Open | Priority: Minor | Resolution: 
Created: 15/Jul/22 21:35
Affects Version/s: 3.2.2

 

regexp_extract only returns a single matched group. In a lot of cases we need to parse the entire string and get all the groups and for that we'll need to call it as many times as there are groups. The regexp_extract_all function doesn't solve this problem as it only works if all the groups have the same regex pattern.

 

_Example:_

I will provide an example and the current workaround that I use to solve this,

If I have the following dataframe and I would like to match the column 'a' with this pattern
{code:java}
"([A-Za-z]+), [A-Za-z]+, (\\d+)"{code}
|a|
|Hello, World, 1234|
|Good, bye, friend|

 

My expected output  is as follows:
|a|extracted_a|
|Hello, World, 1234|[Hello, 1234]|
|Good, bye, friend|[]|

 

However, to achieve this I have to take the following approach which seems very hackish.
1. Use regexp_replace to create a temporary string built using the extracted groups:
{code:java}
df.withColumn("extr" , F.regexp_replace("a", "([A-Za-z]+), [A-Za-z]+, (\\d+)", "$1_$2")){code}
A side effect of regexp_replace is that if the regex fails to match the entire string is returned.

 
|a|extracted_a|
|Hello, World, 1234|Hello_1234|
|Good, bye, friend|Good, bye, friend|

2. So, to achieve the desired result, a check has to be done to prune the rows that did not match with the pattern :
{code:java}
df = df.withColumn("extracted_a" , F.when(F.col("extracted_a")==F.col("a") , None).otherwise(F.col("extracted_a"))){code}
 
to get the following intermediate dataframe,
|a|extracted_a|
|Hello, World, 1234|Hello_1234|
|Good, bye, friend|null|

 

3. Before finally splitting the column 'extracted_a' based on underscores
{code:java}
df = df.withColumn("extracted_a" , F.split("extracted_a" , "[_]")){code}
which results in the desired result:

 
 
|a|extracted_a
|
|Hello, World, 1234|[Hello, 1234]|
|Good, bye, friend|null|
Bug: Improve LocalDirsFeatureStep to randomize local directories
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/22 16:56
Affects Version/s: 3.4.0

In org.apache.spark.util  getConfiguredLocalDirs  

 
{code:java}
if (isRunningInYarnContainer(conf)) {
  // If we are in yarn mode, systems can have different disk layouts so we must set it
  // to what Yarn on this system said was available. Note this assumes that Yarn has
  // created the directories already, and that they are secured so that only the
  // user has access to them.
  randomizeInPlace(getYarnLocalDirs(conf).split(","))
} else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {
  conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)
} else if (conf.getenv("SPARK_LOCAL_DIRS") != null) {
  conf.getenv("SPARK_LOCAL_DIRS").split(",")
}{code}
randomizedInplace is not called conf.getenv("SPARK_LOCAL_DIRS").split(",") .  This is what used in case of K8s and the shuffle locations are not randomized. 

IMHO , this should be randomized , so that all the directories have equal changes of pushing the data as was done on yarn side
Bug: HeapByteBuffer#get(int) is a hotspot path when using BytePackerForLong#unpack8Values with ByteBuffer input API
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jul/22 12:03
Affects Version/s: 3.4.0

HeapByteBuffer#get(int) is a hotspot path when scan Int or Long data from Parquet DataPage V2 as follows:

 

!image-2022-07-26-20-05-38-582.png|width=716,height=363!
Bug: V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jul/22 04:19
Affects Version/s: 3.4.0

When building V2 In Predicate in V2ExpressionBuilder, InSet.dataType (which is BooleanType) is used to build the LiteralValue, InSet.child.dataType should be used instead.
Bug: Extract `constructUnsupportedMergeException` method to  deduplicate code in `AccumulatorV2`
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Jul/22 12:48
Affects Version/s: 3.4.0

{code:java}
throw new UnsupportedOperationException(
  s"Cannot merge ${this.getClass.getName} with ${other.getClass.getName}") {code}
Bug: Dataset.as(StructType) fills missing new columns with null value
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jul/22 09:19
Affects Version/s: 3.4.0

As a followup of [https://github.com/apache/spark/pull/37011#discussion_r917700960] , it would be great to fill missing new columns with null values, instead of failing out loud. Note it would only work for nullable columns.
Bug: Put Literal values on the right side of the data source filter after translating Catalyst Expression to data source filter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jul/22 00:24
Affects Version/s: 3.4.0

After translating Expression to data source filter, we want the Literal value to be on the right side of the filter.
For example: 1 > a
After translate to Predicate, we want to have a < 1
Bug: Simplify conflict binary comparison
Status: In Progress | Priority: Minor | Resolution: 
Created: 22/Jul/22 11:23
Affects Version/s: 3.3.0

Currently, We do not handle conflicting binary comparison expressions, Let's say have a query, it filter `a < 0 and a > 1`, this condition should be a `FalseLiteral`, we can fold ahead of time to avoid unnecessary calculations.
Bug: Add the Support for Custom Authentication for Thrift Server using Http Servlet Request input Param
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Jul/22 05:53
Affects Version/s: 3.3.0

As per the CustomAuthentication(hive.server2.transport.mode=HTTP) in Thrift Server only String username and String password is supported in the authenticate method of PasswdAuthenticationProvider([https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/PasswdAuthenticationProvider.java#L37]). There is no support for the getting Header param of Request, input param of request, request URI, getMethod if these param is required by the Custom Authentication Service(third party sdk, In House Authentication Service).

In order to support CustomAuthentication for Custom Authentication Service(third party sdk, In House Authentication Service) there is need to add the code change in Spark Code , than only user can add the Custom Authentication.

For Example
{code:java}
public class SampleAuthenticator implements PasswdAuthenticationProvider {
  static Logger logger = Logger.getLogger(SampleAuthenticator.class.getName());

  public SampleAuthenticator() {
    logger.log(Level.INFO, "initialize SampleAuthenticator");
  }
  @Override
  public void Authenticate(String user, String  password)
      throws AuthenticationException {
  }

 @Override
  public void AuthenticateRequest(HttpServletRequest request)
      throws AuthenticationException {
        Map<String, List<String>> requestHeaders = new HashMap(); 
        // derrive the request headerValue from the HttpServletRequest request
        logger.log(Level.INFO, "SampleAuthenticator AuthenticateRequest");
        dummyAuthenticate(request.getMethod, requestHeaders, request.);
      }
  public void dummyAuthenticate(String httpMethod, Map<String, List<String>> requestHeaders, Uri uri) {
     // Code added for custom Authentication
  }    
}
{code}
So there is need to support void AuthenticateRequest(HttpServletRequest request) in the Spark Code base, to integrate the custom Authentication in apache spark
Bug: Incorrect Spark SQL UI Page
Status: Open | Priority: Minor | Resolution: 
Created: 19/Jul/22 10:02
Affects Version/s: 3.0.1

We have a long running spark application, and we notice that after running a couple of days, the SQL UI looks weird, there are some running queries with wrong submit time and duration, but the queries already finished long ago, also the running job IDs and Succeeded job IDs are not related to this query, and it keeps growing.
!image-2022-07-19-18-03-39-886.png!
Bug: add ignore for the recently added and failing mypy error 'type-var' #37210
Status: In Progress | Priority: Minor | Resolution: 
Created: 18/Jul/22 22:40
Affects Version/s: 3.2.2

A new error introduced to mypy is found in this repository. This PR adds comment to ignore that error.
{quote}spark ([https://github.com/apache/spark])
 * python/pyspark/ml/wrapper.py:267: error: A function returning TypeVar should receive at least one argument containing the same Typevar [type-var]{quote}
Reference:

[python/mypy#13166|https://github.com/python/mypy/pull/13166]
Bug: Use commons-text LevenshteinDistance instead of commons-langs3 `StringUtils.getLevenshteinDistance`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jul/22 05:27
Affects Version/s: 3.4.0

[WARNING] /basedi/spark-source/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:79: [deprecation @ org.apache.spark.sql.catalyst.util.StringUtils.orderStringsBySimilarity.$anonfun | origin=org.apache.commons.lang3.StringUtils.getLevenshteinDistance | version=] method getLevenshteinDistance in class StringUtils is deprecated
Bug: Support url encode/decode as built-in function
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jul/22 07:46
Affects Version/s: 3.4.0

Currently, Spark don't support url encode/decode as built-in functions, the user might use reflect instead, It's a bit of a hassle, And often these functions are useful.

This pr aims to add url encode/decode as built-in function support.
Bug: For the `arrayOfAnyAsSeq` scenario in `GenericArrayDataBenchmark`, using Scala 2.13 is slower than Scala 2.12
Status: In Progress | Priority: Minor | Resolution: 
Created: 13/Jul/22 14:19
Affects Version/s: 3.4.0

Run `GenericArrayDataBenchmark` with Scala 2.13 and 2.12, for the `arrayOfAnyAsSeq` scenario in `GenericArrayDataBenchmark`, using Scala 2.13 is slower than Scala 2.12:

*Scala 2.12*
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_322-b06 on Linux 5.13.0-1021-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
constructor:                              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
arrayOfAnyAsSeq                                      25             29           2        395.1           2.5       0.1X{code}
*Scala 2.13*
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
constructor:                              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
arrayOfAnyAsSeq                                     241            243           1         41.4          24.1       0.0X {code}
the test code as follows:
{code:java}
benchmark.addCase("arrayOfAnyAsSeq") { _ =>
  val arr: Seq[Any] = new Array[Any](arraySize)
  var n = 0
  while (n < valuesPerIteration) {
    new GenericArrayData(arr)
    n += 1
  }
} {code}
the constructor of GenericArrayData as follows:
{code:java}
def this(seq: scala.collection.Seq[Any]) = this(seq.toArray) {code}
 

The performance difference is due to the following reasons：

*When using Scala 2.12:*

The class type of `arr` is `s.c.mutable.WrappedArrayWrappedArray$ofRef`, `toArray` return `array.asInstanceOf[Array[U]]`, there is no memory copy.

*When using Scala 2.13:*

The class type of `arr` is `s.c.immutable.ArraySeq$ofRef`, `toArray` will call `IterableOnceOps#toArray`, the corresponding implementation uses memory copy.
Bug: Use setBufferedIo instead of withBufferedIo to cleanup log4j2 deprecated api usage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jul/22 02:20
Affects Version/s: 3.4.0

 fix the following log4j2 compilation warning
{code:java}
[WARNING] /basedir/spark-source/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala:69: [deprecation @ org.apache.spark.util.logging.DriverLogger.addLogAppender.log4jFileAppender | origin=org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.Builder.withBufferedIo | version=] method withBufferedIo in class Builder is deprecated
 {code}
Bug: 3.2.2 Window function doc references TIMESTAMPNTZ
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jul/22 22:09
Affects Version/s: 3.2.2

The Window function doc for 3.2.2 (RC1) references {{TIMESTAMPNTZ}} [here|https://github.com/apache/spark/blob/ea5af851e78f569365b240e3a81b8108d8b2f650/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3520], [here|https://github.com/apache/spark/blob/ea5af851e78f569365b240e3a81b8108d8b2f650/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3576], and [here|https://github.com/apache/spark/blob/ea5af851e78f569365b240e3a81b8108d8b2f650/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3520].

As far as I can tell, {{TIMESTAMPNTZ}} data type is not supported in 3.2.2.
Bug: Support Avro schema evolution
Status: In Progress | Priority: Minor | Resolution: 
Created: 13/Jul/22 21:27
Affects Version/s: 3.3.0

currently the avro source in connector/avro does not yet support schema evolution.
from source code of AvroUtils:
{code:java}
// Schema evolution is not supported yet. Here we only pick first random readable sample file to
// figure out the schema of the whole dataset.
 {code}
i added schema evolution for our inhouse spark version. if there is interest in this i could contribute it.