av9ash/GitBugs huggingface dataset:
Bug: Spark SQL window function with order by cause result incorrect
Status: Closed | Priority: Major | Resolution: Invalid
Created: 12/Oct/20 02:51
Affects Version/s: 3.0.1

Prepare the data

CREATE TABLE IF NOT EXISTS product_catalog (
name STRING,category STRING,location STRING,price DECIMAL(10,2));

INSERT OVERWRITE product_catalog VALUES 
('Nest Coffee', 'drink', 'Toronto', 15.5),
('Pepesi', 'drink', 'Toronto', 9.99),
('Hasimal', 'toy', 'Toronto', 5.9),
('Fire War', 'game', 'Toronto', 70.0),
('Final Fantasy', 'game', 'Montreal', 79.99),
('Lego Friends 15005', 'toy', 'Montreal', 12.99),
('Nesion Milk', 'drink', 'Montreal', 8.9);

1. Query without ORDER BY after PARTITION BY col,  the result is correct.

SELECT
category, price,
max(price) over(PARTITION BY category) as max_p,
min(price) over(PARTITION BY category) as min_p,
sum(price) over(PARTITION BY category) as sum_p,
avg(price) over(PARTITION BY category) as avg_p,
count(*) over(PARTITION BY category) as count_w
FROM
product_catalog;

|| category    || price      || max_p  || min_p    || sum_p    || avg_p           || count_w   ||
| drink           | 8.90      | 15.50    | 8.90 | 34.39 | 11.463333 | 3 |
| drink           | 9.99      | 15.50    | 8.90 | 34.39 | 11.463333 | 3 |
| drink           | 15.50    | 15.50    | 8.90 | 34.39 | 11.463333 | 3 |
| game          | 79.99    | 79.99    | 70.00 | 149.99 | 74.995000 | 2 |
| game          | 70.00    | 79.99 | 70.00 | 149.99 | 74.995000 | 2 |
| toy              | 12.99    | 12.99 | 5.90 | 18.89 | 9.445000 | 2 |
| toy              | 5.90      | 12.99 | 5.90 | 18.89 | 9.445000 | 2 |
7 rows selected (0.442 seconds)

2 Query with ORDER BY after PARTITION BY col,  the result is NOT correct. Min result is ok. Why other results are like that?

SELECT
category, price,
max(price) over(PARTITION BY category ORDER BY price) as max_p,
min(price) over(PARTITION BY category ORDER BY price) as min_p,
sum(price) over(PARTITION BY category ORDER BY price) as sum_p,
avg(price) over(PARTITION BY category ORDER BY price) as avg_p,
count(*)   over(PARTITION BY category ORDER BY price) as count_w
FROM
product_catalog;

|| category    || price      || max_p  || min_p    || sum_p    || avg_p           || count_w   ||
| drink     | 8.90   | 8.90   | 8.90   | 8.90    | 8.900000   | 1        |
| drink     | 9.99   | 9.99   | 8.90   | 18.89   | 9.445000   | 2        |
| drink     | 15.50  | 15.50  | 8.90   | 34.39   | 11.463333  | 3        |
| game      | 70.00  | 70.00  | 70.00  | 70.00   | 70.000000  | 1        |
| game      | 79.99  | 79.99  | 70.00  | 149.99  | 74.995000  | 2        |
| toy       | 5.90   | 5.90   | 5.90   | 5.90    | 5.900000   | 1        |
| toy       | 12.99  | 12.99  | 5.90   | 18.89   | 9.445000   | 2        |
7 rows selected (0.436 seconds)

Does it seem that we can only order by the columns after partition by clause?
I do not think there are such limitation in standard SQL.
Bug: array_contains doesn't pick element in the array but just at the end
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/20 09:54
Affects Version/s: 2.4.7

Hello,

given a column of array of strings. I used array_contains to find all the array with the string "Beer". array_contains is recognise just the string "Beer" if it is at the end of the array but not if it is in the middle of the array.

beer=rep_trans_count.withColumn("keep", array_contains(rep_trans_count.TRANSACTIONSLIST,"Beer")).filter(col("keep")==True)
Bug: ORC format does not propagate Hadoop config from DS options to underlying HDFS file system
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/20 09:17
Affects Version/s: 3.1.0

When running:
{code:java}
spark.read.format("orc").options(conf).load(path)
{code}
The underlying file system will not receive the `conf` options.
Bug: LibSVM format does not propagate Hadoop config from DS options to underlying HDFS file system
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/20 07:28
Affects Version/s: 3.1.0

When running:
{code:java}
spark.read.format("libsvm").options(conf).load(path)
{code}
The underlying file system will not receive the `conf` options.
Bug: Classify dialect exceptions in JDBC v2 Table Catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/20 08:24
Affects Version/s: 3.1.0

The current implementation of v2.jdbc.JDBCTableCatalog  don't care of exceptions defined by org.apache.spark.sql.connector.catalog.TableCatalog at all like
* NoSuchNamespaceException
* NoSuchTableException
* TableAlreadyExistsException
it either throw dialect exception or generic exception AnalysisException.

Since we split forming of dialect specific statements and their execution, we should extend dialect APIs and ask them how to convert their exceptions to TableCatalog  exceptions.
Bug: Bug with self joined tables after posexploded column
Status: Open | Priority: Major | Resolution: 
Created: 03/Oct/20 12:30
Affects Version/s: 3.0.1

 

Test below should pass success, but not. 

DataFrames have only one difference, validDF data column type is Array[Int], and in testDF type is Array[Vector].

This bug we catch when use MinHashLSH from Spark ML
{code:java}
// Bug with self joined tables with posexplode columns
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.scalatest.flatspec.AnyFlatSpec
import org.apache.spark.sql.functions.{udf, col, struct, posexplode}
import org.apache.spark.ml.linalg.Vectors

import scala.collection.mutable


case class TestRow(s: String, l: Array[Long], data: Array[Int])

class SparkBug extends AnyFlatSpec{

  "Test case" should "equal count" in {
    val conf = new SparkConf()
      .setMaster("local[8]")
      .set("spark.sql.codegen.wholeStage", "true")//its important, if set false, test will pass success

    val spark = SparkSession.builder().config(conf).getOrCreate()

    val vecUdf = udf(
      (s: mutable.WrappedArray[Long]) =>
        s.map(v => Vectors.dense(v))
    )

    val r = new scala.util.Random(1000)

    import spark.implicits._

    val rowCount = 200000//its important, if row count less than 200000(150000 for example), test will pass success

    val df = (0 to rowCount).map{
      case _ =>
        TestRow(
          "",
          (0 to 40).map(t => t.toLong).toArray,
          (0 to 4)
            .map(_ => r.nextInt(100000)).toArray
        )
    }.toDF()
      .coalesce(1)//Important, without coalesce(1) test pass success

    val testDF = df
      .withColumn("data", vecUdf(col("data")))
      .select(
        struct(col("*")).as("dd"),
        posexplode(col("data"))
      )

    val validDF = df.select(struct(col("*")).as("dd"), posexplode(col("data")))
    //Difference between validDF and testDF, that testDF data col have Array[Vector] type, and valid DF data col have Array[Long]

    val testCount = testDF
      .join(testDF, Seq("pos", "col"))
      .distinct()//Important, without distinct test pass success
      .count

    val validCount = validDF
      .join(validDF, Seq("pos", "col"))
      .distinct()
      .count

    //count should be equal, but not
    assert(testCount == validCount)
  }
}

{code}
Bug: Support interval type in PySpark
Status: Open | Priority: Major | Resolution: 
Created: 02/Oct/20 15:10
Affects Version/s: 3.1.0

At the moment PySpark doesn't support interval types at all. For example calling the following

{code:python}
spark.sql("SELECT current_date() - current_date()")                                                                                                                                  
{code}

or 

{code:python}
from pyspark.sql.functions import current_timestamp                                                                                                                                  
spark.range(1).select(current_timestamp() - current_timestamp())  
{code}

results in

{code}
Traceback (most recent call last):
...
ValueError: Could not parse datatype: interval
{code}

At minimum, we should support {{CalendarIntervalType}} in the schema, so queries using it don't fail on conversion.

Optionally we should have to provide conversions between internal and external types. That however, might be tricky, as {{CalendarInterval}} seems to have different semantics than {{datetime.timedetla}}.

Also see https://issues.apache.org/jira/browse/SPARK-21187?focusedCommentId=16474664&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16474664
Bug: Add support for CalendarIntervalType in PySpark schema
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Oct/20 15:13
Affects Version/s: 3.1.0

Queries resulting in interval types shouldn't fail in PySpark, i.e. the following

{code:python}
from pyspark.sql.functions import current_timestamp                                                                                                                                  

spark.range(1).select(current_timestamp() - current_timestamp())    
{code}

should result in

{code:python}
DataFrame[(current_timestamp() - current_timestamp()): interval]
{code}

instead of {{ValueError}}.
Bug: Add mapping between external Python types and CalendarIntervalType
Status: Open | Priority: Major | Resolution: 
Created: 02/Oct/20 15:17
Affects Version/s: 3.1.0

We might consider adding support for at least collecting the results of queries containing interval types

{code:python}
spark.range(1).select(current_timestamp() - current_timestamp()).first()
{code}

and possibly a conversion in the opposite direction.

An obvious candidate would be {{datetime.timedelta}} which is mapped to {{net.razorvine.pickle.objects.TimeDelta}}, but semantics of these and {{CalendarInteval}} differs.
Bug: Upgrade Apache Arrow to 2.0.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Oct/20 20:59
Affects Version/s: 3.0.1

Apache Arrow 2.0.0 has [just been released|https://cwiki.apache.org/confluence/display/ARROW/Arrow+2.0.0+Release]. This proposes to upgrade Spark's Arrow dependency to use 2.0.0, from the current 1.0.1.
Bug: Enables 'fail_if_no_tests' when reporting test results
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 19/Oct/20 05:26
Affects Version/s: 2.4.8

SPARK-33069 skipped because it raises a false alarm when there are no test cases. This is now fixed in https://github.com/ScaCap/action-surefire-report/issues/29
Bug: Lazy Load of SparkContext.addFiles
Status: Open | Priority: Minor | Resolution: 
Created: 12/Oct/20 14:18
Affects Version/s: 3.0.1

In my spark job, I may have various random files that may or may not be used by each task.

I would like to avoid copying all of the files to every executor until it is actually needed.

 

What I've tried:
 * SparkContext.addFiles w/ SparkFiles.get . In testing, all files were distributed to all clients.
 * Broadcast variables. Since I _don't_ know what files I'm going to need until I have started the task, I have to broadcast all the data at once, which leads to nodes getting data, and then caching it to disk. In short, the same issues as SparkContext.addFiles, but with the added benefit of having the ability to create a mapping of paths to files.

What I would like to see:
 * SparkContext.addFiles(file, Enum.LazyLoad) w/ SparkFiles.get(file, Enum.WaitForAvailability) or Future<?> future = SparkFiles.get(file)

 

 

Notes: https://issues.apache.org/jira/browse/SPARK-4290?focusedCommentId=14205346&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14205346 indicated that `SparkFiles.get` would be required to get the data on the local driver, but in my testing that did not appear to be the case.
Bug: `kvstore` and `unsafe` doc tasks fail
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/20 22:51
Affects Version/s: 3.1.0

`build/sbt publishLocal` task fails in two modules:
{code:java}
[error] stack trace is suppressed; run last kvstore / Compile / doc for the full output
[error] stack trace is suppressed; run last unsafe / Compile / doc for the full output
{code}
{code:java}
 sbt:spark-parent> kvstore/Compile/doc 
[info] Main Java API documentation to /home/gemelen/work/src/spark/common/kvstore/target/scala-2.12/api... 
[error] /home/gemelen/work/src/spark/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:167:1:  error: malformed HTML 
[error]    * An alias class for the type "ConcurrentHashMap<Comparable<Object>, Boolean>", which is used 
[error]                                                    ^ 
[error] /home/gemelen/work/src/spark/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:167:1:  error: unknown tag: Object 
[error]    * An alias class for the type "ConcurrentHashMap<Comparable<Object>, Boolean>", which is used 
[error]                                                               ^ 
[error] /home/gemelen/work/src/spark/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:167:1:  error: bad use of '>' 
[error]    * An alias class for the type "ConcurrentHashMap<Comparable<Object>, Boolean>", which is used
[error]                                                                                ^
{code}
{code:java}
 sbt:spark-parent> unsafe/Compile/doc 
[info] Main Java API documentation to /home/gemelen/work/src/spark/common/unsafe/target/scala-2.12/api... 
[error] /home/gemelen/work/src/spark/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:566:1:  error: malformed HTML 
[error]    * Trims whitespaces (<= ASCII 32) from both ends of this string. [error]
{code}
Bug: Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Oct/20 01:50
Affects Version/s: 3.1.0

In PostgresIntegrationSuite, evaluation of classOf[Seq.isAssignableFrom] fails due to ClassCastException. 
The reason is the same as what is resolved in SPARK-29292 but this happens at test time, not compile time.
Bug: Relax pyspark typing for sql str functions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Oct/20 13:17
Affects Version/s: 3.1.0

Several pyspark.sql.functions have overly strict typing, in that the type is more restrictive than the functionality. Specifically, the function allows specifying the column to operate on with a pyspark.sql.Column or a str. This is handled internally by [_to_java_column|https://github.com/apache/spark/blob/491a0fb08b0c57a99894a0b33c5814854db8de3d/python/pyspark/sql/column.py#L39-L50], which accepts a Column or string.

There is a pre-existing type for this: [ColumnOrName|https://github.com/apache/spark/blob/72ad9dcd5d484a8dd64c08889de85ef9de2a6077/python/pyspark/sql/_typing.pyi#L37]. ColumnOrName is used for many of the type definitions of pyspark.sql.functions arguments, but [not for|https://github.com/apache/spark/blob/72ad9dcd5d484a8dd64c08889de85ef9de2a6077/python/pyspark/sql/functions.pyi#L158-L162] locate, lpad, rpad, repeat, and split.
{code:java}
def locate(substr: str, str: Column, pos: int = ...) -> Column: ...
def lpad(col: Column, len: int, pad: str) -> Column: ...
def rpad(col: Column, len: int, pad: str) -> Column: ...
def repeat(col: Column, n: int) -> Column: ...
def split(str: Column, pattern: str, limit: int = ...) -> Column: ...{code}
ColumnOrName was not added by [~zero323] since Maciej "was concerned that this might be confusing or ambiguous", because these functions take a column to operate on as well strings which are used in the operation.

But I think ColumnOrName makes clear that this variable refers to the column and not a string parameter. Also there are other ways to address confusion, such as via the docstring or by changing the argument name for the column to col from str.

Finally, there's considerable convenience for users to not have to wrap column names in pyspark.sql.functions.col. Elsewhere the API seems pretty consistent in its willingness to accept columns by name and not Column object (at least when there is not alternative meaning for a string value, exception would be .when/.otherwise).

For example, we were calling pyspark.sql.functions.split with a string value for the str argument (specifying which column to split). And I noticed this when we enforced typing with pyspark-stubs in preparation for pyspark 3.1. For users that will enable typing in 3.1, this is a restriction in functionality.

Pre-existing PRs to address this:
 * [https://github.com/apache/spark/pull/30209]
 * [https://github.com/zero323/pyspark-stubs/pull/420]
Bug: Spark-shell does not display accented chara
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 04/Oct/20 10:34
Affects Version/s: 3.0.1

It seems to be a duplicate of *FLEX-18425*, which is duplicate of SDK-17398 that does not exist anymore. But the bug remains.

(1) I create a txt file "café.txt" that contains two lines : 
{quote}Café

CafÃ©
{quote}
(2) I type the following command :

*spark.read.csv("café.txt").show()*

It is displayed as following :

*spark.read.csv("caf.txt").show()*

But it works and it returns this : 
{quote}+-----+
  |   _c0|
 +-----+
  |  Caf|
  |Café|
 +-----+
{quote}
We notice a shift after "Caf" and "Café".

(3) The two following commands works. The written textfiles have the same content as "café.txt" 

*spark.read.csv("café.txt").write.format("text").save("café2")*

*sc.textFile("café.txt").saveAsTextFile("café3")*

 

Once again, the Spark-shell display this : 

*spark.read.csv("caf.txt").write.format("text").save("caf2")*

*sc.textFile("caf.txt").saveAsTextFile("caf3")*

 

(4)If I type 7 "é" an then 7 Backspace, by using the "é" key of my french keyboard, then the scala prompt disappears. I have a new prompt when I type Return.

 

The issue (4) as well as the shift in (2) seem to be related to the difference between counted characters and displayed characters.

 

(5) I notice that I haven't got this issue by launching Spark from Ubuntu, thanks to "Windows Subsystem for Linux" Version 2.
Bug: Fix SparkBuild.scala to recognize build settings for Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Oct/20 00:02
Affects Version/s: 3.0.1

In SparkBuild.scala, a variable 'scalaBinaryVersion' is hardcoded as '2.12'.
So, an environment variable 'SPARK_SCALA_VERSION' is also to be '2.12'.
This issue causes some test suites (e.g. SparkSubmitSuite) to be error.

{code}
===== TEST OUTPUT FOR o.a.s.deploy.SparkSubmitSuite: 'user classpath first in driver' =====

20/10/02 08:55:30.234 redirect stderr for command /home/kou/work/oss/spark-scala-2.13/bin/spark-submit INFO Utils: Error: Could not find or load m
ain class org.apache.spark.launcher.Main
20/10/02 08:55:30.235 redirect stderr for command /home/kou/work/oss/spark-scala-2.13/bin/spark-submit INFO Utils: /home/kou/work/oss/spark-scala-
2.13/bin/spark-class: line 96: CMD: bad array subscript
{code}

The reason of this error is that environment variables 'SPARK_JARS_DIR' and 'LAUNCH_CLASSPATH' is defined in bin/spark-class as follows.
{code}
SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars"
LAUNCH_CLASSPATH="${SPARK_HOME}/launcher/target/scala-$SPARK_SCALA_VERSION/classes:$LAUNCH_CLASSPATH"
{code}
Bug: Avoid using map instead of foreach to avoid potential side effect at callers of OrcUtils.readCatalystSchema
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Oct/20 05:15
Affects Version/s: 3.0.1

This is a kind of a followup of SPARK-32646. New JIRA was filed to control the fixed versions properly.

When you use {{map}}, it might be lazily evaluated and not executed. To avoid this,  we should better use {{foreach}}.

See also SPARK-16694
Bug: [SS] Fix job hang in cp mode when total cores less than total kafka partition 
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Oct/20 16:17
Affects Version/s: 3.0.1

In continuous processing mode, EpochCoordinator won't add offsets to query until got ReportPartitionOffset from all partitions. Normally, each kafka topic partition will be handled by one core, if total cores is smaller than total kafka topic partition counts, the job will hang without any error message. 

Failing the job with error message is better solution.
Bug: Update how to build doc for Scala 2.13 with sbt
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Oct/20 13:33
Affects Version/s: 3.0.1

In the current doc, how to build Spark for Scala 2.13 with sbt is described like:
{code}
./build/sbt -Dscala.version=2.13.0
{code}
But build fails with this command because scala-2.13 profile is not enabled and scala-parallel-collections is absent.
Bug: Introduce new API to FileCommitProtocol allow flexible file naming
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Oct/20 06:29
Affects Version/s: 3.1.0

This Jira is to propose a new version for `FileCommitProtocol` ([https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala] ), e.g. `FileCommitProtocolV2`.

The motivation is currently we have two requirements to change the API for FileCommitProtocol:

(1).Support write Hive ORC/Parquet bucketed table ([https://github.com/apache/spark/pull/30003] ): need to add new parameter `prefix` into method `newTaskTempFile` and `newTaskTempFileAbsPath`, to allow spark writes hive/presto-compatible bucketed files.

(2).Fix commit collision in dynamic partition overwrite mode ([https://github.com/apache/spark/pull/29000] ): need to add new method `getStagingDir` to allow customize dynamic partition staging directory to avoid commit collision.

 

The reason to propose FileCommitProtocolV2 instead of changing `FileCommitProtocol` directly, is that the API for FileCommitProtocolV2 is kind of public where we allow customized commit protocol subclass to use during run-time ([https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L146] ). So if we change the API (e.g. adding method, or changing existing method signature), it will break external subclass for the commit protocol. And we are aware of some of external subclasses for better support of object store, according to [~cloud_fan] .

 

One proposal for `FileCommitProtocolV2` can be:
{code:java}
abstract class FileCommitProtocolV2 {
  // `options` to replace `ext`, where we can put more string-string parameters
  def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String],
options: Map[String, String]): String
 
  // `options` to replace `ext`, where we can put more string-string parameters
  def newTaskTempFileAbsPath(
    taskContext: TaskAttemptContext, absoluteDir: String, options: Map[String, String]): String

  // other new methods, e.g. getStagingDir
  def getStagingDir(path: String, jobId: String): Path

  // rest of FileCommitProtocol methods
  ...
}
{code}
 

FileCommitProtocolV2.instantiate() logic will first try to find a subclass for `FileCommitProtocolV2`, if not will find a subclass for `FileCommitProtocol`, so the current version of `FileCommitProtocol` is still supported.