av9ash/GitBugs huggingface dataset:
Bug: Util methods requiring certain versions of Pandas & PyArrow don't pass through the raised ImportError
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/21 16:12
Affects Version/s: 3.1.1

When checking that the we can import either {{pandas}} or {{pyarrow}}, we except any {{ImportError}} and raise an error declaring the minimum version of the respective package that's required to be in the Python environment.

We don't however, pass the {{ImportError}} that might have been thrown by the package itself. Take {{pandas}} as an example, when we call {{import pandas}}, pandas itself might be in the environment, but can throw an {{ImportError}} [https://github.com/pandas-dev/pandas/blob/0.24.x/pandas/compat/__init__.py#L438] if another package it requires isn't there. This error wouldn't be passed through and we'd end up getting a misleading error message that states that {{pandas}} isn't in the environment, while in fact it is but something else makes us unable to import it.

I believe this can be improved by chaining the exceptions and am happy to provide said contribution.
Bug: Cached Table (parquet) with old Configs Used
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/21 20:00
Affects Version/s: 2.4.4

When a dataframe is cached, the logical plan can contain copies of the spark session meaning the SQLConfs are stored. Then if a different dataframe can replace parts of it's logical plan with a cached logical plan, the cached SQLConfs will be used for the evaluation of the cached logical plan. This is because HadoopFsRelation ignores sparkSession for equality checks (introduced in https://issues.apache.org/jira/browse/SPARK-17358).
{code:java}
test("cache uses old SQLConf") {
  import testImplicits._
  withTempDir { dir =>
    val tableDir = dir.getAbsoluteFile + "/table"
    val df = Seq("a").toDF("key")
    df.write.parquet(tableDir)
    SQLConf.get.setConfString(SQLConf.FILE_COMPRESSION_FACTOR.key, "1")
    val compression1Stats = spark.read.parquet(tableDir).select("key").
      queryExecution.optimizedPlan.collect {
      case l: LogicalRelation => l
      case m: InMemoryRelation => m
    }.map(_.computeStats())

    SQLConf.get.setConfString(SQLConf.FILE_COMPRESSION_FACTOR.key, "10")
    val df2 = spark.read.parquet(tableDir).select("key")
    df2.cache()
    val compression10Stats = df2.queryExecution.optimizedPlan.collect {
      case l: LogicalRelation => l
      case m: InMemoryRelation => m
    }.map(_.computeStats())

    SQLConf.get.setConfString(SQLConf.FILE_COMPRESSION_FACTOR.key, "1")
    val compression1StatsWithCache = spark.read.parquet(tableDir).select("key").
      queryExecution.optimizedPlan.collect {
      case l: LogicalRelation => l
      case m: InMemoryRelation => m
    }.map(_.computeStats())

    // I expect these stats to be the same because file compression factor is the same
    assert(compression1Stats == compression1StatsWithCache)
    // Instead, we can see the file compression factor is being cached and used along with
    // the logical plan
    assert(compression10Stats == compression1StatsWithCache)
  }
}{code}
Bug: SparkR throws node stack overflow
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/21 16:51
Affects Version/s: 3.0.1

SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3.0.1.

Same piece of code works on R-3.3.3 with Spark 2.2.1 (& SparkR 2.4.5)
{code:java}
source('sample.R')
myclsr = myclosure_func()
myclsr$get_some_date('2021-01-01')

## spark.lapply throws node stack overflow
result = spark.lapply(c('2021-01-01', '2021-01-02'), function (rdate) {
    source('sample.R')
    another_closure = myclosure_func()
    return(another_closure$get_some_date(rdate))
})

{code}
Sample.R
{code:java}
## util function, which calls itself
getPreviousBusinessDate <- function(asofdate) {
    asdt <- asofdate;
    asdt <- as.Date(asofdate)-1;

    wd <- format(as.Date(asdt),"%A")
    if(wd == "Saturday" | wd == "Sunday") {
        return (getPreviousBusinessDate(asdt));
    }

    return (asdt);
}

## closure which calls util function
myclosure_func = function() {

    myclosure = list()

    get_some_date = function (random_date) {
        return (getPreviousBusinessDate(random_date))
    }
    myclosure$get_some_date = get_some_date

    return(myclosure)
}

{code}
This seems to have caused by sourcing sample.R twice. Once before invoking Spark session and another within Spark session.
Bug: Respect the default input buffer size in Univocity
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/21 04:42
Affects Version/s: 3.0.2

Currenty Univocity 2.9.1 faces a bug such as https://github.com/uniVocity/univocity-parsers/issues/449.

While this is a bug, another factor is that we don't respect Univocity's default value which makes Spark exposed to non-test coverage in Univocity.

We should resect Univocity's default input buffer value.
Bug: Upgrade Jetty to 9.4.37 to fix CVE-2020-27223
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/21 21:31
Affects Version/s: 3.1.1

Another day, another Jetty CVE :)  Our internal build tools are complaining about Spark's dependency on Jetty 9.4.36 and I found it is because there is another Jetty CVE on the version we recently upgraded to in SPARK-34449. Time for another upgrade to 9.4.37.

 

Find more at:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-27223
https://www.sourceclear.com/vulnerability-database/security/denial-of-servicedos/java/sid-29523
Bug: Port/integrate Koalas documentation into PySpark
Status: Resolved | Priority: Major | Resolution: Done
Created: 29/Mar/21 02:40
Affects Version/s: 3.2.0

This JIRA aims to port [Koalas documentation|https://koalas.readthedocs.io/en/latest/index.html] appropriately to [PySpark documentation|https://spark.apache.org/docs/latest/api/python/index.html].
Bug: PySpark loses metadata in DataFrame fields when selecting nested columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/21 17:42
Affects Version/s: 3.0.1

For a DataFrame schema with nested StructTypes, where metadata is set for fields in the schema, that metadata is lost when a DataFrame selects nested fields.  For example, suppose
{code:java}
df.schema.fields[0].dataType.fields[0].metadata
{code}
returns a non-empty dictionary, then
{code:java}
df.select('Field0.SubField0').schema.fields[0].metadata{code}
returns an empty dictionary, where "Field0" is the name of the first field in the DataFrame and "SubField0" is the name of the first nested field under "Field0".
Bug: Support nested column in Spark Parquet vectorized readers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/21 08:57
Affects Version/s: 3.2.0

The task is to support nested column type in Spark Parquet vectorized reader. Currently Parquet vectorized reader does not support nested column type (struct, array and map). We implemented nested column vectorized reader for FB-ORC in our internal fork of Spark. We are seeing performance improvement compared to non-vectorized reader when reading nested columns. In addition, this can also help improve the non-nested column performance when reading non-nested and nested columns together in one query.

 

Parquet: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L173]
Bug: Mark indeterminate RDD in Web UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/21 08:38
Affects Version/s: 3.2.0

It is somehow hard to track which part is indeterminate in a graph of RDDs. In some cases we may need to track indeterminate RDDs. For example, indeterminate map stage fails and Spark is unable to fallback some parent stages. If Web UI can show up indeterminate RDD like cached RDD, it could be useful to track it.
Bug: Web UI does not correctly get appId
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/21 03:19
Affects Version/s: 2.4.7

Web UI does not correctly get appId when it has `proxy` or `history` in URL.

In my case, it happens on `https://jupyterhub.hosted.us/my-name/proxy/4040/executors/`.
Web developer console says: `jquery-3.4.1.min.js:2 GET https://jupyterhub.hosted.us/user/my-name/proxy/4040/api/v1/applications/4040/allexecutors 404`, and it shows blank pages to me.

There is relative issue in jupyterhub https://github.com/jupyterhub/jupyter-server-proxy/issues/57

https://github.com/apache/spark/blob/2526fdea481b1777b2c4a2242254b72b5c49d820/core/src/main/resources/org/apache/spark/ui/static/utils.js#L93-L105

It should not get from document.baseURI.
A request will occur, but performance impacts will be a bit.
Bug: Adaptive fetch of shuffle mergers for Push based shuffle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 18:35
Affects Version/s: 3.1.0

Currently the shuffle mergers are set during the creation of ShuffleMapStage. In the initial set of stages, there won't be enough executors added which can cause not enough shuffle mergers to be set during the creation of the shuffle map stage. This task is to handle the issue of low merge ratio for initial stages.
Bug: Push down filter through window after TransposeWindow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/21 08:34
Affects Version/s: 3.2.0


{code:scala}
      spark.range(10).selectExpr("id AS a", "id AS b", "id AS c", "id AS d").createTempView("t1")
      val df = spark.sql(
        """
          |SELECT *
          |  FROM (
          |    SELECT b,
          |      sum(d) OVER (PARTITION BY a, b),
          |      rank() OVER (PARTITION BY a ORDER BY c)
          |    FROM t1
          |  ) v1
          |WHERE b = 2
          |""".stripMargin)
{code}

Current optimized plan:
{noformat}
== Optimized Logical Plan ==
Project [b#221L, sum(d) OVER (PARTITION BY a, b ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)#231L, RANK() OVER (PARTITION BY a ORDER BY c ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#232]
+- Filter (b#221L = 2)
   +- Window [rank(c#222L) windowspecdefinition(a#220L, c#222L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY a ORDER BY c ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#232], [a#220L], [c#222L ASC NULLS FIRST]
      +- Project [b#221L, a#220L, c#222L, sum(d) OVER (PARTITION BY a, b ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)#231L]
         +- Window [sum(d#223L) windowspecdefinition(a#220L, b#221L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS sum(d) OVER (PARTITION BY a, b ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)#231L], [a#220L, b#221L]
            +- Project [id#218L AS b#221L, id#218L AS d#223L, id#218L AS a#220L, id#218L AS c#222L]
               +- Range (0, 10, step=1, splits=Some(2))
{noformat}


Expected optimized plan:
{noformat}
== Optimized Logical Plan ==
Project [b#221L, sum(d) OVER (PARTITION BY a, b ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)#231L, RANK() OVER (PARTITION BY a ORDER BY c ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#232]
+- Window [sum(d#223L) windowspecdefinition(a#220L, b#221L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS sum(d) OVER (PARTITION BY a, b ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)#231L], [a#220L, b#221L]
   +- Project [b#221L, d#223L, a#220L, RANK() OVER (PARTITION BY a ORDER BY c ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#232]
      +- Filter (b#221L = 2)
         +- Window [rank(c#222L) windowspecdefinition(a#220L, c#222L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY a ORDER BY c ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#232], [a#220L], [c#222L ASC NULLS FIRST]
            +- Project [id#218L AS b#221L, id#218L AS d#223L, id#218L AS a#220L, id#218L AS c#222L]
               +- Range (0, 10, step=1, splits=Some(2))
{noformat}
Bug: Reading Parquet Files in Spark Extremely Slow for Large Number of Files?
Status: Reopened | Priority: Major | Resolution: 
Created: 06/Mar/21 06:54
Affects Version/s: 2.3.0

Hello Team

I am new to Spark and this question may be a possible duplicate of the issue highlighted here: https://issues.apache.org/jira/browse/SPARK-9347 

We have a large dataset partitioned by calendar date, and within each date partition, we are storing the data as *parquet* files in 128 parts.

We are trying to run aggregation on this dataset for 366 dates at a time with Spark SQL on spark version 2.3.0, hence our Spark job is reading 366*128=46848 partitions, all of which are parquet files. There is currently no *_metadata* or *_common_metadata* file(s) available for this dataset.

The problem we are facing is that when we try to run *spark.read.parquet* on the above 46848 partitions, our data reads are extremely slow. It takes a long time to run even a simple map task (no shuffling) without any aggregation or group by.

I read through the above issue and I think I perhaps generally understand the ideas around *_common_metadata* file. But the above issue was raised for Spark 1.3.1 and for Spark 2.3.0, I have not found any documentation related to this metadata file so far.

I would like to clarify:
 # What's the latest, best practice for reading large number of parquet files efficiently?
 # Does this involve using any additional options with spark.read.parquet? How would that work?
 # Are there other possible reasons for slow data reads apart from reading metadata for every part? We are basically trying to migrate our existing spark pipeline from using csv files to parquet, but from my hands-on so far, it seems that parquet's read time is slower than csv? This seems contradictory to popular opinion that parquet performs better in terms of both computation and storage?
Bug: col(), $"<name>" and df("name") should handle quoted column names properly.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/21 15:17
Affects Version/s: 3.1.1

Quoted column names like `a``b.c` cannot be represented with col(), $"<name>" and df("") because they don't handle such column names properly.

For example, if we have a following DataFrame.
{code}
val df1 = spark.sql("SELECT 'col1' AS `a``b.c`")
{code}

For the DataFrame, this query is successfully executed.
{code}
scala> df1.selectExpr("`a``b.c`").show
+-----+
|a`b.c|
+-----+
| col1|
+-----+
{code}

But the following query will fail because df1("`a``b.c`") throws an exception.
{code}
scala> df1.select(df1("`a``b.c`")).show
org.apache.spark.sql.AnalysisException: syntax error in attribute name: `a``b.c`;
  at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute$.e$1(unresolved.scala:152)
  at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute$.parseAttributeName(unresolved.scala:162)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveQuoted(LogicalPlan.scala:121)
  at org.apache.spark.sql.Dataset.resolve(Dataset.scala:221)
  at org.apache.spark.sql.Dataset.col(Dataset.scala:1274)
  at org.apache.spark.sql.Dataset.apply(Dataset.scala:1241)
  ... 49 elided
{code}
Bug: Setting CSV reader option "multiLine" to "true" causes URISyntaxException when colon is in file path
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/21 19:40
Affects Version/s: 3.0.0

Setting the CSV reader's "multiLine" option to "True" throws the following exception when a ':' character is in the file path.

 
{code:java}
java.net.URISyntaxException: Relative path in absolute URI: test:dir
{code}
I've tested this in both Spark 3.0.0 and Spark 3.1.1 and I get the same error whether I use Scala, Python, or SQL.

The following code works fine:

 
{code:java}
csvFile = "/FileStore/myDir/test:dir/pageviews_by_second.tsv" 
tempDF = (spark.read.option("sep", "\t").csv(csvFile)
{code}
While the following code fails:

 
{code:java}
csvFile = "/FileStore/myDir/test:dir/pageviews_by_second.tsv"
tempDF = (spark.read.option("sep", "\t").option("multiLine", "True").csv(csvFile)
{code}
Full Stack Trace from Python:

 
{code:java}
--------------------------------------------------------------------------- 
IllegalArgumentException Traceback (most recent call last) <command-8965899> in <module> 
3 csvFile = "/FileStore/myDir/test:dir/pageviews_by_second.tsv" 
4 
----> 5  tempDF = (spark.read.option("sep", "\t").option("multiLine", "True") 

/databricks/spark/python/pyspark/sql/readwriter.py in csv(self, path, schema, sep, encoding, quote, escape, comment, header, inferSchema, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, nullValue, nanValue, positiveInf, negativeInf, dateFormat, timestampFormat, maxColumns, maxCharsPerColumn, maxMalformedLogPerPartition, mode, columnNameOfCorruptRecord, multiLine, charToEscapeQuoteEscaping, samplingRatio, enforceSchema, emptyValue, locale, lineSep, pathGlobFilter, recursiveFileLookup, modifiedBefore, modifiedAfter, unescapedQuoteHandling) 

735 path = [path] 
736 if type(path) == list: 
--> 737 return self._df(self._jreader.csv(self._spark._sc._jvm.PythonUtils.toSeq(path))) 
738 elif isinstance(path, RDD): 
739 def func(iterator): 

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args) 
1302 
1303 answer = self.gateway_client.send_command(command) 
-> 1304 return_value = get_return_value( 
1305 answer, self.gateway_client, self.target_id, self.name) 
1306 

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw) 
114 # Hide where the exception came from that shows a non-Pythonic 
115 # JVM exception message. 
--> 116 raise converted from None 
117 else: 
118 raise IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: test:dir
{code}
Bug: Removes outer join if it only has distinct on streamed side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/21 12:57
Affects Version/s: 3.2.0

For example:
{code:scala}
spark.range(200L).selectExpr("id AS a").createTempView("t1")
spark.range(300L).selectExpr("id AS b").createTempView("t2")
spark.sql("SELECT DISTINCT a FROM t1 LEFT JOIN t2 ON a = b").explain(true)
{code}

Current optimized plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [a#2L], [a#2L]
+- Project [a#2L]
   +- Join LeftOuter, (a#2L = b#6L)
      :- Project [id#0L AS a#2L]
      :  +- Range (0, 200, step=1, splits=Some(2))
      +- Project [id#4L AS b#6L]
         +- Range (0, 300, step=1, splits=Some(2))
{noformat}

Expected optimized plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [a#2L], [a#2L]
+- Project [id#0L AS a#2L]
   +- Range (0, 200, step=1, splits=Some(2))
{noformat}
Bug: YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 22:27
Affects Version/s: 3.1.1

In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are using different versions of Spark. This can be helpful, for example, when running a YARN cluster with a mixed workload of applications running multiple Spark versions, since a given version of the shuffle service is not always compatible with other versions of Spark. (See SPARK-27780 for more detail on this)

YARN versions since 2.9.0 support the ability to run shuffle services within an isolated classloader (see YARN-4577), meaning multiple Spark versions can coexist within a single NodeManager.

To support this from the Spark side, we need to make two enhancements:

* Make the name of the shuffle service configurable. Currently it is hard-coded to be {{spark_shuffle}} on both the client and server side. The server-side name is not actually used anywhere, as it is the value within the {{yarn.nodemanager.aux-services}} which is considered by the NodeManager to be definitive name. However, if you change this in the configs, the hard-coded name within the client will no longer match. So, this needs to be configurable.
* Add a way to separately configure the two shuffle service instances. Since the configurations such as the port number are taken from the NodeManager config, they will both try to use the same port, which obviously won't work. So, we need to provide a way to selectively configure the two shuffle service instances. I will go into details on my proposal for how to achieve this within the PR.
Bug: Support native session window
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 03:49
Affects Version/s: 3.2.0

This issue tracks effort on supporting native session window, on both batch query and streaming query.

This issue is the finalization of SPARK-10816 leveraging SPARK-34888, SPARK-34889, SPARK-35861, SPARK-34891, SPARK-34892.
Bug: Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 03:37
Affects Version/s: 3.2.0

This issue tracks effort on introducing MergingSortWithSessionWindowStateIterator which will ensure the sort order between input rows and rows in state via efficient way. MergingSortWithSessionWindowStateIterator will require precondition that input rows are sorted, and assume that the number of rows in state per group key will be small. As the name represents, the iterator will do merge sort between twos and provide elements one by one.

The precondition will be guaranteed via physical node, and the assume is most likely true unless watermark gap is specified like hours and there're quite lots of old but not late input rows.
Bug: Unify overflow exception error message of integral types
Status: Resolved | Priority: Major | Resolution: Later
Created: 15/Mar/21 14:58
Affects Version/s: 3.2.0

Current, the overflow exception error messages of integral types are different.
For Byte/Short type, the message is "... caused overflow"
For Int/Long, the message is "int/long overflow" since Spark is calling the "*Exact"(e.g. addExact, negateExact) methods from java.lang.Math.

We should unify the error message by changing the message of Byte/Short as "tinyint/smallint overflow"
Bug: Introduce MergingSessionsIterator merging elements directly which belong to the same session
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 03:16
Affects Version/s: 3.2.0

This issue tracks effort on introducing MergingSessionsIterator, which enables to merge elements belong to the same session directly. This would be quite performant compared to UpdatingSessionIterator. Note that MergingSessionsIterator can only apply to the cases aggregation can be applied altogether, so there're still rooms for UpdatingSessionIterator to be used.

This issue also introduces MergingSessionsExec which is the physical node on leveraging MergingSessionsIterator to sort the input rows and aggregate rows according to the session windows.