av9ash/GitBugs huggingface dataset:
Bug: Add QUALIFY Clause
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Apr/20 01:33
Affects Version/s: 3.1.0

In a SELECT statement, the QUALIFY clause filters the results of window functions.

QUALIFY does with window functions what HAVING does with aggregate functions and GROUP BY clauses.

In the execution order of a query, QUALIFY is therefore evaluated after window functions are computed.

Examples:
https://docs.snowflake.com/en/sql-reference/constructs/qualify.html#examples

More details:
https://docs.snowflake.com/en/sql-reference/constructs/qualify.html
https://docs.teradata.com/reader/2_MC9vCtAJRlKle2Rpb0mA/19NnI91neorAi7LX6SJXBw
Bug: DataSourceV2: Catalog API for view metadata
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Apr/20 00:11
Affects Version/s: 3.2.0

SPARK-24252 added a catalog plugin system and `TableCatalog` API that provided table metadata to Spark. This JIRA adds `ViewCatalog` API for view metadata.

Details in [SPIP document|https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing].
Bug: Spark Structure streaming read data twice per every micro-batch.
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 12/Apr/20 19:41
Affects Version/s: 2.4.3

I have a very strange issue with spark structure streaming. Spark structure streaming creates two spark jobs for every micro-batch. As a result, read data from Kafka twice. Here is a simple code snippet.

 
{code:java}
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.streaming.Trigger

object CheckHowSparkReadFromKafka {
  def main(args: Array[String]): Unit = {
    val session = SparkSession.builder()
      .config(new SparkConf()
        .setAppName(s"simple read from kafka with repartition")
        .setMaster("local[*]")
        .set("spark.driver.host", "localhost"))
      .getOrCreate()
    val testPath = "/tmp/spark-test"
    FileSystem.get(session.sparkContext.hadoopConfiguration).delete(new Path(testPath), true)
    import session.implicits._
    val stream = session
      .readStream
      .format("kafka")
      .option("kafka.bootstrap.servers",        "kafka-20002-prod:9092")
      .option("subscribe", "topic")
      .option("maxOffsetsPerTrigger", 1000)
      .option("failOnDataLoss", false)
      .option("startingOffsets", "latest")
      .load()
      .repartitionByRange( $"offset")
      .writeStream
      .option("path", testPath + "/data")
      .option("checkpointLocation", testPath + "/checkpoint")
      .format("parquet")
      .trigger(Trigger.ProcessingTime(10.seconds))
      .start()
    stream.processAllAvailable()
{code}
This happens because if {{.repartitionByRange( $"offset")}}, if I remove this line, all good. But with spark create two jobs, one with 1 stage just read from Kafka, the second with 3 stage read -> shuffle -> write. So the result of the first job never used.

This has a significant impact on performance. Some of my Kafka topics have 1550 partitions, so read them twice is a big deal. In case I add cache, things going better, but this is not a way for me. In local mode, the first job in batch takes less than 0.1 ms, except batch with index 0. But in YARN cluster and Messos both jobs fully expected and on my topics take near 1.2 min.
Bug: Add additional fields in ExpressionDescription for more granular category in documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/20 04:40
Affects Version/s: 3.0.0

Add additional fields in ExpressionDescription so we can have more granular category in function documentation. For example, we want to group window function into finer categories such as ranking functions and analytic functions.

See Hyukjin's comment below for more details;
https://github.com/apache/spark/pull/28170#issuecomment-611917191
Bug: SparkR::orderBy with multiple columns descending produces error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/20 08:33
Affects Version/s: 2.4.5

When specifying two columns within an `orderBy()` function, to attempt to get an ordering by two columns in descending order, an error is returned.
{code:java}
library(magrittr) 
library(SparkR) 
cars <- cbind(model = rownames(mtcars), mtcars) 
carsDF <- createDataFrame(cars) 

carsDF %>% 
  mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")), desc(column("mpg")), desc(column("disp"))))) %>% 
  head() {code}
This returns an error:
{code:java}
 Error in ns[[i]] : subscript out of bounds{code}
This seems to be related to the more general issue that the following code, excluding the use of the `desc()` function also fails:
{code:java}
carsDF %>% 
  mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")), column("mpg"), column("disp")))) %>% 
  head(){code}
Bug: merge small files may need more common setting
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Apr/20 02:39
Affects Version/s: 2.4.5

Hi ,

SparkSql now allow us to use  repartition or coalesce to manually control the small files like the following

/*+ REPARTITION(1) */

/*+ COALESCE(1) */

But it can only be  tuning case by case ,we need to decide whether we need to use COALESCE or REPARTITION,can we try a more common way to reduce the decision by set the target size  as hive did

*Good points:*

1)we will also the new partitions number

2)with an ON-OFF parameter  provided , user can close it if needed

3)the parmeter can be set at cluster level instand of user side,it will be more easier to controll samll files.

4)greatly reduce the pressue of namenode

 

*Not good points:*

1)It will add a new task to calculate the target numbers by stastics the out files.

 

I don't know whether we have planned this in future.

 

Thanks
Bug: Difference in Storage Levels used in cache() and persist() for pyspark dataframes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/20 06:25
Affects Version/s: 2.4.3

There is a difference in default storage level *MEMORY_AND_DISK* in pyspark and scala.

*Scala*: StorageLevel(true, true, false, true)

*Pyspark:* StorageLevel(True, True, False, False)

 

*Problem Description:* 

Calling *df.cache()*  for pyspark dataframe directly invokes Scala method cache() and Storage Level used is StorageLevel(true, true, false, true).

But calling *df.persist()* for pyspark dataframe sets the newStorageLevel=StorageLevel(true, true, false, false) inside pyspark and then invokes Scala function persist(newStorageLevel).

*Possible Fix:*
Invoke pyspark function persist inside pyspark function cache instead of calling the scala function directly.

I can raise a PR for this fix if someone can confirm that this is a bug and the possible fix is the correct approach.
Bug: Support duplicated column names for toPandas with Arrow execution.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 01:48
Affects Version/s: 2.4.5

When we execute {{toPandas()}} with Arrow execution, it fails if the column names have duplicates.

{code:python}
>>> spark.sql("select 1 v, 1 v").toPandas()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/lib/python3.7/site-packages/pyspark/sql/dataframe.py", line 2132, in toPandas
    pdf = table.to_pandas()
  File "pyarrow/array.pxi", line 441, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/Users/ueshin/workspace/databricks-koalas/miniconda/envs/databricks-koalas_3.7/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 653, in table_to_blockmanager
    columns = _deserialize_column_index(table, all_columns, column_indexes)
  File "/path/to/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 704, in _deserialize_column_index
    columns = _flatten_single_level_multiindex(columns)
  File "/path/to/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 937, in _flatten_single_level_multiindex
    raise ValueError('Found non-unique column index')
ValueError: Found non-unique column index
{code}
Bug: Drop builtin function pages from SQL references
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/20 09:26
Affects Version/s: 3.0.0

This ticket intends to drop the built-in function pages from SQL references. We've already had a complete list of built-in functions in the API documents.

See related discussions for more details: 
https://github.com/apache/spark/pull/28170#issuecomment-611917191
Bug: Cluster tried to fetch blocks from blacklisted node of previous stage
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 07/Apr/20 09:16
Affects Version/s: 2.4.2

We enabled blacklist on our Spark application but recently we saw some wierd issue.

Our code is like
  {{rdd.repartitions(...).mapPartitions(...).groupByKey(...).map().collect()}}
In mapPartitions stage, some executors has exception "Can't connect to host xxxxxx: Connection rest by peer" and tasks on them were failed, so all executors under this node were blacklisted, as well as this node. These executors did complete some tasks before blacklisted.

Then in next stage (groupByKey(...).map()), application failed with block fetch failure: IndexOutOfBound Exception when some healthy executor want to fetch block from one of above blacklisted executors.

It happened multiple times.
Bug: Overwriting into dynamic partitions is appending data in pyspark
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 07/Apr/20 11:00
Affects Version/s: 2.4.3

While overwriting data in specific partitions using insertInto  , spark is appending data to specific partitions though the mode is overwrite. Below property is set in config to ensure that we don't overwrite all partitions. If the below property is set to static it is truncating and inserting the data.


 _spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')_


 _df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>)_

However if the above statement is changed to 

_df.write.mode('overwrite').format('parquet').insertInto(<db>.<tbl>,overwrite=True)_

 It starts behaving correct, I mean overwrites the data into specific partition. 

It seems   though the save mode has been mentioned earlier, precedence is given to the parameter set in insertInto method call.  +_*insertInto(<db>.<tbl>,overwrite=True)*_+  

It is happening in pyspark
Bug: add octet_length to functions
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 07/Apr/20 02:03
Affects Version/s: 2.4.5

Many functions are available in statically-typed Scala code via org.apache.spark.sql.functions, for example `length`: 

[https://github.com/apache/spark/blob/f5250a581b765ef8c9044438b5d27461f843b7fd/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2407]

I'm requesting `octet_length` to be included similarly.
Bug: Fix hive/SQLQuerySuite.derived from Hive query file: drop_database_removes_partition_dirs.q
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 06:48
Affects Version/s: 2.0.0

Test "derived from Hive query file: drop_database_removes_partition_dirs.q" can fail if we run it separately but can success running with the whole hive/SQLQuerySuite.
Bug: Error message from DataFrame creation is misleading.
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Apr/20 02:50
Affects Version/s: 2.4.5

*Description:*

DataFrame creation from pandas.DataFrame fails when one of the features contains only NaN values (which is ok).

However, error message mentions wrong feature as the culprit, which makes it hard to find the root cause.

*How to reproduce:*

 
{code:java}
import numpy as np
import pandas as pd
df2 = pd.DataFrame({'a': np.array([np.nan, np.nan], dtype=np.object_), 'b': [np.nan, 'aaa']})
display(spark.createDataFrame(df2[['b']]))   # Works fine
spark.createDataFrame(df2)            # Raises TypeError.
{code}
In the code above, column 'a' is bad. However, the `TypeError` raised in the last command mentions feature 'b' as the culprit:

TypeError: field b: Can not merge type <class 'pyspark.sql.types.DoubleType'> and <class 'pyspark.sql.types.StringType'>
Bug: Unable to insert data with partial dynamic partition with Spark & Hive 3
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 29/Apr/20 13:36
Affects Version/s: 2.3.2

When performing inserting data with dynamic partition, the operation fails if all partitions are not dynamic. For example:

 
{code:sql}
create external table test_insert(a int) partitioned by (part_a string, part_b string) stored as parquet location '<HDFS location>';
 
{code}
The query
{code:sql}
insert into table test_insert partition(part_a='a', part_b) values (3, 'b');
{code}
will fails with errors
{code:xml}
Cannot create partition spec from hdfs://xxxx/ ; missing keys [part_a]
Ignoring invalid DP directory <path to staging directory>
{code}
 

 

 

On the other hand, if I remove the static value of part_a to make the insert fully dynamic, the following query will succeed. Please note that below is not the issue . Issue is above one , where query throws invalid DP directory warning.
{code:sql}
insert into table test_insert partition(part_a, part_b) values (1,'a','b');
{code}
Bug: How can i run spark3.0.0.preview2 version spark on Spark 2.4 CDH cluster?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Apr/20 04:57
Affects Version/s: 3.0.0

I am trying to run spark-submit with spark 3.0.0 dependencies bundled FAT jar on Spark 2.4 CDH cluster but still Spark 2.4 version is running.

How can i by pass spark 2.4 with spark 3.0.0 while doing spark-submit?
Bug: Rename config name "spark.dynamicAllocation.shuffleTimeout" to "spark.dynamicAllocation.shuffleTracking.timeout"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/20 22:55
Affects Version/s: 3.0.0

The "spark.dynamicAllocation.shuffleTimeout" configuration only takes effect if "spark.dynamicAllocation.shuffleTracking.enabled" is true,  so we should re-namespace that configuration so that it's nested under the "shuffleTracking" one.
Bug: Improve cache block migration
Status: Open | Priority: Major | Resolution: 
Created: 24/Apr/20 17:05
Affects Version/s: 3.1.0

We should explore the following improvements to cache block migration:

1) Peer selection (right now may overbalance on certain peers)

2) Do we need to configure the number of blocks to be migrated at the same time

3) Are there any blocks we don't need to replicate (e.g. they are already stored on the desired number of executors even once we remove the executors slated for decommissioning).

4) Do we want to prioritize migrating blocks with no replicas

5) Log the attempt number for debugging 

6) Clarify the logic for determining the number of replicas

7) Consider using TestUtils.waitUntilExecutorsUp in tests rather than count to wait for the executors to come up. imho this is the least important.
Bug: Fix flaky test: o.a.s.scheduler.CoarseGrainedSchedulerBackendSuite.extra resources from executor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/20 14:58
Affects Version/s: 3.0.0

see [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120786/testReport/org.apache.spark.scheduler/CoarseGrainedSchedulerBackendSuite/extra_resources_from_executor/] for details.
{code:java}
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 325 times over 5.01070979 seconds. Last failure message: ArrayBuffer("1", "3") did not equal Array("0", "1", "3").
	at org.scalatest.concurrent.Eventually.tryTryAgain$1(Eventually.scala:432)
	at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:439)
	at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:391)
	at org.apache.spark.scheduler.CoarseGrainedSchedulerBackendSuite.eventually(CoarseGrainedSchedulerBackendSuite.scala:45)
	at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:337)
	at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:336)
	at org.apache.spark.scheduler.CoarseGrainedSchedulerBackendSuite.eventually(CoarseGrainedSchedulerBackendSuite.scala:45)
	at org.apache.spark.scheduler.CoarseGrainedSchedulerBackendSuite.$anonfun$new$12(CoarseGrainedSchedulerBackendSuite.scala:264)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
{code}
Bug: Spark documentation incorrectly claims 3.8 compatibility
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 03/Apr/20 16:47
Affects Version/s: 2.4.5

The Spark documentation ([https://spark.apache.org/docs/latest/]) has this text:
{quote}Spark runs on Java 8, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.5 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).
{quote}
Which suggests that Spark is compatible with Python 3.8. This is not true. For example in the latest ubuntu:18.04 docker image:

 
{code:python}
apt-get update
apt-get install python3.8 python3-pip
pip3 install pyspark
python3.8 -m pip install pyspark
python3.8 -c 'import pyspark'
{code}
Outputs:

{code:python}
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/usr/local/lib/python3.8/dist-packages/pyspark/context.py", line 31, in <module>
    from pyspark import accumulators
  File "/usr/local/lib/python3.8/dist-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/usr/local/lib/python3.8/dist-packages/pyspark/serializers.py", line 72, in <module>
    from pyspark import cloudpickle
  File "/usr/local/lib/python3.8/dist-packages/pyspark/cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/usr/local/lib/python3.8/dist-packages/pyspark/cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)
{code}

I propose the documentation is updated to say "Python 3.4 to 3.7". I also propose the `setup.py` file for pyspark include:
{code:python}
    python_requires=">=3.6,<3.8",
{code}
Bug: Backport SPARK-25595       Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled
Status: Resolved | Priority: Major | Resolution: Done
Created: 23/Apr/20 20:14
Affects Version/s: 2.4.6

Backport SPARK-25595       Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

cc [~Gengliang.Wang]& [~hyukjin.kwon] for comments
Bug: Support JobCleaned Status in SparkListener
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Apr/20 15:30
Affects Version/s: 3.1.0

In Spark, we need do some hook after job cleaned, such as cleaning hive external temporary paths. This has already discussed in SPARK-31346 and [GitHub Pull Request #28129.|https://github.com/apache/spark/pull/28129]
 The JobEnd Status is not suitable for this. As JobEnd is responsible for Job finished, once all result has generated, it should be finished. After finish, Scheduler will leave the still running tasks to be zombie tasks and delete abnormal tasks asynchronously.
 Thus, we add JobCleaned Status to enable user to do some hook after all tasks cleaned in Job. The JobCleaned Status can get from TaskSetManagers, which is related to a stage, and once all stages of the job has been cleaned, then the job is cleaned.
Bug: Numbers with thousands separator or locale specific decimal separator not parsed correctly
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Apr/20 17:50
Affects Version/s: 2.4.5

CSV files that contain numbers with thousands separator (or locale specific decimal separators) are not parsed correctly and are reported as {{null.}}

[https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html]

A user in France might expect "10,100" to be parsed as a float while a user in the US might want Spark to interpret it as an Integer value (10100). 

UnivocityParser is not locale aware and must use NumberFormatter to parse string values to Numbers. 

*US Locale*

{{scala>Source.fromFile("/Users/ranjit.iyer/work/data/us.csv").getLines.mkString("\n")}}
 {{res28: String =}}
 {{"Value"}}
 {{"10,000"}}
 {{"20,000"}}

{{scala> Locale.setDefault(Locale.US)}}

{{scala> val _schema = StructType(StructField("Value", IntegerType, true) :: Nil)}}

{{scala> val df = spark.read.format("csv").option("header", "true").schema(_schema).load("/Users/ranjit.iyer/work/data/us.csv")}}
 {{df: org.apache.spark.sql.DataFrame = [Value: int]}}

{{scala> df.show}}
 {{+-----+}}
 {{|Value|}}
 {{+-----+}}
 {{| null|}}
 {{| null|}}
 {{+-----+}}

*French Local* 

{{scala> Source.fromFile("/Users/ranjit.iyer/work/data/fr.csv").getLines.mkString("\n")}}
 {{res43: String =}}
 {{"Value"}}
 {{"10,123"}}
 {{"20,456"}}

{{scala> Locale.setDefault(Locale.FRANCE)}}

{{scala> val _schema = StructType(StructField("Value", FloatType, true) :: Nil)}}

{{scala> val df = spark.read.format("csv").option("header", "true").schema(_schema).load("/Users/ranjit.iyer/work/data/fr.csv")}}
 {{df: org.apache.spark.sql.DataFrame = [Value: float]}}

{{scala> df.show}}
 {{+-----+}}
 {{|Value|}}
 {{+-----+}}
 {{| null|}}
 {{| null|}}
 {{+-----+}}

The fix is to use a NumberFormatter and I have it working locally and will raise a PR for review.

{{NumberFormat.getInstance.parse(_).intValue()}} 

Thousands separator are quite commonly found on the internet. My workflow has been to copy to Excel, export to csv and analyze in Spark.

[https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population]
Bug: String/Int  to VarcharType  cast not supported in Spark
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Apr/20 04:33
Affects Version/s: 2.3.0

While casting a string column to varchar it does not do the casting at all and column remains string.

 

 I tried to achieve it through VarcharType as below but it errors :

for(i<-colList)

{ Df=Df.withColumn(i,Df(i).cast(VarcharType(1000))) }

*org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(`ColName AS VARCHAR(1000))' due to data type mismatch: cannot cast string to varchar(1000);;*

 

*Also, I tried through selectExpr cast option but no success.*

 

While trying to create an empty dataframe  with VarcharType also throws an error

*scala> var empty_df = spark.createDataFrame(sc.emptyRDD[Row], schema_rdd)*
*scala.MatchError: VarcharType(1000) (of class org.apache.spark.sql.types.VarcharType)*
 *at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$enco*

 

Please suggest a way to cast a string to varchar in spark. As a reading string column in SAS application has performance implications.