av9ash/GitBugs huggingface dataset:
Bug: Vector type column is not possible to create using spark SQL
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/20 17:34
Affects Version/s: 3.0.1

 

Created table with vector type column:
{code:java}
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.VectorUDT
import org.apache.spark.mllib.linalg.Vectors
case class Test(features: Vector) 
Seq(Test(Vectors.dense(Array(1d, 2d, 3d)))).toDF()
 .write
 .mode("overwrite")
 .saveAsTable("pborshchenko.test_vector_spark_0911_1")
{code}
 

Show the create table statement for this created table:
{code:java}
spark.sql("SHOW CREATE TABLE pborshchenko.test_vector_spark_0911_1"){code}
Got:
{code:java}
CREATE TABLE `pborshchenko`.`test_vector_spark_0911_1` (
 `features` STRUCT<`type`: TINYINT, `size`: INT, `indices`: ARRAY<INT>, `values`: ARRAY<DOUBLE>>)
USING parquet{code}
Create the same table with index 2 at the end:
{code:java}
spark.sql("CREATE TABLE `pborshchenko`.`test_vector_spark_0911_2` (\n`features` STRUCT<`type`: TINYINT, `size`: INT, `indices`: ARRAY<INT>, `values`: ARRAY<DOUBLE>>)\nUSING parquet"){code}
Try to insert new values to the table created from SQL:

 
{code:java}
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.VectorUDT
import org.apache.spark.mllib.linalg.Vectors
case class Test(features: Vector)
Seq(Test(Vectors.dense(Array(1d, 2d, 3d)))).toDF()
 .write
 .mode(SaveMode.Append)
 .insertInto("pborshchenko.test_vector_spark_0911_2")
{code}
 

Got:
 
{code:java}
 AnalysisException: Cannot write incompatible data to table '`pborshchenko`.`test_vector_spark_0911_2`': - Cannot write 'features': struct<type:tinyint,size:int,indices:array<int>,values:array<double>> is incompatible with struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;      - Cannot write 'features': struct<type:tinyint,size:int,indices:array<int>,values:array<double>> is incompatible with struct<type:tinyint,size:int,indices:array<int>,values:array<double>>; at org.apache.spark.sql.catalyst.analysis.TableOutputResolver$.resolveOutputColumns(TableOutputResolver.scala:72) at org.apache.spark.sql.execution.datasources.PreprocessTableInsertion.org$apache$spark$sql$execution$datasources$PreprocessTableInsertion$$preprocess(rules.scala:467) at org.apache.spark.sql.execution.datasources.PreprocessTableInsertion$$anonfun$apply$3.applyOrElse(rules.scala:494) at org.apache.spark.sql.execution.datasources.PreprocessTableInsertion$$anonfun$apply$3.applyOrElse(rules.scala:486) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$2(AnalysisHelper.scala:112)    {code}
 
The reason that table created from spark SQL has the type STRUCT, not vector, but this struct is the right representation for vector type.

AC: Should be possible to create a table using spark SQL with vector type column and after that write to it without any errors.
Bug: Spark reading data in scientific notation
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 09/Nov/20 10:52
Affects Version/s: 2.3.4

File is having below data

DAta
1200404151072.121111111111
1200404151073
1200404151074.1232323
1200404151075.124344
1200404151076.12
1200404151077.123433333
1200404151078.12
1200404151079.12544545454554
1251080.1234444444444444444444
10000000000000000000000000000

 

Spark is reading with scientific notation as we wanted to read data as it is available in file with accurate datatype not with string datatype.

+--------------------+
| DAta|
+--------------------+
|1.200404151072121E12|
| 1.200404151073E12|
|1.200404151074123...|
|1.200404151075124...|
| 1.20040415107612E12|
|1.200404151077123...|
| 1.20040415107812E12|
|1.200404151079125...|
| 1251080.1234444445|
| 1.0E28|
+--------------------
Bug: Allow specifying a minimum number of bytes in a split of a file
Status: Open | Priority: Major | Resolution: 
Created: 24/Nov/20 13:36
Affects Version/s: 3.0.1

*Background*
 Long time ago I have written a way for reading a (usually large) Gzipped file in a way that allows better distribution of the load over an Apache Hadoop cluster: [https://github.com/nielsbasjes/splittablegzip]

Seems like people still need this kind of functionality and it turns out my code works without modification in conjunction with Apache Spark.
 See for example:
 - SPARK-29102
 - [https://stackoverflow.com/q/28127119/877069]
 - [https://stackoverflow.com/q/27531816/877069]

So [~nchammas] provided documentation to my project a while ago on how to use it with Spark.
 [https://github.com/nielsbasjes/splittablegzip/blob/master/README-Spark.md]

*The problem*
 Now some people have indicated getting errors from this feature of mine.

Fact is that this functionality cannot read a split if it is too small (the number of bytes read from disk and the number of bytes coming out the compression are different). So my code uses the {{io.file.buffer.size}} setting but also has a hard coded lower limit split size of 4 KiB.

Now the problem I found when looking into the reports I got is that Spark does not have a minimum number of bytes in a split.

In fact: When I created a test file and then set the {{spark.sql.files.maxPartitionBytes}} to exactly 1 byte less than the size of my test file my library gave the error:
{{java.lang.IllegalArgumentException: The provided InputSplit (562686;562687] is 1 bytes which is too small. (Minimum is 65536)}}

I found the code that does this calculation here https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala#L74

*Proposed enhancement*
So what I propose is to have a new setting ({{spark.sql.files.minPartitionBytes}}  ?) that will guarantee that no split of a file is smaller than a configured number of bytes.

I also propose to have this set to something like 64KiB as a default.

Having some constraints on the values of {{spark.sql.files.minPartitionBytes}} and possibly in relation with {{spark.sql.files.maxPartitionBytes}} would be fine.

*Notes*

Hadoop already has code that does this: https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L456
Bug: New explicit cast syntax rules in ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/20 07:46
Affects Version/s: 3.1.0

In section 6.13 of the ANSI SQL standard,  there are syntax rules for valid combinations of the source and target data types.
To make Spark's ANSI mode more ANSI SQL Compatible，I propose to disallow the following casting in ANSI mode:

{code:java}
TimeStamp <=> Boolean
Date <=> Boolean
Numeric <=> Timestamp
Numeric <=> Date
Numeric <=> Binary
String <=> Array
String <=> Map
String <=> Struct
{code}


The following castings are considered invalid in ANSI SQL standard, but they are quite straight forward. Let's Allow them for now

{code:java}
Numeric <=> Boolean
String <=> Boolean
String <=> Binary
{code}
Bug: Stage level scheduling, allow specifying other spark configs via ResourceProfile
Status: Open | Priority: Major | Resolution: 
Created: 13/Nov/20 18:17
Affects Version/s: 3.1.0

With the addition of stage level scheduling and ResourceProfiles, we only allow certain resources to specified currently.  There are many other configs that users may want to change between changes. We should look at adding in perhaps SparkConf settings.

One very specific one that was brought up in review was ability to change yarn queue (if that is even possible in yarn) between stages because someone might want ot use one queue for ETL and a separate queue for ML where there are nodes with GPU.  Or perhaps node labels separately.
Bug: Support SHOW TABLE EXTENDED in DSv2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 10:10
Affects Version/s: 3.1.0

Current implementation of DSv2 SHOW TABLE doesn't support the EXTENDED mode in:
https://github.com/apache/spark/blob/d6a68e0b67ff7de58073c176dd097070e88ac831/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala#L33
which is supported in DSv1:
https://github.com/apache/spark/blob/7e99fcd64efa425f3c985df4fe957a3be274a49a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L870

Need to add the same functionality to ShowTablesExec.
Bug: Hive partition pruning support Contains, StartsWith and EndsWith predicate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/20 02:12
Affects Version/s: 3.1.0

Hive partition pruning can support Contains, StartsWith and EndsWith predicate:
https://github.com/apache/hive/blob/0c2c8a7f57330880f156466526bc0fdc94681035/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L1074-L1075
https://github.com/apache/hive/commit/0c2c8a7f57330880f156466526bc0fdc94681035#diff-b1200d4259fafd48d7bbd0050e89772218813178f68461a2e82551c52319b282
Bug: Can't parse decimal type from csv file
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 13/Nov/20 14:47
Affects Version/s: 2.1.3

The attached file is a one column csv file containing decimals.

Execute: {color:#de350b}mydf2 = spark_session.read.csv("tsd.csv", header=True, inferSchema=True){color}

Then invoking {color:#de350b}mydf2.schema{color} will result in error:

{color:#ff8b00}ValueError: Could not parse datatype: decimal(6,-7){color}
Bug: Bump ANTLR runtime version to 4.8-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 04:46
Affects Version/s: 3.1.0

This PR intends to upgrade ANTLR runtime from 4.7.1 to 4.8-1.
Release note of v4.8 and v4.7.2 (the v4.7.2 release has a few minor bug fixes for java targets):

 - v4.8: https://github.com/antlr/antlr4/releases/tag/4.8
 - v4.7.2: https://github.com/antlr/antlr4/releases/tag/4.7.2
Bug: Simplify the exception message from Python UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/20 05:05
Affects Version/s: 3.1.0

Currently, the exception message is as below:

{code}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../python/pyspark/sql/dataframe.py", line 427, in show
    print(self._jdf.showString(n, 20, vertical))
  File "/.../python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/.../python/pyspark/sql/utils.py", line 127, in deco
    raise_from(converted)
  File "<string>", line 3, in raise_from
pyspark.sql.utils.PythonException:
  An exception was thrown from Python worker in the executor:
Traceback (most recent call last):
  File "/.../python/lib/pyspark.zip/pyspark/worker.py", line 605, in main
    process()
  File "/.../python/lib/pyspark.zip/pyspark/worker.py", line 597, in process
    serializer.dump_stream(out_iter, outfile)
  File "/.../python/lib/pyspark.zip/pyspark/serializers.py", line 223, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/.../python/lib/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
    for obj in iterator:
  File "/.../python/lib/pyspark.zip/pyspark/serializers.py", line 212, in _batched
    for item in iterator:
  File "/.../python/lib/pyspark.zip/pyspark/worker.py", line 450, in mapper
    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
  File "/.../python/lib/pyspark.zip/pyspark/worker.py", line 450, in <genexpr>
    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
  File "/.../python/lib/pyspark.zip/pyspark/worker.py", line 90, in <lambda>
    return lambda *a: f(*a)
  File "/.../python/lib/pyspark.zip/pyspark/util.py", line 107, in wrapper
    return f(*args, **kwargs)
  File "<stdin>", line 3, in divide_by_zero
ZeroDivisionError: division by zero
{code}

Actually, almost all cases, users only care about {{ZeroDivisionError: division by zero
}}. We don't really have to show the internal stuff in 99% cases.

We could just make it short, for example,

{code}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../python/pyspark/sql/dataframe.py", line 427, in show
    print(self._jdf.showString(n, 20, vertical))
  File "/.../python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/.../python/pyspark/sql/utils.py", line 127, in deco
    raise_from(converted)
  File "<string>", line 3, in raise_from
pyspark.sql.utils.PythonException:
  An exception was thrown from Python worker in the executor:
Traceback (most recent call last):
  File "<stdin>", line 3, in divide_by_zero
ZeroDivisionError: division by zero
{code}
Bug: change 'spark.sql.adaptive.skewedPartitionThresholdInBytes' to 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/20 13:55
Affects Version/s: 3.0.0

In the 'Optimizing Skew Join' section of the following two pages:
 # [https://spark.apache.org/docs/3.0.0/sql-performance-tuning.html]
 # [https://spark.apache.org/docs/3.0.1/sql-performance-tuning.html]

The configuration 'spark.sql.adaptive.skewedPartitionThresholdInBytes' should be changed to 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes', The former is missing the 'skewJoin'.
Bug: set -v couldn't dump all the conf entries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/20 02:28
Affects Version/s: 3.0.1

since scala object is lazy init, it won't be load until some code touched it. For SQL conf entries, it won't be registered if the conf object is never touched. So "set -v" couldn't dump all the defined configs (even if it says so)
Bug: Encoding is not working if multiLine option is true.
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 20/Nov/20 11:34
Affects Version/s: 2.3.4

If we read with mulitLine true and encoding with "ISO-8859-1" then we are getting  value like this {color:#ff0000}AUTO EL*�*TRICA{color}. and if we read with multiLine false  and encoding with "ISO-8859-1" thne we are getting  value like {color:#ff0000}AUTO EL*É*TRICA{color}

Below is the code we are using

{code}
spark.read().option("header", "true").option("inferSchema", true).option("delimiter", ";") .option("quote", "\"") .option("multiLine", true) .option("encoding", "ISO-8859-1").csv("1605860036183.csv").show()
{code}

Sample file is attached in attachement
Bug: Pyspark application will hang due to non Exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 09:39
Affects Version/s: 2.4.5

When a system.exit exception occurs during the process, the python worker exits abnormally, and then the executor task is still waiting for the worker for reading from socket, causing it to hang.
 The system.exit exception may be caused by the user's error code, but spark should at least throw an error to remind the user, not get stuck
 we can run a simple test to reproduce this case:
{code:python}
from pyspark.sql import SparkSession

def err(line):
  raise SystemExit

spark = SparkSession.builder.appName("test").getOrCreate()
spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
spark.stop()
{code}
Bug: Handling of hybrid to proleptic calendar when reading and writing Parquet data not working correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 10:54
Affects Version/s: 3.0.0

The handling of old dates written with older Spark versions (<2.4.6) using the hybrid calendar in Spark 3.0.0 and 3.0.1 seems to be broken/not working correctly.

From what I understand it should work like this:
 * Only relevant for `DateType` before 1582-10-15 or `TimestampType` before 1900-01-01T00:00:00Z
 * Only applies when reading or writing parquet files
 * When reading parquet files written with Spark < 2.4.6 which contain dates or timestamps before the above mentioned moments in time a `SparkUpgradeException` should be raised informing the user to choose either `LEGACY` or `CORRECTED` for the `datetimeRebaseModeInRead`
 * When reading parquet files written with Spark < 2.4.6 which contain dates or timestamps before the above mentioned moments in time and `datetimeRebaseModeInRead` is set to `LEGACY` the dates and timestamps should show the same values in Spark 3.0.1. with for example `df.show()` as they did in Spark 2.4.5
 * When reading parquet files written with Spark < 2.4.6 which contain dates or timestamps before the above mentioned moments in time and `datetimeRebaseModeInRead` is set to `CORRECTED` the dates and timestamps should show different values in Spark 3.0.1. with for example `df.show()` as they did in Spark 2.4.5
 * When writing parqet files with Spark > 3.0.0 which contain dates or timestamps before the above mentioned moment in time a `SparkUpgradeException` should be raised informing the user to choose either `LEGACY` or `CORRECTED` for the `datetimeRebaseModeInWrite`

First of all I'm not 100% sure all of this is correct. I've been unable to find any clear documentation on the expected behavior. The understanding I have was pieced together from the mailing list ([http://apache-spark-user-list.1001560.n3.nabble.com/Spark-3-0-1-new-Proleptic-Gregorian-calendar-td38914.html)] the blog post linked there and looking at the Spark code.

From our testing we're seeing several issues:
 * Reading parquet data with Spark 3.0.1 that was written with Spark 2.4.5. that contains fields of type `TimestampType` which contain timestamps before the above mentioned moments in time without `datetimeRebaseModeInRead` set doesn't raise the `SparkUpgradeException`, it succeeds without any changes to the resulting dataframe compared to that dataframe in Spark 2.4.5
 * Reading parquet data with Spark 3.0.1 that was written with Spark 2.4.5. that contains fields of type `TimestampType` or `DateType` which contain dates or timestamps before the above mentioned moments in time with `datetimeRebaseModeInRead` set to `LEGACY` results in the same values in the dataframe as when using `CORRECTED`, so it seems like no rebasing is happening.

I've made some scripts to help with testing/show the behavior, it uses pyspark 2.4.5, 2.4.6 and 3.0.1. You can find them here [https://github.com/simonvanderveldt/spark3-rebasemode-issue]. I'll post the outputs in a comment below as well.
Bug: grouped by first and last against date column returns incorrect results
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/20 15:39
Affects Version/s: 2.4.6

The attached csv file has two columns, namely "User" and "FromDate".  The import defaults the "FromDate" column as a timestamp. 
 * outDF = spark_session.read.csv("users.csv", inferSchema=True, header=True)
 * outDF.createOrReplaceTempView("table02")

In this default case the following sql generates {color:#de350b}*incorrect*{color} results:

{color:#de350b}*"select count(`User`) as cnt, first(`FromDate`) as `FromDate_First`, last(`FromDate`) as `FromDate_Last`, count(distinct(`FromDate`)) as cntdist from table02 group by `User`"*{color}

{color:#172b4d}However if we read the dataframe like so (where the "FromDate" is read in as a Date, then the above sql query {color:#de350b}*also*{color} generates  *incorrect* {color} results:
 * outDF = spark_session.read.csv("users.csv", inferSchema=True, header=True).selectExpr("`User`", "cast(`FromDate` as date)")
Bug: provided an invalid fromOffset when spark streaming running
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/20 05:14
Affects Version/s: 2.3.2

!image-2020-11-04-17-24-23-782.png|width=923,height=386!

 

 

!image-2020-11-04-17-24-42-544.png|width=1199,height=260!
Bug: Dataframe: data is wrongly presented because of column name
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 03/Nov/20 01:48
Affects Version/s: 2.4.5

Consider the code below: `some_text` column got the `some_int` value, while its value is null in the dataframe.

   !image-2020-11-03-14-57-09-433.png!

 

Renaming the field from `some_text` to `some_apple`, fixed the problem! 🙂

!image-2020-11-03-14-57-37-308.png!

 

 

Here is the code to reproduce the problem
{code:python}
from datetime import datetime
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, DateType, StringType, IntegerType
 
schema = StructType(
  [
    StructField('dfdt', DateType(), True),
    StructField('some_text', StringType(), True),
    StructField('some_int', IntegerType(), True),
  ]
)
 
test_df = spark.createDataFrame([
  Row(dfdt=datetime.strptime('2020-12-18', '%Y-%m-%d'), some_text='cdsvg', some_int=100)
], schema)
 
display(test_df)
{code}
Bug: explode should not filter when used with CreateArray
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 20:55
Affects Version/s: 3.1.0

https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to insert a filter for not null and size > 0 when using inner explode/inline. This is fine in most cases but the extra filter is not needed if the explode is with a create array and not using Literals (it already handles LIterals).  When this happens you know that the values aren't null and it has a size.  It already handles the empty array.

for instance:

val df = someDF.selectExpr("number", "explode(array(word, col3))")

So in this case we shouldn't be inserting the extra Filter and that filter can get pushed down into like a parquet reader as well. This is just causing extra overhead.
Bug: Support --archives option natively
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 11:46
Affects Version/s: 3.1.0

Currently, {{spark-submit --archives}} and {{spark.yarn.dist.archives}} configuration are only supported in Yarn modes:

{code}
spark-submit --help
...
 Spark on YARN only:
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.
{code}

This is actually critical for PySpark to support shipping other packages together, see also https://hyukjin-spark.readthedocs.io/en/stable/user_guide/python_packaging.html#using-zipped-virtual-environment.

Due to this missing feature, PySpark cannot support conda env to ship other packages together.
Bug: [Python][Tests] Replace calls to deprecated test APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 20:39
Affects Version/s: 3.0.0

There are quite a few instances of using deprecated python and pandas APIs for pyspark tests. These should be replaced by the preferred APIs which have been available for some time. For example in unittests: `assertRaisesRegexp()` -> `assertRaisesRegex()`
Bug: Spark Hive SQL returning empty dataframe
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Nov/20 10:30
Affects Version/s: 2.4.6

I am trying to run a sql query on a hive table using hive connector in spark but I am getting an empty dataframe. The query I am trying to run:-

{{sparkSession.sql("select fmid from farmers where fmid between ' 1000405134' and '1000772585'")}}

This is failing but if I remove the leading whitespaces it works.

{{sparkSession.sql("select fmid from farmers where fmid between '1000405134' and '1000772585'")}}

Currently, I am removing leading and trailing whitespaces as a workaround. But the same query with whitespaces works fine in hive console.