av9ash/GitBugs huggingface dataset:
Bug: sql("INSERT INTO v2DataSource ...").collect() double inserts
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 18/Mar/20 01:21
Affects Version/s: 3.0.0

The following unit test fails in DataSourceV2SQLSuite:
{code:java}
test("do not double insert on INSERT INTO collect()") {
  import testImplicits._
  val t1 = s"${catalogAndNamespace}tbl"
  sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format")
  val tmpView = "test_data"
  val df = Seq((1L, "a"), (2L, "b"), (3L, "c")).toDF("id", "data")
  df.createOrReplaceTempView(tmpView)
  sql(s"INSERT INTO TABLE $t1 SELECT * FROM $tmpView").collect()

  verifyTable(t1, df)
} {code}
The INSERT INTO is double inserting when ".collect()" is called. I think this is because the V2 SparkPlans are not commands, and doExecute on a Spark plan can be called multiple times causing data to be inserted multiple times.
Bug: Spark's SQL "group by" local processing operator is broken.
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 05/Mar/20 21:09
Affects Version/s: 2.4.3

When applying "GROUP BY" processing operator (without an "ORDER BY" clause), I expect to see all the grouping columns being grouped together to the same buckets. However, this is not the case.

Steps to reproduce:
 1. Start spark-shell as follows:
 bin\spark-shell.cmd --master local[4] --conf spark.sql.catalogImplementation=in-memory
 2. Load the attached csv file:
 val gosales = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("c:/Users/MichaelGiannakopoulo/Downloads/SampleFile_GOSales.csv")
 3. Create a temp view:
 gosales.createOrReplaceTempView("gosales")
 4. Execute the following sql statement:
 spark.sql("SELECT `Product line`, `Order method type`, sum(`Revenue`) FROM `gosales` GROUP BY `Product line`, `Order method type`").show()

Output: 
 +-----------------------+--------------++----------------------------
|Product line|Order method type|sum(CAST(Revenue AS DOUBLE))|

+-----------------------+--------------++----------------------------
|Golf Equipment|E-mail|92.25|
|Camping Equipment|Mail|0.0|
|Camping Equipment|Fax|null|
|Golf Equipment|Telephone|123.0|
|Camping Equipment|Special|null|
|Outdoor Protection|Telephone|34218.19|
|Mountaineering Eq...|Mail|0.0|
|Camping Equipment|Web|32469.03|
|Personal Accessories|Fax|3318.7|
|Golf Equipment|Sales visit|143.5|
|Mountaineering Eq...|Telephone|null|
|Mountaineering Eq...|E-mail|null|
|Outdoor Protection|Sales visit|20522.42|
|Outdoor Protection|Fax|5857.54|
|Personal Accessories|E-mail|26679.640000000003|
|Mountaineering Eq...|Fax|null|
|Outdoor Protection|Web|340836.85000000003|
|Golf Equipment|Special|0.0|
|Outdoor Protection|E-mail|28505.93|
|Golf Equipment|Web|3034.0|

+-----------------------+--------------++----------------------------

Expected output:
 +-----------------------+--------------++----------------------------
|Product line|Order method type|sum(CAST(Revenue AS DOUBLE))|

+-----------------------+--------------++----------------------------
|Golf Equipment|E-mail|92.25|
|Golf Equipment|Fax|null|
|Golf Equipment|Mail|0.0|
|Golf Equipment|Sales visit|143.5|
|Golf Equipment|Special|0.0|
|Golf Equipment|Telephone|123.0|
|Golf Equipment|Web|3034.0|
|Camping Equipment|E-mail|1303.3999999999999|
|Camping Equipment|Fax|null|
|Camping Equipment|Sales visit|4754.87|
|Camping Equipment|Mail|0.0|
|Camping Equipment|Special|null|
|Camping Equipment|Telephone|5169.65|
|Camping Equipment|Web|32469.03|
|Mountaineering Eq...|E-mail|null|
|Mountaineering Eq...|Fax|null|
|Mountaineering Eq...|Mail|0.0|
|Mountaineering Eq...|Special|null|
|Mountaineering Eq...|Sales visit|null|
|Mountaineering Eq...|Telephone|null|

+-----------------------+--------------++----------------------------

Notice how in the expected output all the grouping columns should be bucketed together without necessarily being in order, which is not the case with the output that spark produces.
Bug: Spark 3.0 built against hadoop2.7 can't start standalone master
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 04/Mar/20 19:22
Affects Version/s: 3.0.0

trying to start a standalone master when building spark branch 3.0 with hadoop2.7 fails with:

 
Exception in thread "main" java.lang.NoClassDefFoundError: org/w3c/dom/ElementTraversal
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at 
[java.net|http://java.net/]
.URLClassLoader.defineClass(URLClassLoader.java:468)
        at 
[java.net|http://java.net/]
.URLClassLoader.access$100(URLClassLoader.java:74)
        at 
[java.net|http://java.net/]
.URLClassLoader$1.run(URLClassLoader.java:369)
...
Caused by: java.lang.ClassNotFoundException: org.w3c.dom.ElementTraversal
        at 
[java.net|http://java.net/]
.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        ... 42 more
Bug: Hit OOM Error - GC Limit
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 27/Mar/20 06:18
Affects Version/s: 2.4.4

MemoryStore is 2.6GB

conf = new SparkConf().setAppName("test")
 //.set("spark.sql.codegen.wholeStage", "false")
 .set("spark.driver.host", "localhost")
 .set("spark.driver.memory", "4g")
 .set("spark.executor.cores","1")
 .set("spark.num.executors","1")
 .set("spark.executor.memory", "4g")
 .set("spark.executor.memoryOverhead", "400m")
 .set("spark.dynamicAllocation.enabled", "true")
 .set("spark.dynamicAllocation.minExecutors","1")
 .set("spark.dynamicAllocation.maxExecutors","2")
 .set("spark.ui.enabled","true") //enable spark UI
 .set("spark.sql.shuffle.partitions",defaultPartitions)
 .setMaster("local[2]")
 sparkSession = SparkSession.builder.config(conf).getOrCreate()

 

val df = SparkFactory.sparkSession.sqlContext
 .read
 .option("header", "true")
 .option("delimiter", delimiter)
 .csv(textFileLocation)

 

joinedDf = upperCaseLeft.as("l")
 .join(upperCaseRight.as("r"), caseTransformedKeys, "full_outer")
 .select(compositeKeysCol ::: nonKeyCols.map(col => mapHelper(col,toleranceValue,caseSensitive)): _*)

 

data = joinedDf.take(maxRecords)
Bug: DataFrameReader.csv incorrectly reads gzip encoded CSV from S3 when it has non-".gz" extension
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 17/Mar/20 22:32
Affects Version/s: 2.4.4

i have large CSV files that are gzipped and uploaded to S3 with Content-Encoding=gzip. the files have file extension ".csv", as most web clients will automatically decompress the file based on the Content-Encoding header. using pyspark to read these CSV files does not mimic this behavior.

works as expected:
{code:java}
df = spark.read.csv('s3://bucket/large.csv.gz', header=True)
{code}
does not decompress and tries to load entire contents of file as the first row:
{code:java}
df = spark.read.csv('s3://bucket/large.csv', header=True)
{code}

it looks like it's relying on the file extension to determine if the file is gzip compressed or not. it would be great if S3 resources, and any other http based resources, could consult the Content-Encoding response header as well.

i tried to find the code that determines this, but i'm not familiar with the code base. any pointers would be helpful. and i can look into fixing it.
Bug: Drop does not work after join with aliases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/20 13:10
Affects Version/s: 2.4.2

 

Hi,

I am seeing a really strange behaviour in drop method after a join with aliases. It doesn't seem to find the column when I reference to it using dataframe("columnName") syntax, but it does work with other combinators like select
{code:java}
case class Record(a: String, dup: String)
case class Record2(b: String, dup: String)

val df = Seq(Record("a", "dup")).toDF
val df2 = Seq(Record2("a", "dup")).toDF 
val joined = df.alias("a").join(df2.alias("b"), df("a") === df2("b"))
val dupCol = df("dup")
joined.drop(dupCol) // Does not drop anything
joined.drop(func.col("a.dup")) // It drops the column  
joined.select(dupCol) // It selects the column
{code}
Bug: Contrived working example that works with multiple URI file storages for Spark cluster mode
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 26/Mar/20 19:05
Affects Version/s: 2.4.5

This Spark SQL Guide --> Data sources --> Generic Load/Save Functions

[https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html]

described a very simple "local file system load of an example file".  

 

I am looking for an example that demonstrates a workflow that exercises different file systems.  For example, 
 # Driver loads an input file from local file system
 # Add a simple column using lit() and stores that DataFrame in cluster mode to HDFS
 # Write that a small limited subset of that DataFrame back to Driver's local file system.  (This is to avoid the anti-pattern of writing large file and out of the scope for this example.  The small limited DataFrame would be some basic statistics, not the actual complete dataset.)

 

The examples I found on the internet only uses simple paths without the explicit URI prefixes.

Without the explicit URI prefixes, the "filepath" inherits how Spark (mode) was called, local stand alone vs YARN client mode.   So a "filepath" will be read/write locally (file system) vs cluster mode HDFS, without these explicit URIs.

There are situations were a Spark program needs to deal with both local file system and YARN client mode (big data) in the same Spark application, like producing a summary table stored on the local file system of the driver at the end.  

If there are any existing alternatives Spark documentation that provides examples that traverse through the different URIs in Spark YARN client mode or a better or smarter Spark pattern or API that is more suited for this, I am happy to accept that as well.  Thanks!
Bug: groupby().applyInPandas, groupby().cogroup().applyInPandas and mapInPandas should ignore type hints
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 11:54
Affects Version/s: 3.1.0

Setting type hints in pandas function API should not matter at this moment. However, currently it tries to infer type hint when it's set.

{code}
import pandas as pd

def pandas_plus_one(v: pd.DataFrame) -> pd.DataFrame:
    return v + 1

spark.range(10).groupby('id').applyInPandas(pandas_plus_one, schema="id long").show()
{code}

{code}
Traceback (most recent call last):
  File "/.../spark/python/pyspark/sql/utils.py", line 98, in deco
    return f(*a, **kw)
  File "/.../spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o34.flatMapGroupsInPandas.
: java.lang.IllegalArgumentException: requirement failed: Must pass a grouped map udf
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.sql.RelationalGroupedDataset.flatMapGroupsInPandas(RelationalGroupedDataset.scala:541)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/pandas/group_ops.py", line 182, in applyInPandas
    jdf = self._jgd.flatMapGroupsInPandas(udf_column._jc.expr())
  File "/.../spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/.../spark/python/pyspark/sql/utils.py", line 102, in deco
    raise converted
pyspark.sql.utils.IllegalArgumentException: requirement failed: Must pass a grouped map udf
{code}

Looks {{groupby().cogroup().applyInPandas}} and {{mapInPandas}} also have the same issues.
Bug: Failure of casting the '1000-02-29' string to the date type
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 21/Mar/20 17:55
Affects Version/s: 2.4.5

The '1000-02-29' is valid date in the Julian calendar used in Spark 2.4.5 for dates before 1582-10-15 but casting the string to the date type fails:
{code:scala}
scala> val df = Seq("1000-02-29").toDF("dateS").select($"dateS".cast("date").as("date"))
df: org.apache.spark.sql.DataFrame = [date: date]

scala> df.show
+----+
|date|
+----+
|null|
+----+
{code}
Creating a dataset from java.sql.Date w/ the same input string works correctly:
{code:scala}
scala> val df2 = Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date"))
df2: org.apache.spark.sql.DataFrame = [date: date]

scala> df2.show
+----------+
|      date|
+----------+
|1000-02-29|
+----------+
{code}
Bug: Add version information to the configuration of Kafka
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 11:56
Affects Version/s: 3.1.0

external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/package.scala
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/package.scala
Bug: Support json_object_keys function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 11:26
Affects Version/s: 3.1.0

This function will return all the keys from outer json object.

 

PostgreSQL  -> [https://www.postgresql.org/docs/9.3/functions-json.html]

Mysql -> [https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html]

MariaDB -> [https://mariadb.com/kb/en/json-functions/]
Bug: Not compatible with new version of scalatest (3.1.0 and above)
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/20 19:02
Affects Version/s: 3.1.0

Since  ScalaTest's style traits and classes were moved and renamed ([http://www.scalatest.org/release_notes/3.1.0]) there are errors as not find FunSpec when I add new version of scalatest in library dependency.
Bug: Optimized Logical Plan  cast('' as timestamp) is null
Status: Closed | Priority: Major | Resolution: Incomplete
Created: 12/Mar/20 10:31
Affects Version/s: 2.3.1

The type of receive_time is timestamp，The value is  '2020-02-10'

SQL：

select case when receive_time<>''
 then 1 else 0 end as tt
 from xxx.xxx
 where order_id=1234；

The output error is 0；

Because cast( as timestamp) is null, receive_time<>'' is false;

 

!image-2020-03-12-18-34-22-571.png!
Bug: Issue with Spark Context Initialization  i.e. SparkSession.builder.getOrCreate()
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Mar/20 05:15
Affects Version/s: 2.4.5

Issue : 

I am trying to initialize spark context using below code : 

 

from pyspark.sql import SparkSession
 spark = SparkSession.builder.getOrCreate()
 import pandas as pd
 sc = spark.sparkContext

 

At getOrCreate() step , code is going to infinite loop and never respond back  either with exception or timeout. 

 

On execution if i  check on VS Terminal or Anaconda cmd prompot - below statement is getting displayed : 

[I 19:13:47.973 NotebookApp] Saving file at /LearnPython/Assignment3/NLP.ipynb
 *The filename, directory name, or volume label syntax is incorrect.*

 

 

Note : This used to work fine till a month back - not sure what chnaged, it stopped working from yesterday.
Bug: Connect Spark Cluster Exception on IDEA develop
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Mar/20 04:12
Affects Version/s: 1.5.1

Connect Spark Cluster Exception on IDEA develop：

WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@192.168.159.129:7077] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main]
Bug: Random Forest in SparkML 2.3.3 vs 2.4.x
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Mar/20 01:24
Affects Version/s: 2.3.3

Hi all,

When I trained the model with the Random Forest algorithm, I got different results in different versions of spark, the same input, label ratio, hyperparameter for all training. Detailed training results in the attached file. Model training results with spark 2.3.3 are much better, so I want to ask if there have been any changes to the random forest (or other algorithms) in mllib?

Many thanks.
Bug: An issue for Spark SQL LIKE-with-ESCAPE clause
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Mar/20 00:31
Affects Version/s: 3.0.0

I try to use LIKE with ESCAPE for Spark 3.0.0-preview2. But I find in it doesn't work in below cases.

The database table

==============

create or replace table test_table_like ( subject string)

insert into $test_table_like values ('100 times'), ('1000 times'), ('100%')

 

Repro

====

val result2 = sparkSession.sql(
 s"select * from test_table_like where subject like '100^%' escape '^' order by 1")

"100%" is expected to returned, but it doesn't. I debug into the code to check the logical plan.

In the logical plan, the LIKE is transformed as "StartsWith(subject#130, 100^)". It looks it is incorrect.
Bug: Spark shell version miss match
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 19/Mar/20 06:54
Affects Version/s: 3.0.0

Hi Team,
 I downloaded the spark3.x latest tar ball from the spark website.

when tried to access the spark-shell I am getting version as 2.4.4. Attaching the screen short.

 
 !screenshot-1.png!
Bug: spark sql runs successfully with query not specifying condition next to where 
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 19/Mar/20 15:08
Affects Version/s: 2.4.5

When having a sql query as follows:

{color:#00875a}_SELECT *_{color}

{color:#00875a}_FROM people_{color}

{color:#00875a}_WHERE_{color}

shouldn't we throw a parsing exception because of __unspecified _condition_  _?_
Bug: Add version information to the configuration of Dstreams
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/20 03:20
Affects Version/s: 3.1.0

[streaming/src/main/scala/org/apache/spark/streaming/StreamingConf.scala|https://github.com/apache/spark/commit/8ee41f3576689f3d164131d1e6041bd347394364#diff-2d3575b1e15bbd0c4feb626d5cda0008]
Bug: Reuse days rebase functions of DateTimeUtils in DaysWritable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/20 19:58
Affects Version/s: 3.0.0

The functions rebaseJulianToGregorianDays() and rebaseGregorianToJulianDays() were added by the PR https://github.com/apache/spark/pull/27915. The ticket aims to replace similar code in org.apache.spark.sql.hive.DaysWritable by the functions to:
# Deduplicate code
# The functions were better tested, and cross checked by reading parquet files saved by Spark 2.4
Bug: percentile_approx function not working as expected
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 31/Mar/20 06:15
Affects Version/s: 2.4.0

I'm using spark-sql-2.4.1v with Java 8 and I'm trying to do find quantiles, i.e. percentile 0, percentile 25, etc, on the given column data of dataframe.

Column values data set is as below 23456.55,34532.55,23456.55

When I use percentile_approx() function the results are not matching to that of Excel percentile_inc() function.

Ex :

for above data set i.e. 23456.55,34532.55,23456.55
percentile_0,percentile_10,percentile_25,percentile_50,percentile_75,percentile_90,percentile_100 respectively 
using percentile_approx() function
23456.55,23456.55,23456.55,23456.55,23456.55,23456.55,23456.55

Using excel i.e. percentile_inc()
23456.55,23456.55,23456.55,23456.55,28994.550000000003,32317.350000000002,34532.55
How to get correct percentiles as excel using percentile_approx() function?

For the details please check it.
Bug: Upgrade Scala to 2.12.14
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/20 19:45
Affects Version/s: 3.2.0

h2. Highlights
 * Performance improvements in the collections library: algorithmic improvements and changes to avoid unnecessary allocations ([list of PRs|https://github.com/scala/scala/pulls?q=is%3Apr+milestone%3A2.12.11+is%3Aclosed+sort%3Acreated-desc+label%3Alibrary%3Acollections+label%3Aperformance])
 * Performance improvements in the compiler ([list of PRs|https://github.com/scala/scala/pulls?q=is%3Apr+milestone%3A2.12.11+is%3Aclosed+sort%3Acreated-desc+-label%3Alibrary%3Acollections+label%3Aperformance+], minor [effects in our benchmarks|https://scala-ci.typesafe.com/grafana/dashboard/db/scala-benchmark?orgId=1&from=1567985515850&to=1584355915694&var-branch=2.12.x&var-source=All&var-bench=HotScalacBenchmark.compile&var-host=scalabench@scalabench@])
 * Improvements to {{-Yrepl-class-based}}, an alternative internal REPL encoding that avoids deadlocks (details on [#8712|https://github.com/scala/scala/pull/8712])
 * A new {{-Yrepl-use-magic-imports}} flag that avoids deep class nesting in the REPL, which can lead to deteriorating performance in long sessions ([#8576|https://github.com/scala/scala/pull/8576])
 * Fix some {{toX}} methods that could expose the underlying mutability of a {{ListBuffer}}-generated collection ([#8674|https://github.com/scala/scala/pull/8674])

h3. JDK 9+ support
 * ASM was upgraded to 7.3.1, allowing the optimizer to run on JDK 13+ ([#8676|https://github.com/scala/scala/pull/8676])
 * {{:javap}} in the REPL now works on JDK 9+ ([#8400|https://github.com/scala/scala/pull/8400])

h3. Other changes
 * Support new labels for creating durations for consistency: {{Duration("1m")}}, {{Duration("3 hrs")}} ([#8325|https://github.com/scala/scala/pull/8325], [#8450|https://github.com/scala/scala/pull/8450])
 * Fix memory leak in runtime reflection's {{TypeTag}} caches ([#8470|https://github.com/scala/scala/pull/8470]) and some thread safety issues in runtime reflection ([#8433|https://github.com/scala/scala/pull/8433])
 * When using compiler plugins, the ordering of compiler phases may change due to [#8427|https://github.com/scala/scala/pull/8427]

For more details, see [https://github.com/scala/scala/releases/tag/v2.12.11].