av9ash/GitBugs huggingface dataset:
Bug: Spark SQL codegen's code() interpolator should treat escapes like Scala's StringContext.s()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 01:17
Affects Version/s: 2.4.0

The {{code()}} string interpolator in Spark SQL's code generator should treat escapes like Scala's builtin {{StringContext.s()}} interpolator, i.e. it should treat escapes in the code parts, and should not treat escapes in the input arguments.

For example,
{code}
val arg = "This is an argument."
val str = s"This is string part 1. $arg This is string part 2."
val code = code"This is string part 1. $arg This is string part 2."
assert(code.toString == str)
{code}
We should expect the {{code()}} interpolator produce the same thing as the {{StringContext.s()}} interpolator, where only escapes in the string parts should be treated, while the args should be kept verbatim.

But in the current implementation, due to the eager folding of code parts and literal input args, the escape treatment is incorrectly done on both code parts and literal args.
That causes a problem when an arg contains escape sequences and wants to preserve that in the final produced code string. For example, in {{Like}} expression's codegen, there's an ugly workaround for this bug:
{code}
      // We need double escape to avoid org.codehaus.commons.compiler.CompileException.
      // '\\' will cause exception 'Single quote must be backslash-escaped in character literal'.
      // '\"' will cause exception 'Line break in literal not allowed'.
      val newEscapeChar = if (escapeChar == '\"' || escapeChar == '\\') {
        s"""\\\\\\$escapeChar"""
      } else {
        escapeChar
      }
{code}
Bug: Spark streaming, kafka broker error, "Failed to get records for spark-executor- .... after polling for 512"
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 06/Feb/20 03:01
Affects Version/s: 2.0.2

We have a spark streaming application reading data from Kafka.
 Data size: 15 Million

Below errors were seen:
 java.lang.AssertionError: assertion failed: Failed to get records for spark-executor- .... after polling for 512 at scala.Predef$.assert(Predef.scala:170)

There were more errors seen pertaining to CachedKafkaConsumer
 at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:74)
 at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:227)
 at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:193) 
 at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
 at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:214)
 at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)
 at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)
 at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
 at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926) 
 at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670) 
 at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
 at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) 
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
 at org.apache.spark.scheduler.Task.run(Task.scala:86)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

 

The spark.streaming.kafka.consumer.poll.ms is set to default 512ms and other Kafka stream timeout settings are default.
 "request.timeout.ms" 
 "heartbeat.interval.ms" 
 "session.timeout.ms" 
 "max.poll.interval.ms" 

Also, the kafka is being recently updated to 0.10 from 0.8. In 0.8, this behavior was not seen. 
No resource issues are seen.
Bug: The data returned from SAS using JDBC reader contains column label
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 05/Feb/20 19:00
Affects Version/s: 2.1.1

When read SAS data using JDBC with SAS SHARE driver, the returned data contains column labels, rather data. 

According to testing result from SAS Support, the results are correct using Java. So they believe it is due to spark reading.
Bug: Add dtype="float32" support to vector_to_array UDF
Status: Resolved | Priority: Major | Resolution: Done
Created: 09/Feb/20 06:56
Affects Version/s: 3.0.0

Previous PR: [https://github.com/apache/spark/blob/master/python/pyspark/ml/functions.py]

In the previous PR, we introduced a UDF to convert a column of MLlib Vecters to a column of lists in python (Seq in scala). Currently, all the floating numbers in a vector is converted to Double in scala. In this issue, we will add a parameter in the python function {{vector_to_array(col)}} that allows converting to Float (32bits) in scala, which would be mapped to a numpy array of dtype=float32.
Bug: Null-safe variant of Dataset.join(Dataset[_], Seq[String])
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/Feb/20 09:02
Affects Version/s: 3.1.0

The {{Dataset.join(Dataset, Seq[String])}} method provides extra convenience over {{Dataset.join(Dataset, joinExprs: Column)}} as it does not duplicate the join columns {{Seq[String]}} in the result {{DataFrame}}. Those columns are compared with {{===}}. When those join columns need to be compared null-safe with {{<=>}}, the join condition becomes very verbose and requires extra {{drop}} operations:
{code:java}
df1.join(df2, df1("a") <=> df2("a") && df1("b") <=> df2("b")).drop(df2("a")).drop(df2("b")).show()
{code}
Elegant would be the following null-safe join operation:
{code:java}
df1.joinNullSafe(df2, joinColumns)
{code}
Possible namings:
 - {{Dataset.joinNullSafe(Dataset[_], Seq[String])}}
 - {{Dataset.joinWithNulls(Dataset[_], Seq[String])}}
 - {{Dataset.join(Dataset[_], Seq[String], <=>)}}

*I am happy to provide a PR if this Dataset API extension is appreciated.*

This request has been sent to the Apache Spark user and [dev|http://apache-spark-developers-list.1001551.n3.nabble.com/Fwd-dataframe-null-safe-joins-given-a-list-of-columns-tt28842.html] mailing list by Marcelo Valle.
Bug: Estimate sizeInBytes from file metadata for parquet files
Status: Open | Priority: Major | Resolution: 
Created: 03/Feb/20 12:37
Affects Version/s: 3.1.0

Currently, Spark will use a compressionFactor when calculating `sizeInBytes` for `HadoopFsRelation`, but this is not accurate and it's hard to choose the best `compressionFactor`. Sometimes, this can causing OOMs due to improper BroadcastHashJoin.

So I propose to use the rowCount in the BlockMetadata to estimate the size in memory, which can be more accurate.
Bug: Reenable to generate Rd files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/20 04:27
Affects Version/s: 2.4.5

In SPARK-30733, due to:

{code}
* creating vignettes ... ERROR
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
package ���htmltools��� was installed by an R version with different internals; it needs to be reinstalled for use with this R version
{code}

It was disable to generate rd files. We should install related packages correctly and reenable it back.
Bug: Remove multiple workers on the same host support from Standalone backend
Status: Open | Priority: Major | Resolution: 
Created: 27/Feb/20 19:25
Affects Version/s: 3.1.0

Based on our experience, there is no scenario that necessarily requires deploying multiple Workers on the same node with Standalone backend. A worker should book all the resources reserved to Spark on the host it is launched, then it can allocate those resources to one or more executors launched by this worker. Since each executor runs in a separated JVM, we can limit the memory of each executor to avoid long GC pause.

The remaining concern is the local-cluster mode is implemented by launching multiple workers on the local host, we might need to re-implement LocalSparkCluster to launch only one Worker and multiple executors. It should be fine because local-cluster mode is only used in running Spark unit test cases, thus end users should not be affected by this change.

Removing multiple workers on the same host support could simplify the deploy model of Standalone backend, and also reduce the burden to support legacy deploy pattern in the future feature developments.

The proposal is to update the document to deprecate the support of system environment `SPARK_WORKER_INSTANCES` in 3.0, and remove the support in the next major version (3.1.0).
Bug: Refine baes class abstraction code style
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/20 15:21
Affects Version/s: 3.1.0

When doing base operator abstraction work, I found there are still some code snippet is  inconsistent with other abstraction code style.
 
Case 1, override keyword missed for some fields in derived classes. The compiler will not capture it if we rename some fields in the future.
[https://github.com/apache/spark/pull/27368#discussion_r376694045]
 
 
Case 2, inconsistent abstract class definition. The updated style will simplify derived class definition.
[https://github.com/apache/spark/pull/27368#discussion_r375061952]
Bug: SQL test running on Windows: hadoop chgrp warnings
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 01/Feb/20 08:34
Affects Version/s: 2.4.4

Running SparkSQL local embedded unit tests on Win10, using winutils.

Got warnings about 'hadoop chgrp'.

See environment info.
{code:bash}
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
{code}

Related info on SO: https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive
Seems like the problem is here: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210
Bug: SPARK 2.4.4 - DROP TABLE and drop HDFS
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Feb/20 11:34
Affects Version/s: 2.4.4

Hi all,

I need to DROP a hive table and clear data in HDFS. But I can't convert it from External to Managed (I have to create external table for my business).

On Spark 2.0.2, I can use script ALTER TABLE <tbl_name> SET TBLPROPERTIES ('EXTERNAL' = 'FALSE'); then I execute script DROP TABLE <tbl_name>; metadata and data was deleted together.

But on spark 2.4.4, it's not work. I need a solution for that (drop a external table and drop data in hdfs or convert a external table to managed table and drop it).

Many thanks.
Bug: PySpark Row can be instantiated with duplicate field names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 16:38
Affects Version/s: 1.6.3

It is possible to create a Row that has fields with the same name when calling `collect()` after a join. Given that the Row constructor itself doesn't allow this, this seems to be undesired behavior.

This can possibly cause correctness issues because different ways of getting values produce different results: {{__get_item__}} will return the leftmost value, while {{asDict()}} will return the rightmost value (because the former uses an index search and the latter uses a dictionary generator).

{{>>> manual_output_row = Row(a=1, b=1, b=2)}}
 \{{ File "<stdin>", line 1}}
 {{SyntaxError: keyword argument repeated}}

{{>>> input_rows = Row(a=1, b=1), Row(a=1, b=2)}}
 {{>>> df1, df2 = (spark.createDataFrame([r]) for r in input_rows)}}
 {{>>> df3 = df1.join(df2, "a")}}
 {{>>> output_row = df3.collect()[0]}}
 {{>>> output_row}}
 {{Row(a=1, b=1, b=2)}}
 {{>>> output_row["b"]}}
 {{1}}
 {{>>> output_row.asDict()["b"]}}
 {{2}} 

**SPARK 1.6.3**
{code}
>>> from pyspark.sql.types import Row
>>> input_rows = Row(a=1, b=1), Row(a=1, b=2)
>>> df1, df2 = (sqlContext.createDataFrame([r]) for r in input_rows)
>>> df3 = df1.join(df2, "a")
>>> output_row = df3.collect()[0]
>>> output_row
Row(a=1, b=1, b=2)
>>> output_row["b"]
1
>>> output_row.asDict()["b"]
2
>>> sc.version
u'1.6.3'
{code}
Bug: Add version information to the configuration of Deploy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/20 07:46
Affects Version/s: 3.0.0

[*spark*|https://github.com/apache/spark]/[core|https://github.com/apache/spark/tree/master/core]/[src|https://github.com/apache/spark/tree/master/core/src]/[main|https://github.com/apache/spark/tree/master/core/src/main]/[scala|https://github.com/apache/spark/tree/master/core/src/main/scala]/[org|https://github.com/apache/spark/tree/master/core/src/main/scala/org]/[apache|https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache]/[spark|https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark]/[internal|https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark/internal]/[config|https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark/internal/config]/*Deploy.scala*
Bug: StreamingQueryManager should avoid keeping reference to terminated StreamingQuery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/20 21:49
Affects Version/s: 3.0.0

Right now StreamingQueryManager will keep the last terminated query until "resetTerminated" is called. When the last terminated query has lots of states (a large sql plan, cached RDDs, etc.), it will waste these memory unnecessarily.
Bug: Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 04:39
Affects Version/s: 3.1.0

"compact" operation in FileStreamSourceLog and FileStreamSinkLog is introduced to solve "small files" problem, but introduced non-trivial latency which is another headache in long run query.

There're bunch of reports from community for the same issue (see SPARK-24295, SPARK-29995, SPARK-30462) - before trying to solve the problem, it would be better to measure the latency (elapsed time) and log to help indicating the issue when the additional latency becomes concerns.
Bug: Revise the doc of spark.ui.retainedTasks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/20 00:58
Affects Version/s: 3.0.0

There are configurations for the limitation of UI data.  
`spark.ui.retainedJobs`, `spark.ui.retainedStages` and `spark.worker.ui.retainedExecutors` are the total max number for one application, while the configuration `spark.ui.retainedTasks` is the max number for one stage.

We should revise the documentation to make it clear.
Bug: SQLContext retains reference to unusable instance after SparkContext restarted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/20 16:26
Affects Version/s: 2.4.5

When the underlying SQLContext is instantiated for a SparkSession, the instance is saved as a class attribute and returned from subsequent calls to SQLContext.getOrCreate(). If the SparkContext is stopped and a new one started, the SQLContext class attribute is never cleared so any code which calls SQLContext.getOrCreate() will get a SQLContext with a reference to the old, unusable SparkContext.

A similar issue was identified and fixed for SparkSession in SPARK-19055, but the fix did not change SQLContext as well. I ran into this because mllib still [uses|https://github.com/apache/spark/blob/master/python/pyspark/mllib/common.py#L105] SQLContext.getOrCreate() under the hood.

I've already written a fix for this, which I'll be sharing in a PR, that clears the class attribute on SQLContext when the SparkSession is stopped. Another option would be to deprecate SQLContext.getOrCreate() entirely since the corresponding Scala [method|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SQLContext.html#getOrCreate-org.apache.spark.SparkContext-] is itself deprecated. That seems like a larger change for a relatively minor issue, however.
Bug: spark-master-test-k8s is broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/20 21:53
Affects Version/s: 3.1.0

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20K8s%20Builds/job/spark-master-test-k8s/471/console

{code}
+ /home/jenkins/bin/session_lock_resource.py minikube
  File "/home/jenkins/bin/session_lock_resource.py", line 140
    child_body_func = lambda(success_callback): _lock_and_wait(
                            ^
SyntaxError: invalid syntax
{code}
Bug: Allows to parse a Dataset having different column from 'value' in csv(dataset) API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 11:20
Affects Version/s: 3.0.0

{code}
val ds = spark.range(10).selectExpr("'a, b, c' AS text").as[String]
spark.read.csv(ds).show()
{code}

{code}
org.apache.spark.sql.AnalysisException: cannot resolve '`value`' given input columns: [text];;
'Filter (length(trim('value, None)) > 0)
+- Project [a, b, c AS text#2]
   +- Range (0, 10, step=1, splits=Some(2))
{code}

It fails to create a CSV parsed DataFrame from a String Dataset.
Bug: Support more than 5 typed column in typed Dataset.select API
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Feb/20 07:31
Affects Version/s: 3.1.0

Because Dataset only provides overloading typed select API to at most 5 typed columns, once more than 5 typed columns given, the select API call will go for untyped one.

Currently users cannot call typed select with more than 5 typed columns. There are few options:

1. Expose Dataset.selectUntyped (could rename it) to accept any number (due to the limit of ExpressionEncoder.tuple, at most 22 actually) of typed columns. Pros: not need to add too much code in Dataset. Cons: The returned type is generally Dataset[_], not a specified one like Dataset[(U1, U2)] for the overloading method.

2. Add more overloading typed select APIs up to 22 typed column inputs. Pros: Clear returned type. Cons: A lot of code to be added to Dataset for just corner cases. It can be a breaking change to existing user code that calls untyped select API.
Bug: Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 09:47
Affects Version/s: 3.0.0

To be absolutely sure that Spark 3.0 is compatible with 2.4 when spark.sql.legacy.timeParser.enabled is set to true, need to support SimpleDateFormat and FastDateFormat as legacy parsers/formatters in TimestampFormatter. 

Spark 2.4.x uses the following parsers for parsing/formatting date/timestamp strings:

# DateTimeFormat in CSV/JSON datasource
# SimpleDateFormat - is used in JDBC datasource, in partitions parsing.
# SimpleDateFormat in strong mode (lenient = false). It is used by the date_format, from_unixtime, unix_timestamp and to_unix_timestamp functions.

Spark 3.0 should use the same parsers in those cases.
Bug: Scope Session.active in QueryExecution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 12:51
Affects Version/s: 3.0.0

SparkSession.active is a thread local variable that points to the current thread's spark session. It is important to note that the SQLConf.get method depends on SparkSession.active. In the current implementation it is possible that SparkSession.active points to a different session which causes various problems. Most of these problems arise because part of the query processing is done using the configurations of a different session. For example, when creating a data frame using a new session, i.e., session.sql("..."), part of the data frame is constructed using the currently active spark session, which can be a different session from the one used later for processing the query.

This PR scopes SparkSession.active to prevent the above-mentioned problems. A new method, withActive is introduced on SparkSession that restores the previous spark session after the block of code is executed.
Bug: Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 11:13
Affects Version/s: 3.2.0

All of LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE should support IGNORE NULLS | RESPECT NULLS. For example:
{code:java}
LEAD (value_expr [, offset ])
[ IGNORE NULLS | RESPECT NULLS ]
OVER ( [ PARTITION BY window_partition ] ORDER BY window_ordering ){code}
 
{code:java}
LAG (value_expr [, offset ])
[ IGNORE NULLS | RESPECT NULLS ]
OVER ( [ PARTITION BY window_partition ] ORDER BY window_ordering ){code}
 
{code:java}
NTH_VALUE (expr, offset)
[ IGNORE NULLS | RESPECT NULLS ]
OVER
( [ PARTITION BY window_partition ]
[ ORDER BY window_ordering 
 frame_clause ] ){code}
 

*Oracle:*
[https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/NTH_VALUE.html#GUID-F8A0E88C-67E5-4AA6-9515-95D03A7F9EA0]

*Redshift*
[https://docs.aws.amazon.com/redshift/latest/dg/r_WF_NTH.html]

*Presto*
[https://prestodb.io/docs/current/functions/window.html]

*DB2*
[https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.sqls.doc/ids_sqs_1513.htm]

*Teradata*
[https://docs.teradata.com/r/756LNiPSFdY~4JcCCcR5Cw/GjCT6l7trjkIEjt~7Dhx4w]

*Snowflake*
[https://docs.snowflake.com/en/sql-reference/functions/lead.html]
[https://docs.snowflake.com/en/sql-reference/functions/lag.html]
Bug: RPC listener wildcard bindings (IPv6)
Status: Open | Priority: Major | Resolution: 
Created: 17/Feb/20 10:03
Affects Version/s: 3.1.0

Running Spark Standalone on IPv6-first infrastructure requires that RPC listeners bind IPv6 sockets e.g. {{:::7077}} instead of {{127.0.0.1:7077}} whilst advertising a public DNS name using {{SPARK_PUBLIC_DNS}}. Currently, the various RPC listeners can be configured with a *hostname/IPv4 address* (enforced via {{Utils.checkHost}}) only e.g. via {{spark.driver.bindAddress}}, {{spark.driver.host}}, {{SPARK_LOCAL_HOST}} e.t.c. which are eventually resolved using {{InetSocketAddress}} in [{{TransportServer}}|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L136]

Initial experimentation led to a *hack* (absolutely not a real solution) to force this behaviour by setting {{bindAddress = null}} when the different `RPCEnv`s are created ([link to diff|https://gist.github.com/SteadBytes/b206335c8e2429486a13421b7cfcdc88]).

Since {{TransportServer}} does support the wildcard variant of {{InetSocketAddress}} if {{hostToBind = null}} [here|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L137] this should be possible.

Adding additional options similar to {{spark.driver.bindAddress}} but for other RPC components (master, driver etc) and allowing these to have wildcard/IPv6 values might be an option that avoids breaking existing configurations?

I'm prepared to contribute the work required and upstream any changes.