av9ash/GitBugs huggingface dataset:
Bug: pyspark shell IPython launch throws ".../pyspark/bin/load-spark-env.sh: No such file or directory"
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 18/Apr/20 22:13
Affects Version/s: 2.4.5

I'm trying launching pyspark shell with IPython interface via

{{PYSPARK_DRIVER_PYTHON=ipython pyspark}}

However it hits ".../pyspark/bin/load-spark-env.sh: No such file or directory"

{{(py3-spark) mengyu@mengyu-C02Z7885LVDQ:~/workspace/tmp$ PYSPARK_DRIVER_PYTHON=ipython pyspark}}
{{/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 24: /Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh: No such file or directory}}
{{/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 77: /Users/mengyu/workspace/tmp//Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/spark-submit: No such file or directory}}
{{/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 77: exec: /Users/mengyu/workspace/tmp//Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/spark-submit: cannot execute: No such file or directory}}

 

It is strange because the path "{{/Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh}}" exists.

 

{{$ file /Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh}}{{/Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh: Bourne-Again shell script text executable, ASCII text}}
Bug: The query with the where condition failed,when the partition field is null
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 07/Apr/20 03:38
Affects Version/s: 2.4.4

h3. The problem recurs as follows：
 # create table test_1(id int,name string) partitioned by(profile string)
 # insert into test_1 values(1,null)
 # select * from test_1 where profile is null

Go through the above steps，the result is nothing.But if add the condition profile='__HIVE_DEFAULT_PARTITION__',the result is OK.
h3. The temporary solution:

select * from test_1 where profile is null or profile='__HIVE_DEFAULT_PARTITION__'

The result is OK
h3. Special instructions：

1、The above phenomenon，Only the partition filed type is string can happen

2、The above operation in hive is no problem
h3. Problem orientation：

As far as I'm consider the problem is in org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils and org.apache.spark.sql.catalyst.catalog.CatalogTablePartition.Especially the toRow function in CatalogTablePartition.
Bug: Control a plan explain mode in the events of SQL listeners via SQLConf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/20 04:38
Affects Version/s: 3.1.0

This proposes to add a new SQL config for controlling a plan explain mode in the events of (e.g., `SparkListenerSQLExecutionStart` and `SparkListenerSQLAdaptiveExecutionUpdate`) SQL listeners.

In the current master, the output of `QueryExecution.toString` (this is equivalent to the "extended" explain mode) is stored in these events. I think it is useful to control the content via SQLConf.
Bug: Improve the EXPLAIN FORMATTED's output for DSV2's Scan Node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/20 18:28
Affects Version/s: 3.0.0

Below is the EXPLAIN OUTPUT when using the *DSV2* 

*Output of EXPLAIN EXTENDED* 
{code:java}
+- BatchScan[col.dots#39L] JsonScan DataFilters: [isnotnull(col.dots#39L), (col.dots#39L = 500)], Location: InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-7dad6f63-dc..., PartitionFilters: [], ReadSchema: struct<col.dots:bigint>
{code}
*Output of EXPLAIN FORMATTED* 
{code:java}
 (1) BatchScan
Output [1]: [col.dots#39L]
Arguments: [col.dots#39L], JsonScan(org.apache.spark.sql.test.TestSparkSession@45eab322,org.apache.spark.sql.execution.datasources.InMemoryFileIndex@72065f16,StructType(StructField(col.dots,LongType,true)),StructType(StructField(col.dots,LongType,true)),StructType(),org.apache.spark.sql.util.CaseInsensitiveStringMap@8822c5e0,Vector(),List(isnotnull(col.dots#39L), (col.dots#39L = 500)))
{code}
When using *DSV1*, the output is much cleaner than the output of DSV2, especially for EXPLAIN FORMATTED.

*Output of EXPLAIN EXTENDED* 
{code:java}
+- FileScan json [col.dots#37L] Batched: false, DataFilters: [isnotnull(col.dots#37L), (col.dots#37L = 500)], Format: JSON, Location: InMemoryFileIndex[file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-59..., PartitionFilters: [], PushedFilters: [IsNotNull(`col.dots`), EqualTo(`col.dots`,500)], ReadSchema: struct<col.dots:bigint> 
{code}
*Output of EXPLAIN FORMATTED* 
{code:java}
 (1) Scan json 
Output [1]: [col.dots#37L]
Batched: false
Location: InMemoryFileIndex [file:/private/var/folders/nr/j6hw4kr51wv0zynvr6srwgr00000gp/T/spark-89021d76-5971-4a96-bf10-0730873f6ce0]
PushedFilters: [IsNotNull(`col.dots`), EqualTo(`col.dots`,500)]
ReadSchema: struct<col.dots:bigint>{code}
Bug: LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 16:41
Affects Version/s: 3.0.0

As per discussion with [~maxgekk]:

{{LegacySimpleTimestampFormatter#parse}} misinterprets pre-Gregorian timestamps:
{noformat}
scala> sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
res0: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> val df1 = Seq("0002-01-01 00:00:00", "1000-01-01 00:00:00", "1800-01-01 00:00:00").toDF("expected")
df1: org.apache.spark.sql.DataFrame = [expected: string]

scala> val df2 = df1.select('expected, to_timestamp('expected, "yyyy-MM-dd HH:mm:ss").as("actual"))
df2: org.apache.spark.sql.DataFrame = [expected: string, actual: timestamp]

scala> df2.show(truncate=false)
+-------------------+-------------------+
|expected           |actual             |
+-------------------+-------------------+
|0002-01-01 00:00:00|0001-12-30 00:00:00|
|1000-01-01 00:00:00|1000-01-06 00:00:00|
|1800-01-01 00:00:00|1800-01-01 00:00:00|
+-------------------+-------------------+


scala> 
{noformat}
Legacy timestamp parsing with JSON and CSV files is correct, so apparently {{LegacyFastTimestampFormatter}} does not have this issue (need to double check).
Bug: Check more strictly that a field name can be used as a valid Java identifier for codegen
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 18:14
Affects Version/s: 3.0.0

ScalaReflection checks that a field name can be used as a valid Java identifier by checking whether the field name is not a reserved keyword.

But, in the current implementation, enum is missed.
Further, some characters including numeric literals are not used as valid identifiers but not checked.
Bug: New Adaptive Query Execution in Spark SQL
Status: Closed | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 08:53
Affects Version/s: 3.0.0

SPARK-9850 proposed the basic idea of adaptive execution in Spark. In DAGScheduler, a new API is added to support submitting a single map stage.  The current implementation of adaptive execution in Spark SQL supports changing the reducer number at runtime. An Exchange coordinator is used to determine the number of post-shuffle partitions for a stage that needs to fetch shuffle data from one or multiple stages. The current implementation adds ExchangeCoordinator while we are adding Exchanges. However there are some limitations. First, it may cause additional shuffles that may decrease the performance. We can see this from EnsureRequirements rule when it adds ExchangeCoordinator.  Secondly, it is not a good idea to add ExchangeCoordinators while we are adding Exchanges because we don’t have a global picture of all shuffle dependencies of a post-shuffle stage. I.e. for 3 tables’ join in a single stage, the same ExchangeCoordinator should be used in three Exchanges but currently two separated ExchangeCoordinator will be added. Thirdly, with the current framework it is not easy to implement other features in adaptive execution flexibly like changing the execution plan and handling skewed join at runtime.

We'd like to introduce a new way to do adaptive execution in Spark SQL and address the limitations. The idea is described at [https://docs.google.com/document/d/1mpVjvQZRAkD-Ggy6-hcjXtBPiQoVbZGe3dLnAKgtJ4k/edit?usp=sharing]
Bug: Make ExpressionEncoder thread safe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/20 08:36
Affects Version/s: 3.0.0

ExpressionEncoder is currently not thread-safe because it contains stateful objects that are required for converting objects to internal rows and vise versa. We have been working around this by (excessively) cloning ExpressionEncoders which is not free. I propose that we move the stateful bits of the expression encoder into two helper classes that will take care of the conversions.
Bug: Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Apr/20 17:26
Affects Version/s: 3.0.0

Some users may not know they are creating and/or reading DATE or TIMESTAMP data from before October 15, 1582 (because of data encoding libraries, etc.). Therefore, it may not be clear whether they need to toggle the two rebaseDateTime config settings.

By default, Spark should fail if it reads or writes data from October 15, 1582 or before.
Bug: Improve the structure of auto-generated built-in function pages in SQL references
Status: Open | Priority: Major | Resolution: 
Created: 22/Apr/20 00:04
Affects Version/s: 3.1.0

We now put examples per group in one section in the builtin function documents like this (See: SPARK-31429);
{code}
### Aggrgate Functions
<A list of aggregate functions>
#### Examples
<All examples of the aggregate functions listed above>

### Window Functions
<A list of aggregate functions>
#### Examples
<All examples of the window functions listed above>
...
{code}
But, it might be better to put everything about one function (signature, desc, examples, etc.). So, we need more discussions to improve the structure in the page.
Bug: Bug in the approximate quantile computation.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 13/Apr/20 05:02
Affects Version/s: 3.0.0

I am seeing a bug where passing lower relative error to the {{approxQuantile}} function is leading to incorrect result in the presence of partitions. Setting a relative error 1e-6 causes it to compute equal values for 0.9 and 1.0 quantiles. Coalescing it back to 1 partition gives correct results. This issue was not present in spark version 2.4.5, we noticed it when testing 3.0.0-preview.

{{>>> df = spark.read.csv('file:///tmp/approx_quantile_data.csv', header=True, schema=T.StructType([T.StructField('Store',T.StringType(),True),T.StructField('seconds',T.LongType(),True)]))}}
{{>>> df = df.repartition(200, 'Store').localCheckpoint()}}
{{>>> df.approxQuantile('seconds', [0.8, 0.9, 1.0], 0.0001)}}
{{[1422576000.0, 1430352000.0, 1438300800.0]}}
{{>>> df.approxQuantile('seconds', [0.8, 0.9, 1.0], 0.00001)}}
{{[1422576000.0, 1430524800.0, 1438300800.0]}}
{color:#de350b}{{>>> df.approxQuantile('seconds', [0.8, 0.9, 1.0], 0.000001)}}{color}
{color:#de350b}{{[1422576000.0, 1438300800.0, 1438300800.0]}}{color}
{{>>> df.coalesce(1).approxQuantile('seconds', [0.8, 0.9, 1.0], 0.000001)}}
{{[1422576000.0, 1430524800.0, 1438300800.0]}}
Bug: Support for Kubernetes NFS volume mounts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/20 08:11
Affects Version/s: 3.1.0

Kubernetes supports various kinds of volumes, but Spark for Kubernetes supports only EmptyDir/HostDir/PVC. By adding support for NFS, we can use Spark for Kubernetes with NFS storage.

NFS could be used using PVC when we want to use some clean new empty disk space, but in order to use files in existing NFS shares, we need to use NFS volume mounts.

RP link: [https://github.com/apache/spark/pull/27364]
Bug: Add a hybrid KVStore to make UI loading faster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/20 17:33
Affects Version/s: 3.0.1

This is a follow-up for the work done by Hieu Huynh in 2019.

Add a new class HybridKVStore to make the history server faster when loading event files. When rebuilding the application state from event logs, HybridKVStore will first write data to an in-memory store and having a background thread that keeps pushing the change to levelDB.

I ran some tests on 3.0.1 on mac os:
||kvstore type / log size||100m||200m||500m||1g||2g||
|HybridKVStore|5s to parse, 7s(include the parsing time) to switch to leveldb|6s to parse, 10s to switch to leveldb|15s to parse, 23s to switch to leveldb|23s to parse, 40s to switch to leveldb|37s to parse, 73s to switch to leveldb|
|LevelDB|12s to parse|19s to parse|43s to parse|69s to parse|124s to parse|
For example when loading a 1g file, HybridKVStore takes 23s to parse (that means, users only need to wait for 23s to see the UI), the background thread will still run 17s to copy data to leveldb. And after that, the in memory store can be closed, the entire store now moves to leveldb. So in general, it has 3x - 4x UI loading speed improvement.
Bug: Improve SQL Rest API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/20 20:07
Affects Version/s: 3.1.0

SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI.

*Proposed Improvements:*
1- Support Physical Operations and group metrics per physical operation by aligning Spark UI.
2- Support *wholeStageCodegenId* for Physical Operations
3- *nodeId* can be useful for grouping metrics and sorting physical operations (according to execution order) to differentiate same operators (if used multiple times during the same query execution) and their metrics.
4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, Spark UI does not show empty metrics.
5- Remove line breakers(*\n*) from *metricValue*.
6- *planDescription* can be *optional* Http parameter to avoid network cost where there is specially complex jobs creating big-plans.
7- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. Specially, this can be useful for the user where *metricDetails* array size is high. 
8- Reverse order on *metricDetails* aims to match with Spark UI by supporting Physical Operators' execution order.

*Attachments:*
 Please find both *current* and *improved* versions of the results as attached for following SQL Rest Endpoint:
{code:java}
curl -X GET http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code}
Bug: Introduce SORTED BY clause in CREATE TABLE statement
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/20 08:53
Affects Version/s: 3.0.0

We usually sort on frequently filtered columns when writing data to improve query performance. But there is no these info in the table information.
 

{code:sql}
CREATE TABLE t(day INT, hour INT, year INT, month INT)
USING parquet
PARTITIONED BY (year, month)
SORTED BY (day, hour);
{code}

 

Impala, Oracle and redshift support this clause:
https://issues.apache.org/jira/browse/IMPALA-4166
https://docs.oracle.com/database/121/DWHSG/attcluster.htm#GUID-DAECFBC5-FD1A-45A5-8C2C-DC9884D0857B
https://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data-compare-sort-styles.html
Bug: Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 20:16
Affects Version/s: 2.3.0

With Spark blacklisting, if a task fails on an executor, the executor gets blacklisted for the task. In order to retry the task, it checks if there are idle blacklisted executor which can be killed and replaced to retry the task if not it aborts the job without doing max retries.

In the context of dynamic allocation this can be better, instead of killing the blacklisted idle executor (its possible there are no idle blacklisted executor), request an additional executor and retry the task.

This can be easily reproduced with a simple job like below, although this example should fail eventually just to show that its not retried spark.task.maxFailures times: 

{code:java}
def test(a: Int) = { a.asInstanceOf[String] }
sc.parallelize(1 to 10, 10).map(x => test(x)).collect 
{code}

with dynamic allocation enabled and min executors set to 1. But there are various other cases where this can fail as well.
Bug: Support `java.time.LocalDate` in Parquet filter pushdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/20 15:56
Affects Version/s: 3.0.0

Currently, ParquetFilters supports only java.sql.Date values of DateType, and explicitly casts Any to java.sql.Date, see
https://github.com/apache/spark/blob/cb0db213736de5c5c02b09a2d5c3e17254708ce1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L176

So, any filters refer to date values are not pushed down to Parquet when spark.sql.datetime.java8API.enabled is true.
Bug: reduceGroupValues function for KeyValueGroupedDataset
Status: Open | Priority: Major | Resolution: 
Created: 05/Apr/20 20:33
Affects Version/s: 3.0.0

Problem: in Datasets API, it is a very common pattern to do something like this whenever a complex reduce function is needed:

{code:scala}
ds
  .groupByKey(_.y)
  .reduceGroups((a, b) => out)
  .map(_._2)
{code}

However, the the optimized plan unfortunately ends up with an unnecessary implicit serialization during aggregation step, followed by {{DeserializeToObject + MapElements (from (K, V) => V)}}. For this common use case, there are 2 things we can improve with a specialized {{.reduceGroupValues}}:

1. avoid the extra serialization (baked in to AggregationIterator implementations) and deserialization
2. avoid requiring a {{.map}} and the time it takes to unpack tuples by emitting the values only

Proposal:

{code:scala}
ds
  .groupByKey(_.y)
  .reduceGroupValues((a, b) => out)
{code}
Create an {{AggregationIteratorBase}} superclass that can emit general {{InternalRow}} s instead of just {{UnsafeRow}} s.
Create a new {{AggregationIteratorBase}} implementation called {{ObjectValuesAggregationIterator}} that emits {{InternalRow}} s containing only the values instead of serializing them into {{UnsafeRow}} s on {{Final}} or {{Complete}} modes. Since we don't need to emit the keys, which are serialized, this is not too complicated. To make use of this, have the {{.reduceGroupValues}} return an {{Aggregate}} wrapped in a {{CatalystSerde.serialize}}.
Bug: Coalesce bucketed tables for join if applicable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/20 23:28
Affects Version/s: 3.1.0

The following example of joining two bucketed tables introduces a full shuffle:
{code:java}
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "0")
val df1 = (0 until 20).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k")
val df2 = (0 until 20).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k")
df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
val t1 = spark.table("t1")
val t2 = spark.table("t2")
val joined = t1.join(t2, t1("i") === t2("i"))
joined.explain(true)


== Physical Plan ==
*(5) SortMergeJoin [i#44], [i#50], Inner
:- *(2) Sort [i#44 ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(i#44, 200), true, [id=#105]
:     +- *(1) Project [i#44, j#45, k#46]
:        +- *(1) Filter isnotnull(i#44)
:           +- *(1) ColumnarToRow
:              +- FileScan parquet default.t1[i#44,j#45,k#46] Batched: true, DataFilters: [isnotnull(i#44)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 8 out of 8
+- *(4) Sort [i#50 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(i#50, 200), true, [id=#115]
      +- *(3) Project [i#50, j#51, k#52]
         +- *(3) Filter isnotnull(i#50)
            +- *(3) ColumnarToRow
               +- FileScan parquet default.t2[i#50,j#51,k#52] Batched: true, DataFilters: [isnotnull(i#50)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 4 out of 4
{code}
But one side can be coalesced to eliminate the shuffle.
Bug: bufferPoolsBySize in HeapMemoryAllocator should be thread safe
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 28/Apr/20 08:55
Affects Version/s: 2.4.3

Currently, bufferPoolsBySize in HeapMemoryAllocator uses a Map type whose value type is LinkedList.

LinkedList is not thread safe and may hit the error below
{code:java}
java.util.NoSuchElementExceptionException
    at java.util.LinkedList.removeFirst(LinkedList.java:270) 
    at java.util.LinkedList.remove(LinkedList.java:685)
    at org.apache.spark.unsafe.memory.HeapMemoryAllocator.allocate(HeapMemoryAllocator.java:57){code}
Bug: Update ExpressionDescription for substring, current_date, and current_timestamp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/20 08:14
Affects Version/s: 3.0.0

This jira intends to add entries for substring, current_date, and current_timestamp in the SQL built-in function documents. Specifically, the entries are as follows;

SELECT current_date;
SELECT current_timestamp;
SELECT substring('abcd' FROM 1);
SELECT substring('abcd' FROM 1 FOR 2);
Bug: SparkSession Lifecycle methods to fix memory leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/20 14:36
Affects Version/s: 3.0.0

Follow up to https://issues.apache.org/jira/browse/SPARK-27958 after discussion on [https://github.com/apache/spark/pull/24807]. 

 

Let's instead expose methods that allow the user to manually clean up (terminate) a SparkSession, that also remove the listenerState from the context.
Bug: Build Spark’s own datetime pattern definition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 07:40
Affects Version/s: 3.0.0

This is an umbrella ticket for building Spark's own Datetime patterns and related works.

In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.

However, there are some datetime patterns not compatible between Java 8 and Java 7 APIs, and it's fragile to rely on the JDK API to define Spark's behavior. We should build our own Datetime patterns, which is compatible with Spark 2.4 (the old Java 7 `SimpleDateFormat` API).