av9ash/GitBugs huggingface dataset:
Bug: repartitionAndSortWithinPartitions avoid shuffle with same partitioner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jul/20 07:16
Affects Version/s: 3.1.0

In {{combineByKeyWithClassTag}}, there is a check so that if the partitioner is the same as the one of the RDD:
{code:java}
if (self.partitioner == Some(partitioner)) {
  self.mapPartitions(iter => {
    val context = TaskContext.get()
    new InterruptibleIterator(context, aggregator.combineValuesByKey(iter, context))
  }, preservesPartitioning = true)
} else {
  new ShuffledRDD[K, V, C](self, partitioner)
    .setSerializer(serializer)
    .setAggregator(aggregator)
    .setMapSideCombine(mapSideCombine)
}
 {code}
 

In {{repartitionAndSortWithinPartitions}}, this shuffle can also be skipped in this case.
Bug: Support full outer join in shuffled hash join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jul/20 21:22
Affects Version/s: 3.1.0

Currently for SQL full outer join, spark always does a sort merge join no matter of how large the join children size are. Inspired by recent discussion in [https://github.com/apache/spark/pull/29130#discussion_r456502678] and [https://github.com/apache/spark/pull/29181], I think we can support full outer join in shuffled hash join in a way that - when looking up stream side keys from build side {{HashedRelation}}. Mark this info inside build side {{HashedRelation}}, and after reading all rows from stream side, output all non-matching rows from build side based on modified {{HashedRelation}}.
Bug: Use python3 by default in pyspark and find-spark-home scripts
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jul/20 19:59
Affects Version/s: 3.1.0

This script depends on `find_spark_home.py` which is migrated to `python3` already by using `#!/usr/bin/env python3`.
{code}
FIND_SPARK_HOME_PYTHON_SCRIPT="$(cd "$(dirname "$0")"; pwd)/find_spark_home.py"
{code}
Bug: LogisticRegressionModel prediction optimization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/20 08:58
Affects Version/s: 3.1.0

if needed, method getThreshold and/or following logic to compute rawThreshold is called on each instance.

 
{code:java}
override def getThreshold: Double = {
  checkThresholdConsistency()
  if (isSet(thresholds)) {
    val ts = $(thresholds)
    require(ts.length == 2, "Logistic Regression getThreshold only applies to" +
      " binary classification, but thresholds has length != 2.  thresholds: " + ts.mkString(","))
    1.0 / (1.0 + ts(0) / ts(1))
  } else {
    $(threshold)
  }
} {code}
 
{code:java}
      val rawThreshold = if (t == 0.0) {
            Double.NegativeInfinity
          } else if (t == 1.0) {
            Double.PositiveInfinity
          } else {
            math.log(t / (1.0 - t))
          } {code}
Bug: Kubernetes Configuration to set Service Account to Executors
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 08/Jul/20 21:35
Affects Version/s: 3.0.0

Some cloud providers use Service Accounts to provide resource authorization (one example is described here [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/)] and for this we need to be able to set Service Accounts to the executors.

My idea for development of this feature would be to have a configuration like "spark.kubernetes.authenticate.executor.serviceAccountName" in order to set the executors Service Account, this way it could be possible to allow only certain accesses to the driver and others to the executors or the same access (user's choice).

I am creating this issue so the maintainers can write opinions first, but I intend to create a pull request to address this issue also.
Bug: Document the dependency between spark.metrics.staticSources.enabled and JVMSource registration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jul/20 11:17
Affects Version/s: 3.0.0

SPARK-29654 has introduced the configuration `spark.metrics.register.static.sources.enabled`.
 The current implementation of SPARK-29654 and SPARK-25277 have, as side effect, that when {{spark.metrics.register.static.sources.enabled}} is set to false, the registration of JVM Source is also ignored, even if requested with {{spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource}}
 A PR is proposed to fix this.
Bug: Reset previous search text for datatable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/20 23:27
Affects Version/s: 3.1.0

DataTable is used in stage-page and executors-page for pagination and filter tasks/executors by search text.

In the current implementation, search text is saved so if we visit stage-page for a job, the previous search text is filled in the textbox and the task table is filtered.

I'm sometimes surprised by this behavior as the stage-page lists no tasks because tasks are filtered by the previous search text.
Bug: RecordBinaryComparatorSuite test failures on big-endian systems
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jul/20 14:05
Affects Version/s: 3.0.0

The fix for SPARK-29918 broke two tests on big-endian systems:
 * testBinaryComparatorWhenSubtractionIsDivisibleByMaxIntValue
 * testBinaryComparatorWhenSubtractionCanOverflowLongValue

These tests date from a time where subtraction was being used to do multi-byte comparisons. They try to trigger old bugs by feeding specific values into the comparison. However the fix for SPARK-29918 modified the order in which bytes are compared when comparing 8 bytes at a time on little-endian systems (to match the normal byte-by-byte comparison). This fix did not affect big-endian systems. However the expected output of the tests was modified for all systems regardless of endianness. So the tests broke on big-endian systems.

It is also not clear that the values compared in the tests match the original intent of the tests now that the bytes in those values are compared in order (equivalent to the bytes in the values being reversed).
Bug: Fix timeout config issue in Kafka connector tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jul/20 15:07
Affects Version/s: 3.1.0

While I'm implementing SPARK-32032 I've found a bug in Kafka: https://issues.apache.org/jira/browse/KAFKA-10318. This will cause issues only later when it's fixed but it would be good to fix it now because SPARK-32032 would like to bring in AdminClient where the code blows up with the mentioned ConfigException. This would reduce the code changes in the mentioned jira.
Bug: Only codegen build side separately for shuffled hash join
Status: Open | Priority: Minor | Resolution: 
Created: 31/Jul/20 06:02
Affects Version/s: 3.1.0

Currently `ShuffledHashJoinExec` needs to do code-gen separately for its both children per reason in [https://github.com/apache/spark/pull/29277#discussion_r463415144] . Creating this Jira as a followup, to only code-gen build side separately, but not stream side. Seeing above discussion for known blockers.
Bug: Support grant/revoke access privileges like postgresql
Status: Open | Priority: Minor | Resolution: 
Created: 30/Jul/20 14:06
Affects Version/s: 3.1.0

Support grant/revoke access privileges like postgresql.

[https://www.postgresql.org/docs/9.0/sql-grant.html]

Currently Spark SQL does not support grant/revoke statement, which might be done only in Hive
Bug: NullAwareAntiJoin multi-column support
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 29/Jul/20 03:59
Affects Version/s: 3.0.0

This is a follow up improvement of Issue SPARK-32290.

In SPARK-32290, we already optimize NAAJ from BroadcastNestedLoopJoin to BroadcastHashJoin, which improve total calculation from O(M*N) to O(M), but it's only targeting on Single Column Case, because it's much more complicate in multi column support.

See. [http://www.vldb.org/pvldb/vol2/vldb09-423.pdf] Section 6

 

FYI, code logical for single and multi column is defined at

~/sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql

~/sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql

 

For supporting multi column, I throw the following idea and see if is it worth to do multi-column support with some trade off. I would need to do some data expansion in HashedRelation, and i would call this new type of HashedRelation as NullAwareHashedRelation.

 

In NullAwareHashedRelation, key with null column is allowed, which is opposite in LongHashedRelation and UnsafeHashedRelation; And single key might be expanded into 2^N - 1 records, (N refer to columnNum of the key). for example, if there is a record

(1 ,2, 3) is about to insert into NullAwareHashedRelation, we take C(1,3), C(2,3) as a combination to copy origin key row, and setNull at target position, and then insert into NullAwareHashedRelation. including the origin key row, there will be 7 key row inserted as follow.

(null, 2, 3)

(1, null, 3)

(1, 2, null)

(null, null, 3)

(null, 2, null)

(1, null, null)

(1, 2, 3)

 

with the expanded data we can extract a common pattern for both single and multi column. allNull refer to a unsafeRow which has all null columns.
 * buildSide is empty input => return all rows
 * allNullColumnKey Exists In buildSide input => reject all rows
 * if streamedSideRow.allNull is true => drop the row
 * if streamedSideRow.allNull is false & findMatch in NullAwareHashedRelation => drop the row
 * if streamedSideRow.allNull is false & notFindMatch in NullAwareHashedRelation => return the row

 

this solution will sure make buildSide data expand to 2^N-1 times, but since it is normally up to 2~3 column in NAAJ in normal production query, i suppose that it's acceptable to expand buildSide data to around 7X. I would also have a limitation of max column support for NAAJ, basically should not more than 3.
Bug: Use @parser::members and @lexer::members to avoid generating unused code
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jul/20 02:32
Affects Version/s: 3.1.0

This ticket aims to update {{SqlBse.g4}} for avoiding generating unused code.
Currently, ANTLR generates unused methods and variables; {{isValidDecimal}} and {{isHint}} are only used in the generated lexer.
Bug: Snapshot artifacts can have differing timestamps, making it hard to consume
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jul/20 20:00
Affects Version/s: 3.0.1

Since we use multiple sub components in building Spark we can get into a situation where the timestamps for these components is different. This can make it difficult to consume Spark snapshots in an environment where someone is running a nightly build for other folks to develop on top of.

I believe I have a small fix for this already, but just waiting to verify and then I'll open a PR.
Bug: Decide SMJ Join Orientation adaptively
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Jul/20 08:03
Affects Version/s: 3.1.0

For SortMerge join of type EquiJoin, the left and right side of the joins are decided on the basis of the user order. In SMJ, the left side of the join is streamed and the right side is buffered (matching values). Because of this, B SMJ A would perform better than A SMJ B if, sizeOf(B) > sizeOf(A)

 

With adaptive query execution, once both ShuffleQueryStages corresponding to the join have completed and if none of them have sizes lesser than the broadcast threshold, join orientation can be changed at run time.
Bug: Multiple Kryo registrators can't be used anymore
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/20 11:26
Affects Version/s: 3.0.0

This is a regression in Spark 3.0 as it is working with Spark 2.

According to the docs, it should be possible to register multiple Kryo registrators via Spark config option spark.kryo.registrator . 

In Spark 3.0 the code to parse Kryo config options has been refactored into Scala class [Kryo|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala]. The code to parse the registrators is in [Line 29-32|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala#L29-L32]

{code:scala}
val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
    .version("0.5.0")
    .stringConf
    .createOptional
{code}
but it should be
{code:scala}
val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
    .version("0.5.0")
    .stringConf
    .toSequence
    .createOptional
{code}
 to split the comma seprated list.

In previous Spark 2.x it was done differently directly in [KryoSerializer Line 77-79|https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala#L77-L79]

{code:scala}
private val userRegistrators = conf.get("spark.kryo.registrator", "")
    .split(',').map(_.trim)
    .filter(!_.isEmpty)
{code}

Hope this helps.
Bug: Mismatched row access sizes in tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/20 10:18
Affects Version/s: 3.0.0

The RowEncoderSuite and UnsafeMapSuite tests fail on big-endian systems. This is because the test data is written into the row using unsafe operations with one size and then read back using a different size. For example, in UnsafeMapSuite the test data is written using putLong and then read back using getInt. This happens to work on little-endian systems but these differences appear to be typos and cause the tests to fail on big-endian systems.

I have a patch that fixes the issue.
Bug: Override table renaming in JDBC dialects
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jul/20 20:29
Affects Version/s: 3.1.0

SPARK-32375 adds new method renameTable to JdbcDialect with the default implementation:
{code:sql}
ALTER TABLE table_name RENAME TO new_table_name;
{code}
which is supported by Oracle, MySQL, MariaDB, PostgreSQL and SQLite but other dialects might not support such syntax, for instance SQL Server (using the stored procedure called sp_rename):
{code:sql}
sp_rename 'table_name', 'new_table_name';
{code}

The ticket aims to support table renaming in all JDBC dialects.
Bug: Add JSONOptions to toJSON
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 02/Jul/20 17:30
Affects Version/s: 3.0.0

Actually when calling `toJSON` on a dataFrame with null values, it doesn't print them.

Basically the same idea than https://issues.apache.org/jira/browse/SPARK-23772.

 
{code:java}
val df = spark.sparkContext.parallelize(Seq("1", "2", null)).toDF("col1")
df.toJSON -> {"col1":"1"},{"col1":"2"},{}{code}
 

After the PR:
{code:java}
val result = df.toJSON(Map("ignoreNullFields" -> "false")).collect().mkString(",")
val expected = """{"col1":"1"},{"col1":"2"},{"col1":null}"""
{code}
[~maropu] [~ueshin]

 

[https://github.com/apache/spark/pull/28984/]
Bug: Override datasource implementation during look up via configuration
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 25/Jul/20 19:11
Affects Version/s: 3.1.0

We need a mechanism to override the datasource implementation via configuration.

For example, suppose I have a custom CSV datasource implementation called "my_csv". One way to use it is:

{code}
 val df = spark.read.format("my_csv").load(...)
{code}
Since the source data is the same format (CSV), you should be able to override the default implementation.

One proposal is to do the following:
{code}
spark.conf.set("spark.sql.datasource.override.csv", "my_csv") val df = spark.read.csv(...)
{code}
This has a benefit that the user doesn't have to change the application code to try out a new datasource implementation for the same source format.
Bug: Spark sequence() fails if start and end of range are identical timestamps
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 24/Jul/20 09:02
Affects Version/s: 3.0.0

The following Spark SQL query throws an exception
{code:java}
select sequence(current_timestamp, current_timestamp)
{code}
The error is:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 1
	at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:78){noformat}
 Essentially a copy of https://issues.apache.org/jira/browse/SPARK-31980#
Bug: Guidance for my project 
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 23/Jul/20 12:55
Affects Version/s: 3.0.0

hi,

I am planning to get-read elasticsearch index continuously, and put that data on Data Frame and group that data, search and create an alert. I like to write my code in python.

For this purpose, what should I use, spark, jupter, pyspark...
Bug: Fix error messages during using PIVOT and lateral view
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 15/Jul/20 17:55
Affects Version/s: 3.0.0

Currently when we use `lateral view` and `pivot` together in from clause, if  `lateral view` is before `pivot`, the error message is "LATERAL cannot be used together with PIVOT in FROM clause".if if  `lateral view` is after `pivot`,the query will be normal ,So the error messages "LATERAL cannot be used together with PIVOT in FROM clause" is not accurate, we may improve it.

 

Steps to reproduce:
{code:java}
CREATE TABLE person (id INT, name STRING, age INT, class int, address STRING);
 INSERT INTO person VALUES
 (100, 'John', 30, 1, 'Street 1'),
 (200, 'Mary', NULL, 1, 'Street 2'),
 (300, 'Mike', 80, 3, 'Street 3'),
 (400, 'Dan', 50, 4, 'Street 4');
{code}
 

Query1:

 
{code:java}
SELECT * FROM person
 lateral view outer explode(array(30,60)) tabelName as c_age
 lateral view explode(array(40,80)) as d_age
 PIVOT (
 count(distinct age) as a
 for name in ('Mary','John')
 )
{code}
Result 1:

 
{code:java}
Error: org.apache.spark.sql.catalyst.parser.ParseException: 
 LATERAL cannot be used together with PIVOT in FROM clause(line 1, pos 9)
== SQL ==
 SELECT * FROM person
 ---------^^^
 lateral view outer explode(array(30,60)) tabelName as c_age
 lateral view explode(array(40,80)) as d_age
 PIVOT (
 count(distinct age) as a
 for name in ('Mary','John')
 ) (state=,code=0)
{code}
 

 

Query2:

 
{code:java}
SELECT * FROM person
 PIVOT (
 count(distinct age) as a
 for name in ('Mary','John')
 )
 lateral view outer explode(array(30,60)) tabelName as c_age
 lateral view explode(array(40,80)) as d_age
{code}
 

Reuslt2:

+-------+------++---------------++--------
|id|Mary|John|c_age|d_age|

+-------+------++---------------++--------
|300|NULL|NULL|30|40|
|300|NULL|NULL|30|80|
|300|NULL|NULL|60|40|
|300|NULL|NULL|60|80|
|100|0|NULL|30|40|
|100|0|NULL|30|80|
|100|0|NULL|60|40|
|100|0|NULL|60|80|
|400|NULL|NULL|30|40|
|400|NULL|NULL|30|80|
|400|NULL|NULL|60|40|
|400|NULL|NULL|60|80|
|200|NULL|1|30|40|
|200|NULL|1|30|80|
|200|NULL|1|60|40|
|200|NULL|1|60|80|

+-------+------++---------------++--------

```
Bug: [UI] Reduce unnecessary allexecutors call when render stage page executor summary
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Jul/20 16:35
Affects Version/s: 3.0.0

Currently, stage page executor summary table data are built by merge /allexecutors response and stage.executorSummary. The data needed from /allexecutor are only two fields: hostPort and executorLogs.

By adding both fields in v1.ExecutorStageSummary, we can save one /allexecutors call and make frontend logic simpler
Bug: tree models prediction optimization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/20 04:53
Affects Version/s: 3.1.0

in {{Node}}'s method

 

{color:#e0957b}def {color}{color:#c7a65d}predictImpl{color}(features: Vector): LeafNode

 

use while-loop instead of the recursive way.
Bug: Add a test case to EliminateSortsSuite for protecting ORDER BY in DISTRIBUTE BY
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jul/20 05:36
Affects Version/s: 2.4.7

This is found during reviewing SPARK-32276.

*AFTER SPARK-32276*
{code}
scala> scala.util.Random.shuffle((1 to 100000).map(x => (x % 2, x))).toDF("a", "b").repartition(2).createOrReplaceTempView("t")

scala> sql("select * from (select * from t order by b) distribute by a").write.orc("/tmp/SPARK-32276")

$ ls -al /tmp/SPARK-32276/
total 632
drwxr-xr-x  10 dongjoon  wheel     320 Jul 14 22:08 ./
drwxrwxrwt  14 root      wheel     448 Jul 14 22:08 ../
-rw-r--r--   1 dongjoon  wheel       8 Jul 14 22:08 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel      12 Jul 14 22:08 .part-00000-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    1188 Jul 14 22:08 .part-00043-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    1188 Jul 14 22:08 .part-00191-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel       0 Jul 14 22:08 _SUCCESS
-rw-r--r--   1 dongjoon  wheel     119 Jul 14 22:08 part-00000-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  150735 Jul 14 22:08 part-00043-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  150741 Jul 14 22:08 part-00191-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc
{code}

*BEFORE*
{code}
scala> scala.util.Random.shuffle((1 to 100000).map(x => (x % 2, x))).toDF("a", "b").repartition(2).createOrReplaceTempView("t")

scala> sql("select * from (select * from t order by b) distribute by a").write.orc("/tmp/master")

$ ls -al /tmp/master/
total 56
drwxr-xr-x  10 dongjoon  wheel  320 Jul 14 22:12 ./
drwxrwxrwt  15 root      wheel  480 Jul 14 22:12 ../
-rw-r--r--   1 dongjoon  wheel    8 Jul 14 22:12 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel   12 Jul 14 22:12 .part-00000-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel   16 Jul 14 22:12 .part-00043-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel   16 Jul 14 22:12 .part-00191-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    0 Jul 14 22:12 _SUCCESS
-rw-r--r--   1 dongjoon  wheel  119 Jul 14 22:12 part-00000-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  932 Jul 14 22:12 part-00043-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  939 Jul 14 22:12 part-00191-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc
{code}