av9ash/GitBugs huggingface dataset:
Bug: Forbid time field steps for date start/end in Sequence
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/20 23:12
Affects Version/s: 3.0.0

*Sequence time field steps for date start/end looks strange in spark as follows:*

scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 hour))").head(3)
res0: Array[org.apache.spark.sql.Row] = _Array([2011-03-01], [2011-03-01], [2011-03-01])_ *<- strange result.*

scala> sql("select explode(sequence(cast('2011-03-01' as date), cast('2011-03-02' as date), interval 1 day))").head(3)
res1: Array[org.apache.spark.sql.Row] = Array([2011-03-01], [2011-03-02])

*While this behavior in Prosto make sense：* 

presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' hour);
 Query 20200624_122744_00002_pehix failed: sequence step must be a day interval if start and end values are dates
 presto> select sequence(date('2011-03-01'),date('2011-03-02'),interval '1' day);
 _col0
 [2011-03-01, 2011-03-02]
Bug: Ignore timeout error when remove blocks on the lost executor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/20 15:01
Affects Version/s: 2.4.0

When removing blocks(e.g. RDD, broadcast, shuffle), BlockManagerMaserEndpoint will make RPC calls to each known BlockManagerSlaveEndpoint to remove the specific blocks. The PRC call sometimes could end in a timeout when the executor has been lost, but only notified the BlockManagerMasterEndpoint after the removing call has already happened. The timeout could therefore fail the whole query.

In this case, we actually could just ignore the error since those blocks on the lost executor could be considered as removed already.
Bug: StringConcat can overflow `length`, leads to StringIndexOutOfBoundsException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 17:55
Affects Version/s: 2.4.4

We have query plans that through multiple transformations can grow extremely long in length. These would eventually throw OutOfMemory exceptions (https://issues.apache.org/jira/browse/SPARK-26103 & related https://issues.apache.org/jira/browse/SPARK-25380).

 

We backported the changes from [https://github.com/apache/spark/pull/23169] into our distribution of Spark, based on 2.4.4, and attempted to use the added `spark.sql.maxPlanStringLength`. While this works in some cases, large query plans can still lead to issues stemming from `StringConcat` in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala.

 

The following unit test exhibits the issue, which continues to fail in the master branch of spark:

 
{code:scala}
  test("StringConcat doesn't overflow on many inputs") {    
    val concat = new StringConcat(maxLength = 100)
    0.to(Integer.MAX_VALUE).foreach { _ =>      
      concat.append("hello world")    
     }    
    assert(concat.toString.length === 100)  
} 
{code}
 

Looking at the append method here: [https://github.com/apache/spark/blob/fc6af9d900ec6f6a1cbe8f987857a69e6ef600d1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala#L118-L128]

 

It seems like regardless of whether the string to be append is added fully to the internal buffer, added as a substring to reach `maxLength`, or not added at all the internal `length` field is incremented by the length of `s`. Eventually this will overflow an int and cause L123 to substring with a negative index.
Bug: Hadoop file system config should be effective in data source options 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/20 23:16
Affects Version/s: 2.4.6

Data source options should be propagated into the hadoop configuration of method `checkAndGlobPathIfNecessary`

From org.apache.hadoop.fs.FileSystem.java:
{code:java}
  public static FileSystem get(URI uri, Configuration conf) throws IOException {
    String scheme = uri.getScheme();
    String authority = uri.getAuthority();

    if (scheme == null && authority == null) {     // use default FS
      return get(conf);
    }

    if (scheme != null && authority == null) {     // no authority
      URI defaultUri = getDefaultUri(conf);
      if (scheme.equals(defaultUri.getScheme())    // if scheme matches default
          && defaultUri.getAuthority() != null) {  // & default has authority
        return get(defaultUri, conf);              // return default
      }
    }
    
    String disableCacheName = String.format("fs.%s.impl.disable.cache", scheme);
    if (conf.getBoolean(disableCacheName, false)) {
      return createFileSystem(uri, conf);
    }

    return CACHE.get(uri, conf);
  }
{code}

With this, we can specify URI schema and authority related configurations for scanning file systems.
Bug: Support query execution before/after reading/writing over JDBC
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jun/20 07:55
Affects Version/s: 3.0.0

For ETL workload, there is a common requirement to perform SQL statement before/after reading/writing over JDBC.
 Here's examples;
 - Create a view with specific conditions
 - Delete/Update some records
 - Truncate a table (it is already possible in `truncate` option)
 - Execute stored procedure (it is also requested in SPARK-32014)

Currently `query` options is available to specify SQL statement against JDBC datasource when loading data as DataFrame.
 However, this query is only for reading data, and it does not support the common examples listed above.

On the other hand, there is `sessionInitStatement` option available before writing data from DataFrame.
 This option is to run custom SQL in order to implement session initialization code. Since it runs per session, it cannot be used for non-idempotent operations.

 

If Spark can support executing SQL statement against JDBC datasources before/after reading/writing over JDBC, it can cover a lot of common use-cases.

Note: Databricks' old Redshift connector has similar option like `preactions` and `postactions`. [https://github.com/databricks/spark-redshift]
Bug: YARN: archives rename with # doesn't work for https
Status: Open | Priority: Major | Resolution: 
Created: 29/Jun/20 23:13
Affects Version/s: 2.3.0

This is related to SPARK-10858

The YARN distributed cache feature with --archives where you can rename the archive using a # symbol does not work with the http(s) scheme:


{{--archives http://mirror.sfo12.us.leaseweb.net/centos/6.10/isos/i386/sha1sum.txt#sha1sum}}

This is because URLs can have fragments and therefore the # is interpreted as part of the fragment. We could use a similar trick as we do for the other two schemes file:// and hdfs:// in which first we remove the last fragment, parse the URL, and then reattach the fragment. The [code exists|https://github.com/apache/spark/pull/9035/files] but it is not applied to URLs.
Bug: Support AQE skew join with Union
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jun/20 14:14
Affects Version/s: 3.0.0

Current, the AQE skew join only supports two tables join such as
{code}
SMJ
:-Sort
:    +-Shuffle
+-Sort
     +-Shuffle
{code}
But if the plan contains a Union, the skew join handling not work:
{code}
Union
:-SMJ
:   :-Sort
:   :    +-Shuffle
:   +-Sort
:        +-Shuffle
+-SMJ
:   :-Sort
:   :    +-Shuffle
:   +-Sort
         +-Shuffle
{code}
Bug: Check rules for MERGE INTO should use MergeAction.condition other than MergeAction.children
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/20 06:05
Affects Version/s: 3.0.0

[SPARK-30924|https://issues.apache.org/jira/browse/SPARK-30924] adds some check rules for MERGE INTO one of which ensures the first MATCHED clause must have a condition. However, it uses {{MergeAction.children}} in the checking which is not accurate for the case, and it lets the below case pass the check:

{code:scala}
MERGE INTO testcat1.ns1.ns2.tbl AS target
xxx
WHEN MATCHED THEN UPDATE SET target.col2 = source.col2
WHEN MATCHED THEN DELETE
xxx
{code}

We should use {{MergeAction.condition}} instead.
Bug: Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/20 07:00
Affects Version/s: 3.0.1

Now the {{MERGE INTO}} syntax is,
{code:sql}
MERGE INTO [db_name.]target_table [AS target_alias]
 USING [db_name.]source_table [<time_travel_version>] [AS source_alias]
 ON <merge_condition>
 [ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
 [ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
 [ WHEN NOT MATCHED [ AND <condition> ] THEN <not_matched_action> ]{code}
It would be nice if we support unlimited {{MATCHED}} and {{NOT MATCHED}} clauses in {{MERGE INTO}} statement, because users may want to deal with different "{{AND <condition>}}"s, the result of which just like a series of "{{CASE WHEN}}"s. The expected syntax looks like
{code:sql}
MERGE INTO [db_name.]target_table [AS target_alias]
 USING [db_name.]source_table [<time_travel_version>] [AS source_alias]
 ON <merge_condition>
 [when_clause [, ...]]
{code}

 where {{when_clause}} is 
{code:java}
WHEN MATCHED [ AND <condition> ] THEN <matched_action>{code}
or
{code:java}
WHEN NOT MATCHED [ AND <condition> ] THEN <not_matched_action>{code}
Bug: [Python] Setting `spark.sql.session.timeZone` only partially respected
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Jun/20 22:40
Affects Version/s: 3.0.0

Reopening SPARK-25244 as it is unresolved as of versions 2.4.6 and 3.0.0.

The setting {{spark.sql.session.timeZone}} is respected by PySpark when converting from and to Pandas, as described [here|http://spark.apache.org/docs/latest/sql-programming-guide.html#timestamp-with-time-zone-semantics]. However, when timestamps are converted directly to Pythons {{datetime}} objects, its ignored and the systems timezone is used.

This can be checked by the following code snippet
{code:java}
import pyspark.sql

spark = (pyspark
         .sql
         .SparkSession
         .builder
         .master('local[1]')
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate()
        )

df = spark.createDataFrame([("2018-06-01 01:00:00",)], ["ts"])
df = df.withColumn("ts", df["ts"].astype("timestamp"))

print(df.toPandas().iloc[0,0])
print(df.collect()[0][0])
{code}
Which for me prints (the exact result depends on the timezone of your system, mine is Europe/Berlin)
{code:java}
2018-06-01 01:00:00
2018-06-01 03:00:00
{code}
Hence, the method {{toPandas}} respected the timezone setting (UTC), but the method {{collect}} ignored it and converted the timestamp to my systems timezone.

The cause for this behaviour is that the methods {{toInternal}} and {{fromInternal}} of PySparks {{TimestampType}} class don't take into account the setting {{spark.sql.session.timeZone}} and use the system timezone.
Bug: Easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/20 00:18
Affects Version/s: 3.0.0

Repartition/coalesce is very important to optimize Spark application's performance, however, a lot of users are struggling with determining the number of partitions.
 This issue is to add a easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time.

It will help Spark users to determine the optimal number of partitions.

Expected use-cases:
 - repartition with the calculated parallel tasks

Notes:
 - `SparkContext.maxNumConcurrentTasks` might help but it cannot be accessed by Spark apps.
 - `SparkContext.getExecutorMemoryStatus` might help to calculate the number of available slots to process tasks.
Bug: Dataset.foreachPartition returns object
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/20 11:53
Affects Version/s: 3.0.0

I'm trying to map values from the Dataset[Row], but since 3.0.0 this fails.

In 3.0.0 I'm dealing with an error: "Error:(28, 38) value map is not a member of Object"

 

This is the simplest code that works in 2.4.x, but fails in 3.0.0:
{code:scala}
spark.range(100)
  .repartition(10)
  .foreachPartition(part => println(part.toList))
{code}
Bug: Spark sequence doesn't handle date increments that cross DST
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jun/20 12:49
Affects Version/s: 2.4.4

Run the following query with the Spark default time zone set to "America/Chicago"

 
{code:java}
sequence(cast("2011-03-01" as date), cast("2011-05-01" as date), interval 1 month)
{code}
The result is:

 
{noformat}
[2011-03-01, 2011-03-31, 2011-04-30]{noformat}
I've traced the code what happens is that during this month there is daylight savings and so  the time ends up being 11pm on 3/31 instead of midnight on 4/1.

If you run this with timezone set to GMT or run the sequence functions over months that don't have daylight savings moving forward, it works correctly.

I think this is because the toLong and from long functions are not using a timezone but timestampAddInterval is.  I think the solution is probably to make everything time zone aware, but maybe you can just always use GMT for timestmapAddInterval.
Bug: Add a redirect to sql-ref from sql-reference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/20 00:41
Affects Version/s: 3.0.0

A number of Google searches I’ve done today have turned up [https://spark.apache.org/docs/latest/sql-reference.html], which does not exist any more. Thus, we should add a redirect to sql-ref.html.
Bug: Unaligned benchmark results 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/20 11:21
Affects Version/s: 3.1.0

If the length of benchmark names is greater than 40, benchmark results are not aligned to column names. For example:
{code}
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
make_timestamp():                         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
prepare make_timestamp()                           3636           3673          38          0.3        3635.7       1.0X
make_timestamp(2019, 1, 2, 3, 4, 50.123456)             94             99           4         10.7          93.8      38.8X
make_timestamp(2019, 1, 2, 3, 4, 60.000000)             68             80          13         14.6          68.3      53.2X
make_timestamp(2019, 12, 31, 23, 59, 60.00)             65             79          19         15.3          65.3      55.7X
make_timestamp(*, *, *, 3, 4, 50.123456)            271            280          14          3.7         270.7      13.4X
{code}
Bug: Structured Streaming application continuity when encountering streaming query task level error
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/20 14:26
Affects Version/s: 2.4.5

From the Spark Structured Streaming application continuity perspective, the thread that ran this task was terminated with ERROR SPARK-31995 but to YARN it is still an active running job even though this instance of the Spark Structured Streaming job is no longer making any further processing.  If the monitoring of the Spark Structured Streaming job is done only from the YARN job perspective, it may provide a false status.  In this situation, should the Spark Structure Streaming application fail hard and completely (fail by Spark framework or Application exception handling)?  Or should the developer investigate and develop some ideal monitoring implementation that has the right level of specificity to detect Spark Structured Streaming *task* level failures?  Any references on these topics are much appreciated.
Bug: Extract common code from date-time field expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/20 12:24
Affects Version/s: 3.1.0

The following expressions calls DateTimeUtils functions to get date-time fields from input date or timestamps, so, the common code can be extracted to common traits:
- Second
- Minute
- Hour
- Year
- Month and etc.
Bug: Fix the wrong references of PartialMerge/Final AggregateExpression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/20 07:12
Affects Version/s: 3.0.0

For the PartialMerge/Final AggregateExpression, it should reference the `inputAggBufferAttributes` instead of `aggBufferAttributes` according to `AggUtils.planAggXXX`
Bug: Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/20 08:45
Affects Version/s: 3.0.0

When stopping the HiveServer2, the non-daemon thread stops the server from terminating

{code:java}
"HiveServer2-Background-Pool: Thread-79" #79 prio=5 os_prio=31 tid=0x00007fde26138800 nid=0x13713 waiting on condition [0x0000700010c32000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hive.service.cli.session.SessionManager$1.sleepInterval(SessionManager.java:178)
	at org.apache.hive.service.cli.session.SessionManager$1.run(SessionManager.java:156)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


{code}


Also, causes issues as HIVE-14817 described
Bug: Idle cores not being allocated
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/20 20:58
Affects Version/s: 2.4.5

*Background:*

I have a cluster (2.4.5) using standalone mode orchestrated by Nomad jobs 
running on EC2. We deploy a Scala web server as a long running jar via 
{{spark-submit}} in client mode. Sometimes we get into a state where the 
application ends up with 0 cores due to our in-house autoscaler scaling down 
and killing workers without checking if any of the cores in the worker were 
allocated to existing applications. These applications then end up with 0 
cores, even though there are healthy workers in the cluster. 

However, if i submit a new application or register a new worker in the 
cluster, only then will the master finally reallocate cores to the 
application. This is problematic, because the long running 0 core 
application is stuck. 

Could this be related to the fact that {{schedule()}} is only triggered by new 
workers / new applications as commented here? 
https://github.com/apache/spark/blob/v2.4.5/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L721-L724

If that is the case, should the application be calling {{schedule()}} when 
removing workers after calling {{timeOutWorkers()}}? 
https://github.com/apache/spark/blob/v2.4.5/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L417

The downscaling causes me to see this in my logs, so i am fairly certain 
{{timeOutWorkers()}} is being called: 

{code}
20/06/08 11:40:56 INFO Master: Application app-20200608114056-0006 requested 
to set total executors to 1. 
20/06/08 11:40:56 INFO Master: Launching executor app-20200608114056-0006/0 
on worker worker-20200608113523-<IP_ADDRESS>-7077 
20/06/08 11:41:44 WARN Master: Removing 
worker-20200608113523-<IP_ADDRESS>-7077 because we got no heartbeat in 60 
seconds 
20/06/08 11:41:44 INFO Master: Removing worker 
worker-20200608113523-<IP_ADDRESS>-7077 on <IP_ADDRESS>:7077 
20/06/08 11:41:44 INFO Master: Telling app of lost executor: 0 
20/06/08 11:41:44 INFO Master: Telling app of lost worker: 
worker-20200608113523-10.158.242.213-7077 
{code}
Bug: make_interval does not accept seconds >100
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/20 11:23
Affects Version/s: 3.0.0

In make_interval(years, months, weeks, days, hours, mins, secs), secs are defined as Decimal(8, 6), which turns into null if the value of the expression overflows 100 seconds.
Larger seconds values should be allowed.

This has been reported by Simba, who wants to use make_interval to implement translation for TIMESTAMP_ADD ODBC function in Spark 3.0.
ODBC {fn TIMESTAMPADD(SECOND, integer_exp, timestamp} fails when integer_exp returns seconds values >= 100.
Bug: cleanup hive scratch dir should work for the developer api startWithContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 14:07
Affects Version/s: 3.0.0

Comparing to the long-running ThriftServer via start-script, we are more likely to hit the issue  https://issues.apache.org/jira/browse/HIVE-10415 / https://issues.apache.org/jira/browse/SPARK-31626 in the developer API startWithContext
Bug: 3.0.0 release build fails
Status: Closed | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 20:35
Affects Version/s: 3.0.0

Hi,
I try to build the spark release 3.0.0 by myself.

I got the following error.
{code}                                          
20/06/16 15:20:49 WARN PrefixSpan: Input data is not cached.
20/06/16 15:20:50 WARN Instrumentation: [b307b568] regParam is zero, which might cause numerical instability and overfitting.
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
'vignetteInfo' is not an exported object from 'namespace:tools'
Execution halted
{code}

I can reproduce this error with a small Dockerfile.

EDIT: Issue is solved with ubuntu 20.04 as base image.
{code}
FROM ubuntu:20.04 as builder

ENV MVN_VERSION=3.6.3 \
    M2_HOME=/opt/apache-maven \
    MAVEN_HOME=/opt/apache-maven \
    MVN_HOME=/opt/apache-maven \
    MVN_SHA512=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0 \
    MAVEN_OPTS="-Xmx3g -XX:ReservedCodeCacheSize=1g" \
    R_HOME=/usr/lib/R \
    GIT_REPO=https://github.com/apache/spark.git \
    GIT_BRANCH=v3.0.0 \
    SPARK_DISTRO_NAME=hadoop3.2 \
    SPARK_LOCAL_HOSTNAME=localhost

# Preparation
RUN /usr/bin/apt-get update && \
    # APT
    INSTALL_PKGS="openjdk-8-jdk-headless git wget python3 python3-pip python3-setuptools r-base r-base-dev pandoc pandoc-citeproc libcurl4-openssl-dev libssl-dev libxml2-dev texlive qpdf language-pack-en" && \
    DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get -y install --no-install-recommends $INSTALL_PKGS && \
    rm -rf /var/lib/apt/lists/* && \
    Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" && \
    # Maven
    /usr/bin/wget -nv -O apache-maven.tar.gz "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=maven/maven-3/${MVN_VERSION}/binaries/apache-maven-${MVN_VERSION}-bin.tar.gz" && \
    echo "${MVN_SHA512} apache-maven.tar.gz" > apache-maven.sha512 && \
    sha512sum --strict -c apache-maven.sha512 && \
    tar -xvzf apache-maven.tar.gz -C /opt && \
    rm -v apache-maven.sha512 apache-maven.tar.gz && \
    /bin/ln -vs /opt/apache-maven-${MVN_VERSION} /opt/apache-maven && \
    /bin/ln -vs /opt/apache-maven/bin/mvn /usr/bin/mvn

# Spark Distribution Build
RUN mkdir -p /workspace && \
    cd /workspace && \
    git clone --branch ${GIT_BRANCH} ${GIT_REPO} && \
    cd /workspace/spark && \
    ./dev/make-distribution.sh --name ${SPARK_DISTRO_NAME} --pip --r --tgz -Psparkr -Phadoop-3.2 -Phive-2.3 -Phive-thriftserver -Pyarn -Pkubernetes
{code}

I am very grateful to all helpers.
Bug: Create date/timestamp formatters once before collect in `hiveResultString()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 18:52
Affects Version/s: 3.0.1

Spark 2.4 re-uses one instance of SimpleDateFormat while formatting timestamps in toHiveString. Currently, toHiveString() creates timestampFormatter per each value. Even w/ caching, it causes additional overhead comparing to Spark 2.4. The ticket aims to create an instance of TimestampFormatter before collect in hiveResultString()