av9ash/GitBugs huggingface dataset:
Bug: Add `apply()` for a single field to `YearMonthIntervalType` and `DayTimeIntervalType`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 08:45
Affects Version/s: 3.2.0

Add 2 methods:
{code:scala}
  def apply(field: Byte): YearMonthIntervalType = YearMonthIntervalType(field, field)
  def apply(field: Byte): DayTimeIntervalType = DayTimeIntervalType(field, field)
{code}
Bug: Add Support in the ESS to serve merged shuffle block meta and data to executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 18:03
Affects Version/s: 3.1.0

With push-based shuffle enabled, the reducers send 2 different requests to the ESS:

1. Request to fetch the metadata of the merged shuffle block.
 2. Requests to fetch the data of the merged shuffle block in chunks which are by default 2MB in size.
 The ESS should be able to serve these requests and this Jira targets all the changes in the network-common and network-shuffle modules to be able to support this.
Bug: Show proper error message when update column types to year-month/day-time interval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/21 18:25
Affects Version/s: 3.2.0

Updating column types to interval types are prohibited for V2 source tables.
So, if we attempt to update the type of a column to the conventional interval type, an error message like "Error in query: Cannot update <table> field <column> to interval type;".

But, for year-month/day-time interval types, another error message like "Error in query: Cannot update <table> field <column>:<type> cannot be cast to interval year;"
It's not consistent.
Bug: Truncate java.time.Duration by fields of day-time interval type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 09:24
Affects Version/s: 3.2.0

Truncate input java.time.Duration instances using fields of DayTimeIntervalType. For example, if DayTimeIntervalType has the end field HOUR, granularity of DayTimeIntervalType values should hours too.
Bug: Truncate java.time.Period by fields of year-month interval type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/21 16:30
Affects Version/s: 3.2.0

Truncate input java.time.Period instances using fields of YearMonthIntervalType. For example, if YearMonthIntervalType has the end field YEAR, granularity of YearMonthIntervalType values should years too.
Bug: Spark fails on unrecognized hint in subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/21 00:29
Affects Version/s: 3.0.2

Spark queries to fail on unrecognized hints in subqueries. An example to reproduce:
{code:sql}
SELECT /*+ use_hash */ 42;
-- 21/06/08 01:28:05 WARN HintErrorLogger: Unrecognized hint: use_hash()
-- 42

SELECT *
FROM (
    SELECT /*+ use_hash */ 42
);
-- 21/06/08 01:28:07 WARN HintErrorLogger: Unrecognized hint: use_hash()
-- Error in query: unresolved operator 'Project [*];
-- 'Project [*]
-- +- SubqueryAlias __auto_generated_subquery_name
--    +- Project [42 AS 42#2]
--       +- OneRowRelation
{code}
Bug: Cannot resolve column when there is a unrecognized hint in subquery
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Jun/21 08:48
Affects Version/s: 3.0.0

{code:java}
select id,name from (select /*+ BROADCAS(t1) */ t1.id,t2.name from test_hint t1 inner join tmp_db.test_hint t2 on(t1.id=t2.id));
{code}
 

Run the above SQL and we will see the error: 
{code:java}
Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.AnalysisException: cannot resolve '`id`' given input columns: [__auto_generated_subquery_name.id, __auto_generated_subquery_name.name]; line 1 pos 7;
'Project ['id, 'name]
+- SubqueryAlias __auto_generated_subquery_name
   +- Project [id#64, name#67]
      +- Join Inner, (id#64 = id#66)
         :- SubqueryAlias t1
         :  +- SubqueryAlias spark_catalog.tmp_db.test_hint
         :     +- HiveTableRelation [`tmp_db`.`test_hint`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#64, name#65], Partition Cols: []]
         +- SubqueryAlias t2
            +- SubqueryAlias spark_catalog.tmp_db.test_hint
               +- HiveTableRelation [`tmp_db`.`test_hint`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#66, name#67], Partition Cols: []]        at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
        at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
        at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:341)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsUp$1(QueryPlan.scala:104)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:116)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:116)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:127)
        at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:132)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
        at scala.collection.immutable.List.foreach(List.scala:392)
{code}
Bug: Timestamp object inserts extra conversion to local time when extracting row
Status: Open | Priority: Major | Resolution: 
Created: 19/Jun/21 18:10
Affects Version/s: 3.0.1

When converting our code from spark 2.4 to spark 3.0, we noticed that several of our time related tests were failing.  Most of these were fixed by setting the spark session and java driver/executor times to be identical and UTC.  However, the timezone conversion tests were still failing because when we used df.first() or .collect() to extract a single row of the dataframe, the local computer time was being introduced to the datetime object and inserting an extra conversion. 

Below is my MWE. 

These tests passed when we were working wtih pyspark 2.4.6/7, but have failed in pyspark 3.0.1/2.  We can not currently upgrade to python 3.9 because we need pyarrow support and we can not move to pyspark 3.1 because apache sedona does not support that.
{code:java}
// time_zone_MWE.py

import datetime
import pyspark.sql.functions as sf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

if __name__ == "__main__":    
   conf = SparkConf()    
   arguments = {
        "spark.master": "local[2]",
        "spark.sql.parquet.enableVectorizedReader": "False",
        "spark.sql.execution.arrow.pyspark.enabled": "True",
        "spark.sql.execution.arrow.pyspark.fallback.enabled": "True",
        "spark.submit.deployMode": "client",
        "spark.sql.session.timeZone": "UTC",
        "spark.sql.legacy.timeParserPolicy": "LEGACY",
        "spark.cleaner.referenceTracking.cleanCheckpoints": "True",
        "spark.driver.extraJavaOptions": "-Duser.timezone=UTC",
        "spark.executor.extraJavaOptions": "-Duser.timezone=UTC",
    }
    conf.setAll(arguments.items())
    spark = SparkSession.builder.config(conf=conf).getOrCreate()
    # Test Dataframe with local time and the timezone
    timezone_data = spark.createDataFrame(
        [("2008-02-02 13:36:44", "Asia/Vladivostok")], ["datetime", "timezone"]
    )
    # expected conversion should be UTC+10 should be 2008-02-02 03:36:44
    df = timezone_data.withColumn(
        "utc_ts",
        sf.to_utc_timestamp(
            timezone_data.datetime.cast("timestamp"), timezone_data.timezone
        ),
    )
    df = df.withColumn("utc_int", df.utc_ts.cast("int"))
    # This shows correct time
    print(df.show())
    # This shows a local computer system offset
    print(df.first())
    # Comparing the two integers will work
    assert (
        int(
            datetime.datetime(
                2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
            ).timestamp()
        )
        == df.first()["utc_int"]
    )
    # Comparing the two datetime objects will not (but worked in pyspark 2.4.6)
    assert df.first()["utc_ts"] == datetime.datetime(
        2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
    )

{code}
Bug: Always enable the `pandas_metadata` in DataFrame.parquet
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 18/Jun/21 04:09
Affects Version/s: 3.2.0

We have a argument named `pandas_metadata` in [ps.read_parquet|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.read_parquet.html], but seems we can just always enable so that it always respect the pandas metadata when reading the Parquet file written by pandas.
Bug: Refactor LinearRegression - make huber support virtual centering
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 03/Jun/21 01:49
Affects Version/s: 3.2.0

1, make huber regression support virtual centering

2, as to \{LeastSquares}, it always compute without intercept, and estimate the intercept after optimizing the linear part. So just re-org the LeastSquares part
Bug: Increase the default JVM stack size of SBT/Maven
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 07:37
Affects Version/s: 3.2.0

The jenkins SBT/Maven keep failing with stack overflow error:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139542

We should increase the JVM stack size.
Bug: Lateral join should only be used with subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/21 17:18
Affects Version/s: 3.2.0

This is a follow up for SPARK-34382. Currently the keyword LATERAL can be used in front of a `relationPrimary`, which consists of more than just subqueries, for example:

select * from t1, lateral t2

Such syntax is not allowed in Postgres. LATERAL should only be used in front of a subquery.
Bug: View should not capture configs used in `RelationConversions`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/21 02:00
Affects Version/s: 3.2.0

RelationConversions is actually a optimization rule while it's executed in the analysis phase. For view, it's designed to only capture sementic configs, so we should ignore the configs related to `RelationConversions`
Bug: Repartition before writing data source tables
Status: Open | Priority: Major | Resolution: 
Created: 17/Jun/21 07:26
Affects Version/s: 3.2.0

This umbrella ticket aim to track repartition before writing data source tables. It contains:
 # repartition by dynamic partition column before writing dynamic partition tables.
 # repartition before writing normal tables to avoid generating too many small files.
 # Improve local shuffle reader.
Bug: HighlyCompressedMapStatus should record accurately the size of skewed shuffle blocks
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Jun/21 14:57
Affects Version/s: 3.0.2

HighlyCompressedMapStatus currently cannot record accurately the size of shuffle blocks which much greater than other block but small than `spark.shuffle.accurateBlockThreshold`, which is likely to lead OOM when fetch shuffle blocks. We have to tune some extra properties like `spark.reducer.maxReqsInFlight` to prevent it, so it is better to fix it in HighlyCompressedMapStatus.
Bug: Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 18:44
Affects Version/s: 3.2.0

SQLMetrics allows creation of different metrics like sizing, timing metrics etc. some of those names for the metric can be duplicated and along with that the `Some` wrapper objects adds additional memory overhead. In our internal platform, this has caused huge memory usage on driver causing Full GC to kick in every so often.

cc [~mridulm80]
Bug: fix special char in CSV header with filter pushdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 14:10
Affects Version/s: 3.2.0

This is a followup of [#31964|https://github.com/apache/spark/pull/31964]

We should only quote the column name when nested column predicate pushdown is enabled, otherwise the data source side may not have the logic to parse the quoted column name and fail. This is not a problem before [#31964|https://github.com/apache/spark/pull/31964] , as we don't quote the column name if there is no dot in the name. But [#31964|https://github.com/apache/spark/pull/31964] changed it.
Bug: Support fields by the day-time interval type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 19:25
Affects Version/s: 3.2.0

Extend the DayTimeIntervalType class and add two parameters that should specify the start and end fields in the type. This will allow to support of the following sub-types:
# INTERVAL DAY
# INTERVAL DAY TO HOUR
# INTERVAL DAY TO MINUTE
# INTERVAL HOUR
# INTERVAL HOUR TO MINUTE
# INTERVAL HOUR TO SECOND
# INTERVAL MINUTE
# INTERVAL MINUTE TO SECOND
# INTERVAL SECOND
Bug: Parse day-time interval literals to tightest types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 10:09
Affects Version/s: 3.2.0

Modify AstBuilder.visitInterval to parse day-time interval literals to tightest types. For example:
INTERVAL '10 20:30' DAY TO MINUTE should be parsed as DayTimeIntervalType(DAY, MINUTE) but not as DayTimeIntervalType(DAY, SECOND).
Bug: Contention on SQLConf.sqlConfEntries and SQLConf.staticConfKeys
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 14:30
Affects Version/s: 3.1.2

The global locks used to protect {{SQLConf.sqlConfEntries}} map and the {{SQLConf.staticConfKeys}} set can cause significant contention on the SQLConf instance in a concurrent setting. 

Using copy-on-write versions should reduce the contention given that modifications to the configs are relatively rare.
Bug: [K8s] Spark executors memory request should be allowed to deviate from limit
Status: Open | Priority: Major | Resolution: 
Created: 11/Jun/21 06:59
Affects Version/s: 3.1.2

Currently the driver and executor memor requests always equals the limit.
As stated in [SPARK-23825|https://issues.apache.org/jira/browse/SPARK-23825], this is a reasonable default and is especially important for the driver.

For executors however, it might be usefull for users to deviate from this default for executors.

In typical development environments on K8s, the namespace quotas are an upper bound to the memory request that is possible.
The limits however can be much higher. For development, spark is often run in client mode. While the driver should request the memory it needs, we want to leverage all the resources of the cluster whith executors if they are free - and can live with an executor maybe beeing killed eventually. 

Thus I propose the introduction of {{spark.{driver,executor}.limit.memory}} similar to the {{spark.{driver,executor}.limit.cpu}}.
Bug: Overflow on converting valid Timestamp to Microseconds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/21 15:45
Affects Version/s: 3.2.0


{code:java}
scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._

scala> instantToMicros(microsToInstant(Long.MinValue))
java.lang.ArithmeticException: long overflow
  at java.lang.Math.multiplyExact(Math.java:892)
  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:389)
  ... 49 elided


{code}
Bug: Complete arithmetic operators involving bool literals, Series, and Index
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/21 17:50
Affects Version/s: 3.2.0

Completing arithmetic operators involving bool literals, Series, and Index consists of two main tasks:
 * Support arithmetic operations against bool literals
 * Support operators (+, *) between bool Series/Indexes.

We ought to match pandas' behaviors.
Bug: spark.speculation causes incorrect query results with TRANSFORM
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/21 12:43
Affects Version/s: 2.4.8

SQL as follow gets incorrect results sometimes when spark.speculation is true: 
{code:java}
SELECT count(1)
FROM
  (SELECT TRANSFORM(tmpa1.*) USING "python test.py" AS (dt)
   FROM
     (SELECT dt
      FROM test_table)tmpa1)tmpa2{code}
With spark.speculation=true, the count result is less than the correct one. It's more likely to get incorrect result when there are more speculative tasks. 

`test.py`:
{code:java}
import sys
for line in sys.stdin:
    line = line.strip()
    arr = line.split()
    print "\t".join(arr){code}
 

spark-sql command:
{code:java}
./bin/spark-sql --master yarn \ 
--conf spark.speculation=true \ 
--conf spark.shuffle.service.enabled=true \ 
--conf spark.dynamicAllocation.enabled=true \ 
--conf spark.dynamicAllocation.executorIdleTimeout=5s \ 
--conf spark.dynamicAllocation.initialExecutor=1 \ 
--conf spark.dynamicAllocation.maxExecutors=40
{code}
Bug: Support arithmetic operators (+, *) among bool Series/Index
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Jun/21 21:04
Affects Version/s: 3.2.0

We ought to support arithmetic operators (+, *) between bool Series/Indexes, following the behavior of pandas, for example
{code:java}
>>> pd.Series([True, False, True]) * pd.Series([True, False, False])
0 True
1 False
2 False
dtype: bool
{code}
Note: allow_bool_index_ops and allow_bool parameters of is_valid_operand_for_numeric_arithmetic can be consolidated then. Refer to discussions in https://github.com/apache/spark/pull/32785#discussion_r647100350
Bug: Avoid write null to StateStore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/21 23:57
Affects Version/s: 3.0.2

According to {{get}} method doc in StateStore API, it returns non-null row if the key exists. So basically we should avoid write null to StateStore. You cannot distinguish if the returned null row is because the key doesn't exist, or the value is actually null. And due to the defined behavior of {{get}}, it is quite easy to cause NPE error if the caller doesn't expect to get a null if the caller believes the key exists.
Bug: Support dynamic executor range for dynamic allocation
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Jun/21 10:15
Affects Version/s: 3.2.0

Currently, Spark allows users to set scalability within a Spark application using dynamic allocation. spark.dynamicAllocation.minExecutors & spark.dynamicAllocation.maxExecutors are used for scaling up and down. Within an application，Spark tactfully use them to request executors from cluster manager according to the real-time workload. Once set, the range is fixed through the whole application lifecycle. This is not very convenient for long-running application when the range should be changeable for some cases, such as:
1. the cluster manager itself or the queue will scale up and down, which looks very likely to happen in modern cloud platforms
2. the application is long-running, but the timeliness, priority, e.t.c are not only determined by the workload with the application, but also by the traffic across the cluster manager or just different moments
3. e.t.c.
Bug: List Python 3.9 installed libraries in build_and_test workflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 03:27
Affects Version/s: 3.2.0

In the build_and_test workflow, tests are run against both Python 3.6 and Python 3.9. However, only libraries installed in Python 3.6 are listed. We should list Python 3.9's installed libraries as well. That would be helpful for debugging.
Bug: Support LZ4 compression in ORC data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 13:24
Affects Version/s: 3.2.0

Apache ORC supports LZ4 compression, but we cannot set LZ4 compression in the ORC data source
{code:java}
scala> spark.range(10).write.option("compression", "lz4").orc("/tmp/lz4")
java.lang.IllegalArgumentException: Codec [lz4] is not available. Available codecs are uncompressed, lzo, snappy, zlib, none, zstd.{code}
Bug: ExpandExec should not introduce unnecessary exchanges
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Jun/21 12:17
Affects Version/s: 3.2.0

The Expand is commonly introduced by the RewriteDistinctAggregates optimizer rule.
In that case there can be several attributes that are kept as they are by the Expand.
If the child's output is partitioned by those attributes, then so will be the output of the Expand.
In general case the Expand can output data with arbitrary partitioning, so set it as UNKNOWN partitioning.
Bug: Decommission executors in batches to avoid overloading network by block migrations.
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Jun/21 10:18
Affects Version/s: 3.1.2

Currenlty, each executor is asked to starts offloading rdd and shuffle blocks as soon it is decommissioned. This can overload the network bandwidth of the application.

 

We should limit the number of executors migrating rdd and shuffle blocks at one time. This would be configurable to enable users to decommission executors conservatively or aggressively based on the use cases.
Bug: Decommission executors in batches to avoid overloading network by block migrations.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Jun/21 10:09
Affects Version/s: 3.1.2

Currenlty, each executor is asked to starts offloading rdd and shuffle blocks as soon it is decommissioned. This can overload the network bandwidth of the application.

 

We should limit the number of executors migrating rdd and shuffle blocks at one time. This would be configurable to enable users to decommission executors conservatively or aggressively based on the use cases.