av9ash/GitBugs huggingface dataset:
Bug: Support setting "since" version in FunctionRegistry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/21 12:24
Affects Version/s: 3.2.0

Spark 3.2.0 includes two new functions "regexp" and "regexp_like", which are identical to "rlike". However, in the generated documentation. the since versions of both functions are "1.0.0" since they are based on the expression "RLike".
https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#regexp
https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#regexp_like

FunctionRegistry should support setting "since" version to make sure the generated documentation is correct.
Bug: Migrate all ParsedStatement to the new v2 command framework
Status: Open | Priority: Major | Resolution: 
Created: 25/Aug/21 14:02
Affects Version/s: 3.3.0

The ParsedStatement needs to be pattern matched in two analyzer rules and results to a lot of duplicated code.

The new v2 command framework defines a few basic logical plan nodes such as UnresolvedTable, and we only need to resolve these basic nodes, and pattern match v2 commands only once in the rule `ResolveSessionCatalog` for v1 command fallback.

We should migrate all the ParsedStatement to v2 command framework.
Bug: Improve test coverage for pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Done
Created: 13/Aug/21 00:55
Affects Version/s: 3.3.0

There are many codes in pandas-on-Spark are not being tested, for example:
 * (Series|DataFrame).to_clipboard

             !Screen Shot 2021-08-13 at 9.56.48 AM.png|width=548,height=125!            
 * `value` and `method` argument for Series.fillna

             !Screen Shot 2021-08-13 at 9.59.23 AM.png|width=551,height=34!

 

The red line above screen capture means that "this line is not being tested".

Now the test coverage of pandas-on-Spark is 89.93% for total, 93.43% for frame.py (which is including DataFrame API), 89.04% for indexes/*.py (which is including Index API) and 93.43% for series.py (which is including Series API).

Not necessarily cover the 100% of codes, since some test such as `DataFrame.to_delta` is untestable for now, but we should cover the codes as much as possible for healthy of project.

You can find more missing tests and percentage of coverage in [code cov report|https://app.codecov.io/gh/apache/spark].
Bug: Add DSV2 filters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/21 16:10
Affects Version/s: 3.3.0

add DS V2 filters:
* EqualTo
* EqualNullSafe
* GreaterThan
* GreaterThanOrEqual
* LessThan
* LessThanOrEqual
* InIsNull
* IsNotNull
* And
* Or
* Not
* StringStartsWith
* StringEndsWith
* StringContains
* AlwaysTrue
* AlwaysFalse
Bug: Add hive-thriftserver to parent maven pom file
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/Aug/21 07:36
Affects Version/s: 3.1.2

hive-thriftserver module does not exist in the parent pom file, it's not friendly to developers.
 * IDE can not check syntax errors on runtime code patching.
 * IDE can not support grammar prompt
Bug: Add `thousands` argument to `ps.read_csv`.
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Aug/21 07:51
Affects Version/s: 3.2.0

When reading csv file in pandas, pandas automatically detect the thousand separator if `thousands` argument is specified.
{code:java}
>>> pd.read_csv(path, sep=";")
    name  age        job      money
0  Jorge   30  Developer  1,000,000
1    Bob   32  Developer    1000000

>>> pd.read_csv(path, sep=";", thousands=",")
    name  age        job    money
0  Jorge   30  Developer  1000000
1    Bob   32  Developer  1000000{code}
However, pandas-on-Spark doesn't support it.
Bug: Implement __getitem__ of label-based MultiIndex
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Aug/21 18:20
Affects Version/s: 3.3.0

 
{code:java}
>>> psdf = ps.DataFrame({'a':[1, 2, 3], 'b':[4,5,6], 'c':[7,8,9]}, index=pd.MultiIndex.from_tuples([('a', 'x'), ('a', 'y'), ('b', 'z')]))

>>> psdf.loc[('a', 'x')]
Traceback (most recent call last):
...
KeyError: 'x'{code}
Bug: Redact sensitive information in Spark Thrift Server UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/21 23:04
Affects Version/s: 3.1.2

Spark UI displays sensitive information on "JDBC/ODBC Server" tab

The reason of the issue is in [org.apache.spark.sql.hive.thriftserver.ui.SqlStatsPagedTable|https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala#L166] class [here|https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala#L266-L268]
{code:scala}
      <td>
        <span class="description-input">
          {info.statement}
        </span>
      </td>
{code}
Bug: Spark on K8s: Driver pod keeps running when executor allocator terminates with Fatal exception
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/21 20:09
Affects Version/s: 2.4.8

In Spark on Kubernetes, the class ExecutorPodsSnapshotsStoreImpl creates a thread which is responsible for creating new executor pods. The thread catches all 'NonFatal' exceptions, logs them and ignores these NonFatal exceptions. However, from Fatal exceptions, it only handles {color:#20999d}IllegalArgumentException {color}exception and terminates the driver pod in that case. Other Fatal exceptions are not handled at all, which means that if such a Fatal exception occurs, the executor creation thread abruptly terminates, while the main thread keeps running. Thus, the Spark application/job would keep running indefinitely without making any progress.

To fix this, 2 of the possible options are:

*Option#1*: Fail the driver pod whenever any Fatal exception happens. However, this approach has following disadvantages:
 # A few number of executors may have already been created when this Fatal exception happens. These executors can still take the job to completion, although slower than expected as all executors were not launched. Thus, we would fail the job instead of letting the job succeed slowly.
 # JVM can sometimes recover from Fatal exceptions on its own. Thus, we are not giving a chance to driver pod to recover from failure, rather we are killing it on first occurrence of {{Fatal}} exception.

*Option#2*: Fail the driver pod only when there are 0 executors running currently

In this approach, we fail the driver pod only when number of currently running executors is 0. This is so that we don’t kill a job which can potentially complete. Thus, 1 single running executor can keep the driver pod from dying. This may mean that job may make very slow progress when actual number of requested executors is very large.

*Option#3*: Expose configurations which allows customers to control this behavior.

spark.kubernetes.driver.executor-creation.terminate-on-fatal-exception=<true|false>
Bug: Reading a valid JSON file causes a NPE
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 26/Aug/21 13:43
Affects Version/s: 3.1.2

Executing the following Scala code:
{code:java}
spark.read.option("multiLine", true).json("/mnt/reproduce/test.json").show
{code}
Using the following data in test.json:
{code:java}
[
  {"value": 42},
  {"value": 43},
  null,
  {"value": 44}
]
{code}
Causes the following failure:
{code:java}
Job aborted due to stage failure.
Caused by: FileReadException: Error while reading file dbfs:/mnt/reproduce/test.json.
Caused by: NullPointerException:
{code}
The documentation is not clear about the expected behavior but my personal expectation would be to get this instead of an exception:
{code:java}
+-----+
|value|
+-----+
|   42|
|   43|
| null|
|   44|
+-----+
{code}
Bug: Executors don't get rescheduled in standalone mode when worker dies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/21 12:36
Affects Version/s: 3.0.1

This is reproducible with an application that uses less cores than what are available on the workers:

E.g. with 1 application with 1 executor, when the worker with the executor is killed, the application will not get another executor assigned even if there are enough resources in the cluster. This seems to be a regression, caused by [https://github.com/apache/spark/commit/51de86baed0776304c6184f2c04b6303ef48df90#diff-ca694acef669f50f9b45ca0d32ab6f5a516270bb26b33c4abb704e2dc00a1a03] .

That causes an assertion error on the master because it get's an executorStateChange from 'RUNNING' to 'RUNNING' instead of 'FAILED':
{noformat}
2021-08-13 14:04:12,554 [dispatcher-event-loop-2] INFO : I have been elected leader! New state: ALIVE
2021-08-13 14:04:12,554 [dispatcher-event-loop-2] INFO : I have been elected leader! New state: ALIVE
2021-08-13 14:04:56,489 [dispatcher-event-loop-10] INFO : Registering worker 172.27.64.1:58636 with 12 cores, 30.7 GiB RAM
2021-08-13 14:04:59,949 [dispatcher-event-loop-6] INFO : Registering worker 172.27.64.1:58694 with 12 cores, 30.7 GiB RAM
2021-08-13 14:05:20,212 [dispatcher-event-loop-2] INFO : Registering app query-frontend-null-172.27.64.1
2021-08-13 14:05:20,212 [dispatcher-event-loop-2] INFO : Registered app query-frontend-null-172.27.64.1 with ID app-20210813140520-0000
2021-08-13 14:05:20,228 [dispatcher-event-loop-2] INFO : Launching executor app-20210813140520-0000/0 on worker worker-20210813140459-172.27.64.1-58694
2021-08-13 14:05:37,991 [dispatcher-event-loop-9] ERROR: Ignoring errorjava.lang.AssertionError: assertion failed: executor 0 state transfer from RUNNING to RUNNING is illegal at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.deploy.master.Master$$anonfun$receive$1.applyOrElse(Master.scala:323) at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
 {noformat}
Bug: ExecutorClassLoader no longer works with Http based Class Servers
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Aug/21 10:12
Affects Version/s: 3.1.2

There are 2 primary issues,
 * If the classServer uri is `{{http://host:port}}` instead of `{{http://host:port/}}` the getPath on the URI object returns empty string and subsequently causes Path creation to fail with Path cannot be empty++
 * This is a regression issue and transitively a hadoop issue too, the Http Filesystem fails with URI not absolute error unlike other filesystems when the path doesn't have scheme or authority. This used to work when there was http specific implementation and no longer work with Filesystem based implementation

https://issues.apache.org/jira/browse/HADOOP-17870
Bug: ORC vectorized reader should properly check maximal number of fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/21 03:55
Affects Version/s: 3.2.0

Debugged internally and found a bug where we should disable vectorized reader now based on schema recursively. Currently we check `schema.length` to be no more than `wholeStageMaxNumFields` to enable vectorization. `schema.length` does not take nested columns sub-fields into condition (i.e. view nested column same as primitive column). This check will be wrong when enabling vectorization for nested columns. We should follow same check from `WholeStageCodegenExec` to check sub-fields recursively. This will not cause correctness issue but will cause performance issue where we may enable vectorization for nested columns by mistake when nested column has a lot of sub-fields.
Bug: Special timestamp_ntz value should be converted in the session tz
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/21 16:19
Affects Version/s: 3.2.0

Currently, special timestamp_ntz strings like *today*, *tomorrow*, *yesterday* are converted to timestamp_ntz values using JVM timezone which is incorrect. The conversion should base on the session time zone. The examples below demonstrate the problem:

{code:sql}
$ export TZ="Europe/Amsterdam"
$ ./bin/spark-sql -S
spark-sql> select timestamp_ntz'now';
2021-08-25 18:12:36.233

spark-sql> set spark.sql.session.timeZone=America/Los_Angeles;
spark.sql.session.timeZone	America/Los_Angeles
spark-sql> select timestamp_ntz'now';
2021-08-25 18:14:40.547
{code}
The special timestamp_ntz strings are converted in "Europe/Amsterdam" tz, and the converter doesn't respect Spark settings.
Bug: Support foldable special datetime values in CAST
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/21 20:01
Affects Version/s: 3.2.0

The PR https://github.com/apache/spark/pull/32714 disallowed special datetime values in the CAST expression, and allowed them only in typed literals. So, the following code doesn't work anymore:

{code:sql}
spark-sql> select date('today');
NULL
spark-sql> select cast('today' as date);
NULL
{code}
Though, users can replace them by:
{code:sql}
spark-sql> select date'today';
2021-08-23
{code}

but the changes can break users apps. The purpose of this ticket is to support special datetime strings when the strings in casts are foldable.
Bug: Redact sensitive information in Spark Thrift Server log
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/21 22:04
Affects Version/s: 3.1.2

Spark Thrift Server logs query without sensitive information redaction in [org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.scala|https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala#L188]
{code:scala}
  override def runInternal(): Unit = {
    setState(OperationState.PENDING)
    logInfo(s"Submitting query '$statement' with $statementId")
{code}
Logs
{code:sh}
21/08/03 20:49:46 INFO SparkExecuteStatementOperation: Submitting query 'CREATE OR REPLACE TEMPORARY VIEW test_view
    USING org.apache.spark.sql.jdbc
    OPTIONS (
        url="jdbc:mysql://example.com:3306",
        driver="com.mysql.jdbc.Driver",
        dbtable="example.test",
        user="my_username",
        password="my_password"
    )' with 37e5d2cb-aa96-407e-b589-7cb212324100
21/08/03 20:49:46 INFO SparkExecuteStatementOperation: Running query with 37e5d2cb-aa96-407e-b589-7cb212324100
{code}
Bug: Spark HistoryPage may show 'NotFound' in some multiple attempts cases
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Aug/21 07:21
Affects Version/s: 2.4.8

Current historypage show a attemptId column in case of hasMultipleAttempts is true.

if hasMultipleAttempts is false, remove the attemptId column.

 

But, applications in yarn could be failed even not logged to spark application history.

 

(application_1628518360417_0029 is normal case. Don't have to consider it) 

*application_1628518360417_0028*'s attemptId is 2, but size of attempt in spark history is 1.

(First try failed to starting spark driver in cluster mode, and second one was succeeded.)

For this case, the attemptId column is needed.

 
{code:java}
[
{
"id": "application_1628518360417_0029",
"name": "Spark Pi",
"attempts": [
{
"attemptId": "1",
"startTime": "2021-08-25T05:20:15.521GMT",
"endTime": "2021-08-25T05:20:30.398GMT",
"lastUpdated": "2021-08-25T05:20:30.475GMT",
"duration": 14877,
"sparkUser": "elixir-kook",
"completed": true,
"appSparkVersion": "2.4.5",
"endTimeEpoch": 1629868830398,
"lastUpdatedEpoch": 1629868830475,
"startTimeEpoch": 1629868815521
}
]
},
{
"id": "application_1628518360417_0028",
"name": "Spark Pi",
"attempts": [
{
"attemptId": "2",
"startTime": "2021-08-25T05:19:22.850GMT",
"endTime": "2021-08-25T05:19:44.662GMT",
"lastUpdated": "2021-08-25T05:19:44.726GMT",
"duration": 21812,
"sparkUser": "elixir-kook",
"completed": true,
"appSparkVersion": "2.4.5",
"endTimeEpoch": 1629868784662,
"lastUpdatedEpoch": 1629868784726,
"startTimeEpoch": 1629868762850
}
]
},
....
]
{code}
  !Screen Shot 2021-08-25 at 2.35.30 PM.png!

 

attemptId should be exist.
Bug: Split the JSON/CSV option of datetime format to in read and in write
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/21 18:46
Affects Version/s: 3.2.0

This is a follow up of https://issues.apache.org/jira/browse/SPARK-36418. Need to split JSON and CSV options *dateFormat* and *timestampFormat*. In write, should be the same but in read the option shouldn't be set to a default value. In this way, DateFormatter and TimestampFormatter will use the CAST logic.
Bug: JacksonParser should throw exception when data type unsupported.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/21 11:16
Affects Version/s: 3.2.0

Currently, when set spark.sql.timestampType=TIMESTAMP_NTZ, the behavior is different between from_json and from_csv.

{code:java}
-- !query
select from_json('{"t":"26/October/2015"}', 't Timestamp', map('timestampFormat', 'dd/MMMMM/yyyy'))
-- !query schema
struct<from_json({"t":"26/October/2015"}):struct<t:timestamp_ntz>>
-- !query output
{"t":null}
{code}




{code:java}
-- !query
select from_csv('26/October/2015', 't Timestamp', map('timestampFormat', 'dd/MMMMM/yyyy'))
-- !query schema
struct<>
-- !query output
java.lang.Exception
Unsupported type: timestamp_ntz
{code}


We should make from_json throws exception too.
Bug: Special strings for dates are no longer automatically converted without the "date" keyword
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Aug/21 20:53
Affects Version/s: 3.1.2

Comparison of a date type and a special string fail with DBR 9.0.

Works in earlier versions, returns "false":
 {{select current_date > 'today'}}

With DBR 9.0 the same query returns "null"

Version that works in DBR 9.0:

{{select current_date > *date* 'today'}}

It's especially bad because people have written a lot of queries without using this keyword and now all of those queries just silently fail as the comparison returns a NULL.

Works (or rather fails) in the same way with all comparison operations.
Bug: Exceptions in NoSuchItemException.scala have to be case classes to preserve specific exceptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/21 20:21
Affects Version/s: 3.1.2

Exceptions in NoSuchItemException.scala are not case classes. This is causing issues because in Analyzer's [executeAndCheck|https://github.com/apache/spark/blob/888f8f03c89ea7ee8997171eadf64c87e17c4efe/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L196-L199] method always calls the `copy` method on the exception. However, since these exceptions are not case classes, the `copy` method was always delegated to `AnalysisException::copy`, which is not the specialized version
Bug: Add sphinx-plotly-directive in Spark release Dockerfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/21 09:22
Affects Version/s: 3.2.0

After https://github.com/apache/spark/pull/32726, Python doc build requires sphinx-plotly-directive.
We should install it from spark-rm/Dockerfile to make sure do-release-docker.sh can run successfully. 
Also, we should mention it in the README of docs.
Bug: the 'seconds' parameter of 'make_timestamp' should accept integer type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/21 11:00
Affects Version/s: 3.2.0

With ANSI mode, {{SELECT make_timestamp(1, 1, 1, 1, 1, 1)}} fails, because the 'seconds' parameter needs to be of type DECIMAL(8,6), and INT can't be implicitly casted to DECIMAL(8,6) under ANSI mode.

We should update the function {{make_timestamp}} to allow integer type 'seconds' parameter.
Bug: Environment variables part in config doc isn't properly documented.
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 18/Aug/21 03:17
Affects Version/s: 3.1.2

It says environment variables are not reflected through spark-env.sh in YARN cluster mode but I believe they are. I think this part of the document should be removed.

[https://spark.apache.org/docs/latest/configuration.html#environment-variables]
Bug: Store the RocksDB format in the checkpoint for a streaming query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 07:28
Affects Version/s: 3.2.0

RocksDB provides backward compatibility but it doesn't always provide forward compatibility. It's better to store the RocksDB format version in the checkpoint so that it would give us more information to provide the rollback guarantee when we upgrade the RocksDB version that may introduce incompatible change in a new Spark version.

A typical case is when a user upgrades their query to a new Spark version, and this new Spark version has a new RocksDB version which may use a new format. But the user hits some bug and decide to rollback. But in the old Spark version, the old RocksDB version cannot read the new format.

In order to handle this case, we will write the RocksDB format version to the checkpoint. When restarting from a checkpoint, we will force RocksDB to use the format version stored in the checkpoint. This will ensure the user can rollback their Spark version if needed.
Bug: Skip Retrieving keytab with SparkFiles.get if keytab found in the CWD of Yarn Container
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Aug/21 10:09
Affects Version/s: 3.1.0

Currently we have the logic to deal with the JDBC keytab provided by the "--files" option

{{if (keytabParam != null && FilenameUtils.getPath(keytabParam).isEmpty)}}
 \{{{}}
 {{}}{{val result = SparkFiles.get(keytabParam)}}
 {{}}{{logDebug(s"Keytab path not found, assuming --files, file name used on executor: $result")}}
 {{}}{{result}}
 {{}}} {{else {}}
 {{}}{{logDebug("Keytab path found, assuming manual upload")}}
 {{}}{{keytabParam}}
 {{}}}

Spark has already created the soft link for any file submitted by the "--files" option. Here is an example.

testusera1.keytab -> /var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/filecache/12/testusera1.keytab

 

So there is no need to call the SparkFiles.get to absolute path of the keytab file. We can directly use the variable `keytabParam` as the keytab file path.

 

Moreover, SparkFiles.get will get a wrong path of keytab for the driver in cluster mode. In cluster mode, the keytab is available at the following location for both the driver and executors

{{/var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/container_1628584679772_0030_01_000001/testusera1.keytab}}

but SparkFiles.get brings the following wrong location for the driver

/var/opt/hadoop/temp/nm-local-dir/usercache/testusera1/appcache/application_1628584679772_0003/spark-8fb0f437-c842-4a9f-9612-39de40082e40/userFiles-5075388b-0928-4bc3-a498-7f6c84b27808/testusera1.keytab
Bug: Add common class/trait for ANSI interval types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 16:12
Affects Version/s: 3.2.0

Currently, there are many places where we check both YearMonthIntervalType and DayTimeIntervalType in the same match case, like

{code:scala}
case _: YearMonthIntervalType | _: DayTimeIntervalType => false
{code}

Need to add new trait or abstract class that should be extended by YearMonthIntervalType and DayTimeIntervalType. So, we can transform the code above to:
{code:scala}
case _: AnsiIntervalType => false
{code}
Bug: Avoid finalizing when there's no push at all in a shuffle
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Aug/21 02:00
Affects Version/s: 3.2.0

When all partition data of a map output is bigger than spark.shuffle.push.maxBlockSizeToPush, there will be no push. When all map outputs don't have push, the shuffle doesn't have push. In that case, we don't need to launch the finalizing request.
Bug: Disallow comparison between Interval and String
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/21 09:34
Affects Version/s: 3.2.0

If a binary comparison contains interval type and string type, we can't decide which
interval type the string should be promoted as. There are many possible interval
types, such as year interval, month interval, day interval, hour interval, etc.