av9ash/GitBugs huggingface dataset:
Bug: Employ correct data type for web_returns and store_returns in TPCDS tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/21 16:36
Affects Version/s: 3.2.0


{noformat}
2.2.2 Datatype
2.2.2.1 Each column employs one of the following datatypes:
a) Identifier means that the column shall be able to hold any key value generated for that column.
b) Integer means that the column shall be able to exactly represent integer values (i.e., values in increments of
1) in the range of at least ( − 2n − 1) to (2n − 1 − 1), where n is 64.
c) Decimal(d, f) means that the column shall be able to represent decimal values up to and including d digits,
of which f shall occur to the right of the decimal place; the values can be either represented exactly or
interpreted to be in this range.
d) Char(N) means that the column shall be able to hold any string of characters of a fixed length of N.
Comment: If the string that a column of datatype char(N) holds is shorter than N characters, then trailing
spaces shall be stored in the database or the database shall automatically pad with spaces upon retrieval such
that a CHAR_LENGTH() function will return N.
e) Varchar(N) means that the column shall be able to hold any string of characters of a variable length with a
maximum length of N. Columns defined as "varchar(N)" may optionally be implemented as "char(N)".
f) Date means that the column shall be able to express any calendar day between January 1, 1900 and
December 31, 2199.
2.2.2.2 The datatypes do not correspond to any specific SQL-standard datatype. The definitions are provided to
highlight the properties that are required for a particular column. The benchmark implementer may employ any internal representation or SQL datatype that meets those requirements.
{noformat}

one thing might be clear that we should replace bigint type which is now used in web_returns and store_returns with int type.
another thing that might need to be further discussed is - shall we use bigint to meet 2.2.2.1 b)?
Bug: Task Scheduling Plugin in Spark
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Apr/21 06:37
Affects Version/s: 3.2.0

Spark scheduler schedules tasks to executors in an arbitrary way. The schedule schedules the tasks by itself. Although there is locality configuration, the configuration is used for data locality purposes. Generally we cannot suggest the scheduler where a task should be scheduled to. Normally it is not a problem because the general task is executor-agnostic. But for special tasks, for example stateful tasks in Structured Streaming, state store is maintained at the executor side. Changing task location means reloading checkpoint data from the last batch. It has disadvantages from the performance perspective and also casts some limitations when we want to implement advanced features in Structured Streaming.
Bug: Spark App in container will not exit when exception happen
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Apr/21 01:00
Affects Version/s: 3.1.1

when submitting an app on k8s, someone can using an unauthed account to submit, and exception happened. But the app will not exit until 5 minutes later.
the log:
 !screenshot-1.png! 

the trace:
 [^21664.stack] 

I have create a PR:  https://github.com/apache/spark/pull/32101. Maybe help
Bug: Format ANSI intervals in Hive style
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/21 07:34
Affects Version/s: 3.2.0

Hive:

{code:java}
0: jdbc:hive2://localhost:10000/default> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
+-----------------------+
|          _c0          |
+-----------------------+
| 1 01:02:03.000001000  |
+-----------------------+
{code}

Spark:

{code:java}
spark-sql> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
INTERVAL '1 01:02:03.000001' DAY TO SECOND
{code}

Need to align Spark's output to Hive.
Bug: Spark 2.4.x version does not support numeric
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Apr/21 10:40
Affects Version/s: 2.4.7

as said in this pr:  [https://github.com/apache/spark/pull/31891] , 

 

So seems the key difference between DEICMAL and NUMERIC are:
{quote}implementation-defined decimal precision equal to or greater than the value of the specified precision.
{quote}
Decimal can have _at least_ specified precision (which means can have more) whereas numeric should have exactly specified precision.

Spark's decimal satisfy both so I think NUMERIC as a synonym of DECIMAL makes sense.

 

my  code  needs to use NUMERIC type in spark2.4 version
Bug: Nested column pruning fails to extract case-insensitive struct field from array
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/21 06:40
Affects Version/s: 2.4.7

Under case-insensitive mode, nested column pruning rule cannot correctly push down extractor of a struct field of an array of struct, e.g.,

{code}
val query = spark.table("contacts").select("friends.First", "friends.MiDDle")
{code}
Bug: Improve performance for reading smallint in vectorized Parquet reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/21 07:30
Affects Version/s: 3.2.0

Currently {{VectorizedRleValuesReader}} reads short in the following way:
{code:java}
for (int i = 0; i < n; i++) {
  c.putShort(rowId + i, (short)data.readInteger());
}
{code}

For PLAIN encoding {{readInteger}} is done via:
{code:java}
public final int readInteger() {
  return getBuffer(4).getInt();
}
{code}
which means it needs to repeatedly call {{slice}} buffer which is more expensive than calling it once in a big chunk and then reading the ints out.
Bug: Spark driver mistakenly classifies OOM error of executor (on K8s pod) as framework error
Status: Open | Priority: Major | Resolution: 
Created: 09/Apr/21 09:58
Affects Version/s: 3.1.1

*Issue Description*

I'm having a Spark application with 1 driver (on bare metal) + 1 executor (on K8s) (please note it's just for testing purpose). The corresponding configuration can be found bellowing.

When I run task for loading & computing on a XML file; due to the size of XML file is large (which I intended to) the executor got OOM error

#Aborting due to java.lang.OutOfMemoryError: Java heap space
 # 
 # A fatal error has been detected by the Java Runtime Environment:
 # 
 # Internal Error (debug.cpp:308), pid=19, tid=0x00007fff765ae700
 # fatal error: OutOfMemory encountered: Java heap space

 

However, the driver doesn't recognize this error as task failure scenario. Instead, it consider this as a framework issue and continue retrying the task

INFO TaskSchedulerImpl:57 - Executor 1 on 10.87.88.44 killed by driver.
 INFO TaskSetManager:57 - task 0.0 in stage 0.0 (TID 0) failed because while it was being computed, its executor exited for a reason unrelated to the task. *{color:#de350b}Not counting this failure towards the maximum number of failures for the task{color}.*
 INFO BlockManagerMasterEndpoint:57 - Trying to remove executor 1 from BlockManagerMaster.

 

This results in the fact that, the Spark application keeps retrying the task forever and locks other tasks from running

 

*Expectation* 

Spark driver should classify OOM on executor pod as a failure due to task and increase the count of max failure time

 

*Configuration*

spark.kubernetes.container.image: "spark_image_path"
 spark.kubernetes.container.image.pullPolicy: "Always"
 spark.kubernetes.namespace: "qa-namespace"
 spark.kubernetes.authenticate.driver.serviceAccountName: "svc-account"
 spark.kubernetes.executor.request.cores: "2"
 spark.kubernetes.executor.limit.cores: "2"
 spark.executorEnv.SPARK_ENV: "dev"
 spark.executor.memoryOverhead: "1G"
 spark.executor.memory: "6g"
 spark.executor.cores: "2"
 spark.executor.instances: "3"
 spark.driver.maxResultSize: "1g"
 spark.driver.memory: "10g"
 spark.driver.cores: "2"
 spark.eventLog.enabled: 'true'
 spark.driver.extraJavaOptions: "-Dcom.sun.management.jmxremote \
 -Dcom.sun.management.jmxremote.authenticate=false \
 -Dcom.sun.management.jmxremote.ssl=false \
 -XX:+UseG1GC \
 -XX:+PrintFlagsFinal \
 -XX:+PrintReferenceGC -verbose:gc \
 -XX:+PrintGCDetails \
 -XX:+PrintGCTimeStamps \
 -XX:+PrintAdaptiveSizePolicy \
 -XX:+UnlockDiagnosticVMOptions \
 -XX:+G1SummarizeConcMark \
 -XX:InitiatingHeapOccupancyPercent=35 \
 -XX:ConcGCThreads=20 \
 -XX:+PrintGCCause \
 -XX:+AlwaysPreTouch \
 -Dlog4j.debug=true -Dlog4j.configuration=[file:///].... "
 spark.sql.session.timeZone: UTC
Bug: Fix git error when pushing the tag after release script succeeds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/21 18:29
Affects Version/s: 2.4.8

When I ran release script for cutting 2.4.8 RC1, either in dry-run or normal run at the last step "push the tag after success", I encounter the following error:

{code}
fatal: Not a git repository (or any parent up to mount parent ....)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
{code}
Bug: Explicit representation of star in MergeIntoTable's Update and Insert action
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/21 23:36
Affects Version/s: 3.1.1

Currently, UpdateAction and InsertAction in the MergeIntoTable implicitly represent `update set *` and `insert *` with empty assignments. That means there is no way to differentiate between the representations of "update all columns" and "update no columns". For SQL MERGE queries, this inability does not matter because the SQL MERGE grammar that generated the MergeIntoTable plan does not allow "update no columns". However, other ways of generating the MergeIntoTable plan may not have that limitation, and may want to allow specifying "update no columns".  For example, in the Delta Lake project we provide a type-safe Scala API for Merge, where it is perfectly valid to produce a Merge query with an update clause but no update assignments. Currently, we cannot use MergeIntoTable to represent this plan, thus complicating the generation, and resolution of merge query from scala API. 

This should be fixed by having an explicit representation of * in the UpdateAction and InsertAction.
Bug: Block unsupported correlated scalar subquery in Aggregate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/21 20:08
Affects Version/s: 3.2.0

Currently, Spark supports Aggregate to host correlated scalar subqueries, but in some cases, those subqueries cannot be rewritten properly in the `RewriteCorrelatedScalarSubquery` rule. The error messages are also confusing. Hence we should block these cases in CheckAnalysis.
  

Case 1: correlated scalar subquery in the grouping expressions but not in aggregate expressions 
{code:java}
SELECT SUM(c2) FROM t t1 GROUP BY (SELECT SUM(c2) FROM t t2 WHERE t1.c1 = t2.c1)
{code}
We get this error:
{code:java}
java.lang.AssertionError: assertion failed: Expects 1 field, but got 2; something went wrong in analysis  
{code}
because the correlated scalar subquery is not rewritten properly: 
{code:java}
== Optimized Logical Plan ==
Aggregate [scalar-subquery#5 [(c1#6 = c1#6#93)]], [sum(c2#7) AS sum(c2)#11L]
:  +- Aggregate [c1#6], [sum(c2#7) AS sum(c2)#15L, c1#6 AS c1#6#93]
:     +- LocalRelation [c1#6, c2#7]
+- LocalRelation [c1#6, c2#7]
{code}
 

Case 2: correlated scalar subquery in the aggregate expressions but not in the grouping expressions 
{code:java}
SELECT (SELECT SUM(c2) FROM t t2 WHERE t1.c1 = t2.c1), SUM(c2) FROM t t1 GROUP BY c1
{code}
We get this error:
{code:java}
java.lang.IllegalStateException: Couldn't find sum(c2)#69L in [c1#60,sum(c2#61)#64L]
{code}
because the transformed correlated scalar subquery output is not present in the grouping expression of the Aggregate:
{code:java}
== Optimized Logical Plan ==
Aggregate [c1#60], [sum(c2)#69L AS scalarsubquery(c1)#70L, sum(c2#61) AS sum(c2)#65L]
+- Project [c1#60, c2#61, sum(c2)#69L]
   +- Join LeftOuter, (c1#60 = c1#60#95)
      :- LocalRelation [c1#60, c2#61]
      +- Aggregate [c1#60], [sum(c2#61) AS sum(c2)#69L, c1#60 AS c1#60#95]
         +- LocalRelation [c1#60, c2#61]
{code}
Bug: Redact map-type options in the output of explain()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/21 13:32
Affects Version/s: 3.0.0

The `explain()` method prints the arguments of tree nodes in logical/physical plans. The arguments could contain a map-type option which contains sensitive data.
We should map-type options in the output of explain(), otherwise we will see sensitive data in explain output or Spark UI.
Bug: skip.header.line.count dose not work in hive partitioned table
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/21 02:12
Affects Version/s: 2.1.1

when I use hive2.4.0 create external hive partitioned table with csv file, and 
{code:java}
TBLPROPERTIES (
'skip.header.line.count' = '1'
)
LOCATION 'XXXX'{code}
 

the result of hive dosen't contain header, but result of spark also contain header
Bug: Followup for Refactor TreeNode's children handling methods into specialized traits (SPARK-34906)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/21 13:26
Affects Version/s: 3.1.1

This is a followup for https://issues.apache.org/jira/browse/SPARK-34906

In this PR we:
 * Introduce the QuaternaryLike trait for node types with 4 children.
 * Specialize more node types
 * Fix a number of style errors that were introduced in the original PR.
Bug: pyspark 2.4.7 in conda
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Apr/21 18:03
Affects Version/s: 2.4.7

Pyspark 2.4.7 does not seem to be available yet in conda: [https://anaconda.org/conda-forge/pyspark/files] shows only versions up to 2.4.6 (for major version 2).
{noformat}
conda install pyspark=2.4.7{noformat}
gives the following:

 

 
{code:java}
ResolvePackageNotFound:
13:44:43
    - pyspark=2.4.7
{code}
It would be great if this package were available through conda!
Bug: Support multiple fields when nested column pruning on Generator output 
Status: Open | Priority: Major | Resolution: 
Created: 05/Apr/21 06:22
Affects Version/s: 3.2.0

SPARK-34638 enhances nested column pruning rule on Generator's output. SPARK-34638 supports single field case, e.g. {{df.select(explode($"items").as("item")).select($"item.a")}}. This ticket is open for tracking multiple-field support.
Bug: Use zstd codec name in ORC file names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/21 19:32
Affects Version/s: 3.2.0

Like the other ORC supported codecs, we had better have `zstd` in the Spark generated ORC file names.

*SNAPPY*
{code}
scala> spark.range(10).repartition(1).write.option("compression", "snappy").orc("/tmp/snappy")

$ ls -al /tmp/snappy 
total 24
drwxr-xr-x   6 dongjoon  wheel  192 Apr  4 12:17 .
drwxrwxrwt  14 root      wheel  448 Apr  4 12:17 ..
-rw-r--r--   1 dongjoon  wheel    8 Apr  4 12:17 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel   12 Apr  4 12:17 .part-00000-833bb7ad-d1e1-48cc-9719-07b2d594aa4c-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    0 Apr  4 12:17 _SUCCESS
-rw-r--r--   1 dongjoon  wheel  231 Apr  4 12:17 part-00000-833bb7ad-d1e1-48cc-9719-07b2d594aa4c-c000.snappy.orc
{code}

*ZSTD (AS-IS)*
{code}                                                                                
scala> spark.range(10).repartition(1).write.option("compression", "zstd").orc("/tmp/zstd")

$ ls -al /tmp/zstd  
total 24
drwxr-xr-x   6 dongjoon  wheel  192 Apr  4 12:17 .
drwxrwxrwt  14 root      wheel  448 Apr  4 12:17 ..
-rw-r--r--   1 dongjoon  wheel    8 Apr  4 12:17 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel   12 Apr  4 12:17 .part-00000-2f403ce9-7314-4db5-bca3-b1c1dd83335f-c000.orc.crc
-rw-r--r--   1 dongjoon  wheel    0 Apr  4 12:17 _SUCCESS
-rw-r--r--   1 dongjoon  wheel  231 Apr  4 12:17 part-00000-2f403ce9-7314-4db5-bca3-b1c1dd83335f-c000.orc
{code}
Bug: Upgrade Janino from 3.0.16 to 3.1.4
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Apr/21 06:18
Affects Version/s: 3.2.0

From the [change log|http://janino-compiler.github.io/janino/changelog.html],  the janino 3.0.x line has been deprecated,  we can use 3.1.x line instead of it.
Bug: Support TPCDSQueryBenchmark in Benchmarks
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Apr/21 02:49
Affects Version/s: 3.2.0

Benchmarks.scala currently does not support TPCDSQueryBenchmark. We should make it supported. See also https://github.com/apache/spark/pull/32015#issuecomment-811119046
Bug: CTE Execution fails for Sql Server
Status: Open | Priority: Minor | Resolution: 
Created: 01/Apr/21 02:49
Affects Version/s: 3.0.1

h2. Issue

We have a simple Sql statement that we intend to execute on SQL Server. This has a CTE component.

Execution of this yields to an error that looks like follows
{code:java}
java.sql.SQLException: Incorrect syntax near the keyword 'WITH'.{code}
We are using the jdbc driver *net.sourceforge.jtds.jdbc.Driver* (version 1.3.1)

This is a particularly annoying issue and due to this we are having to write inner queries that are fair bit inefficient.
h2. SQL statement

(not the actual one but a simplified version with renamed parameters)

 
{code:sql}
WITH OldChanges as (
   SELECT distinct 
        SomeDate,
        Name
   FROM [dbo].[DateNameFoo] (nolock)
   WHERE SomeDate!= '2021-03-30'
       AND convert(date, UpdateDateTime) = '2021-03-31'

SELECT * from OldChanges {code}
h3. Update on 2021-04-21

We tried *com.microsoft.sqlserver.jdbc.SQLServerDriver* driver as well. This also yields to the same issue.
Bug: Rpc env not shutdown when shutdown method call by endpoint onStop
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 20/Apr/21 17:10
Affects Version/s: 3.0.0

when i use this code to work,  Rpc thread hangs up and not close gracefully. i think when rpc thread called shutdown on OnStop method, it will try to put MessageLoop.PoisonPill to return and stop thread in rpc pool. In spark 3.x, it will make others thread return & stop but current thread which call OnStop method to await current pool to stop. it makes current thread not stop, and pending program.

I'm not sure that needs to be improved or not?

 
{code:java}
//代码占位符{code}
test("Rpc env not shutdown when shutdown method call by endpoint onStop") {
 val rpcEndpoint = new RpcEndpoint {
    override val rpcEnv: RpcEnv = env
    override def onStop(): Unit = {
      env.shutdown()
 env.awaitTermination()         
 }
    override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
 case m => context.reply(m)
 }
  }
  env.setupEndpoint("test", rpcEndpoint)
  rpcEndpoint.stop()
  env.awaitTermination()
}
Bug:  Raise TypeError in inappropriate type case rather than ValueError
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Apr/21 14:47
Affects Version/s: 3.2.0

There are many wrong error type usages on ValueError type.

When an operation or function is applied to an object of inappropriate type, we should use TypeError rather than ValueError.

such as:

[https://github.com/apache/spark/blob/355c39939d9e4c87ffc9538eb822a41cb2ff93fb/python/pyspark/sql/dataframe.py#L1137]

[https://github.com/apache/spark/blob/355c39939d9e4c87ffc9538eb822a41cb2ff93fb/python/pyspark/sql/dataframe.py#L1228]

 

We should do some correction in some right time, note that if we do these corrections, it will break some catch on original ValueError.

 

[1] https://docs.python.org/3/library/exceptions.html#TypeError
Bug: Pyspark asDict() returns wrong child field for nested dataframe
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 07/Apr/21 22:13
Affects Version/s: 3.0.1

Hello! I upgraded a job to Spark 3.0.1 (from 2.4.4) and encountered this issue. The job uses asDict(True) in pyspark. I reproduced the issue with a concise schema and code. Consider this example schema:
{code:java}
root
 |-- id: integer (nullable = false)
 |-- struct_1: struct (nullable = true)
 | |-- array_1_1: array (nullable = true)
 | | |-- element: string (containsNull = false)
 |-- struct_2: struct (nullable = true)
 | |-- array_2_1: array (nullable = true)
 | | |-- element: string (containsNull = false){code}
I created 100 rows with the above schema filled it with some numbers and checked the row.asDict(True) against the input. For some rows
{code:java}
struct_1.array_1_1{code}
is missing. Instead I get
{code:java}
struct_1.array_2_1{code}
And I also observe this happens when array_1_1 is null. Example assert failure:
{code:java}
AssertionError: {'id': 7, 'struct_1': {'array_2_1': None}, 'struct_2': {'array_2_1': None}} != {'id': 7, 'struct_1': {'array_1_1': None}, 'struct_2': {'array_2_1': None}}

{code}
 I have attached a minimal script that reproduces this issue.
Bug: setup.py will copy different version netty jars into deps/jars when execute PySpark pip packaging tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Apr/21 06:48
Affects Version/s: 3.2.0

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137500/console]

 
copying deps/jars/netty-all-4.1.51.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-buffer-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-codec-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-common-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-handler-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-resolver-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-transport-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-transport-native-epoll-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
 

Master relies on netty 4.1.51 but there will copy 4.1.51 and 4.1.50 to pyspark-3.2.0.dev0/deps/jars

 

-------------------------------------

 

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137547/console]

 
copying deps/jars/netty-all-4.1.63.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-buffer-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-codec-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-common-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-handler-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-resolver-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-transport-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
copying deps/jars/netty-transport-native-epoll-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars
 

SPARK-35132 try to upgrade netty to 4.1.63.Final, but  still copy 4.1.63 and 4.1.50 to pyspark-3.2.0.dev0/deps/jars
Bug: Spark UI Stages->Tasks should show local launch time
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 14/Apr/21 02:15
Affects Version/s: 3.1.1

Hello!

The launch time of the task displayed on the Spark Stage tab is the UTC time zone, which confuses users in non-UTC time zones. After checking some of the code, it seems that it can be solved by modifying the following code.

[File involved |https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js]

 

Current :
{code:java}
// code placeholder
function formatDate(date) {
  if (date <= 0) return "-";
  else {
     var dt = new Date(date.replace("GMT", "Z"));
     return formatDateString(dt);
  }
}

{code}
Updated (example):
{code:java}
// code placeholder
function formatDate(date) {
    if (date <= 0) {
        return "-";
    }
    else {
        let local = new Date();
        let gmtTime = new Date(date.split(".")[0].replace("T", " "));
        let res = new Date(gmtTime.getTime() - (local.getTimezoneOffset() * 60000)).getTime();
        return formatTimeMillis(res);
    }
}
{code}