av9ash/GitBugs huggingface dataset:
Bug: NestedColumnAliasing support pruning adjacent projects
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Feb/23 02:51
Affects Version/s: 3.5.0

CollapseProject won't combine adjacent projects into one, e.g. non-cheap expression has been accessed more than once with the below project. Then there would be possible to appear some adjacent project nodes that NestedColumnAliasing does not support pruning.
Bug: scala: accept user_agent in spark connect's connection string
Status: Open | Priority: Major | Resolution: 
Created: 20/Feb/23 15:30
Affects Version/s: 3.3.2

Currently, the Spark Connect service's {{client_type}} attribute (which is really user agent) is set to {{_SPARK_CONNECT_PYTHON}} to signify PySpark.

Accept an optional {{user_agent}} parameter in the connection string and plumb this down to the Spark Connect service.

This enables partners using Spark Connect to set their application as the user agent,
which then allows visibility and measurement of integrations and usages of spark
connect.

This is already done for the Python client: https://github.com/apache/spark/commit/b887d3de954ae5b2482087fe08affcc4ac60c669
Bug:  python: accept user_agent in spark connect's connection string
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 11:44
Affects Version/s: 3.3.2

Currently, the Spark Connect service's {{client_type}} attribute (which is really user agent) is set to {{_SPARK_CONNECT_PYTHON}} to signify PySpark.

Accept an optional {{user_agent}} parameter in the connection string and plumb this down to the Spark Connect service.

This enables partners using Spark Connect to set their application as the user agent,
which then allows visibility and measurement of integrations and usages of spark
connect.
Bug: reduce spark connect service retry time
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 20/Feb/23 07:39
Affects Version/s: 3.3.2

https://github.com/apache/spark/blob/5fc44dabe5084fb784f064afe691951a3c270793/python/pyspark/sql/connect/client.py#L411

 

Currently, 15 retries with the current backoff strategy result in the client sitting in
the retry loop for ~400 seconds in the worst case. This means, applications and
users using the spark connect client will hang for >6 minutes with no response.
Bug: Upgrade Netty to 4.1.89
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/23 11:32
Affects Version/s: 3.5.0

This release contains a fix for two regressions that were introduced by 4.1.88.Final:
 * Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) ([#13216|https://github.com/netty/netty/pull/13216])
 * dyld: Symbol not found: _netty_jni_util_JNI_OnLoad when upgrading from 4.1.87.Final to 4.1.88.Final ([#13214|https://github.com/netty/netty/pull/13214])
Bug: why executor memory used is shown greater than total available memory on spark ui
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/23 13:48
Affects Version/s: 2.4.5

*I have a spark  streaming job that is running for around last 3 weeks. When I open the Executors tab on spark web UI, it shows*
 # {{memory used - 36.1GB}}
 # total available memory for storage - 3.2GB

*Please refer to the below screenshot of Spark UI*

!https://i.stack.imgur.com/nmk39.jpg!
Bug: ProtoToPlanTestSuite should analyze its input plans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 17:37
Affects Version/s: 3.4.0

ProtoToPlanTestSuite should analyze its input plans. We are moving to testing functions, and most of these are left as unresolved function by the spark planner. This is not enough to ensure that we are in fact binding to the correct function.
Bug: dataset.where() omit quotes if where IN clause has more than 10 operands
Status: Open | Priority: Major | Resolution: 
Created: 15/Feb/23 15:43
Affects Version/s: 3.3.0

dataset.where()/filter() omit string quotes if where IN clause has more than 10 operands. With datasourceV1 works as expected. 
Attached files: java-code.txt, stacktrace.txt, sql.txt
 - Spark verison 3.3.0
 - Scala version 2.12
 - DatasourceV2
 - Postgres
 - Postrgres JDBC Driver: 42+
 - Java8
Bug: Consolidating the PySpark version upgrade note pages into a single page to make it easier to read
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/23 21:53
Affects Version/s: 3.4.0

Creating a new PySpark migration guide sub page and consolidating the existing 9 separate pages into this one new page. This makes it easier to take a look across multiple version upgrades by simply scrolling on the page.

Also, this is similar to the Spark Core Migration Guide page here:

[https://spark.apache.org/docs/latest/core-migration-guide.html]

 

Updating the existing main Migration Guide page to point to this new sub page and also making some minor language updates to help readers.
Bug: insertInto fails when the column names are different from the table columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/23 22:57
Affects Version/s: 3.4.0

{noformat}
File "/.../python/pyspark/sql/connect/readwriter.py", line 518, in pyspark.sql.connect.readwriter.DataFrameWriter.insertInto
Failed example:
    df.selectExpr("age AS col1", "name AS col2").write.insertInto("tblA")
Exception raised:
    Traceback (most recent call last):
      File "/.../lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.readwriter.DataFrameWriter.insertInto[3]>", line 1, in <module>
        df.selectExpr("age AS col1", "name AS col2").write.insertInto("tblA")
      File "/.../python/pyspark/sql/connect/readwriter.py", line 477, in insertInto
        self.saveAsTable(tableName)
      File "/.../python/pyspark/sql/connect/readwriter.py", line 495, in saveAsTable
        self._spark.client.execute_command(self._write.command(self._spark.client))
      File "/.../python/pyspark/sql/connect/client.py", line 553, in execute_command
        self._execute(req)
      File "/.../python/pyspark/sql/connect/client.py", line 648, in _execute
        self._handle_error(rpc_error)
      File "/.../python/pyspark/sql/connect/client.py", line 718, in _handle_error
        raise convert_exception(info, status.message) from None
    pyspark.errors.exceptions.connect.AnalysisException: Cannot resolve 'age' given input columns: [col1, col2].
{noformat}
Bug: Mask function's generated code does not handle null input
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/23 16:46
Affects Version/s: 3.4.0

Example:
{noformat}
create or replace temp view v1 as
select * from values
(null),
('AbCD123-@$#')
as data(col1);

cache table v1;

select mask(col1) from v1;
{noformat}
This query results in a {{NullPointerException}}:
{noformat}
23/02/07 16:36:06 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
{noformat}
The generated code calls {{UnsafeWriter.write(0, value_0)}} regardless of whether {{Mask.transformInput}} returns null or not. The {{UnsafeWriter.write}} method for {{UTF8String}} does not expect a null pointer.
{noformat}
/* 031 */     boolean isNull_1 = i.isNullAt(0);
/* 032 */     UTF8String value_1 = isNull_1 ?
/* 033 */     null : (i.getUTF8String(0));
/* 034 */
/* 035 */
/* 036 */
/* 037 */
/* 038 */     UTF8String value_0 = null;
/* 039 */     value_0 = org.apache.spark.sql.catalyst.expressions.Mask.transformInput(value_1, ((UTF8String) references[0] /* literal */), ((UTF8String) references[1] /* literal */), ((UTF8String) references[2] /* literal */), ((UTF8String) references[3] /* literal */));;
/* 040 */     if (false) {
/* 041 */       mutableStateArray_0[0].setNullAt(0);
/* 042 */     } else {
/* 043 */       mutableStateArray_0[0].write(0, value_0);
/* 044 */     }
/* 045 */     return (mutableStateArray_0[0].getRow());
/* 046 */   }
{noformat}

The bug is not exercised by a literal null input value, since there appears to be some optimization that simply replaces the entire function call with a null literal:
{noformat}
spark-sql> explain SELECT mask(NULL);
== Physical Plan ==
*(1) Project [null AS mask(NULL, X, x, n, NULL)#47]
+- *(1) Scan OneRowRelation[]

Time taken: 0.026 seconds, Fetched 1 row(s)
spark-sql> SELECT mask(NULL);
NULL
Time taken: 0.042 seconds, Fetched 1 row(s)
spark-sql> 
{noformat}
Bug: Fix SparkR install.spark function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/23 05:50
Affects Version/s: 3.3.0

{code}
$ R

R version 4.2.1 (2022-06-23) -- "Funny-Looking Kid"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(SparkR)

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked from ‘package:base’:

    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
    rank, rbind, sample, startsWith, subset, summary, transform, union

> install.spark()
Spark not found in the cache directory. Installation will start.
MirrorUrl not provided.
Looking for preferred site from apache website...
Preferred mirror site found: https://dlcdn.apache.org/spark
Downloading spark-3.3.2 for Hadoop 2.7 from:
- https://dlcdn.apache.org/spark/spark-3.3.2/spark-3.3.2-bin-hadoop2.7.tgz
trying URL 'https://dlcdn.apache.org/spark/spark-3.3.2/spark-3.3.2-bin-hadoop2.7.tgz'
simpleWarning in download.file(remotePath, localPath): downloaded length 0 != reported length 196
{code}
Bug: Use spark.sql.timestampType for data source inference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/23 23:27
Affects Version/s: 3.4.0

With the configuration `spark.sql.timestampType`,  TIMESTAMP in Spark is a user-specified alias associated with one of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. This is quite complicated to Spark users.

There is another option `spark.sql.sources.timestampNTZTypeInference.enabled` for schema inference. I would like to introduce it in [https://github.com/apache/spark/pull/40005] but having two flags seems too much. After thoughts, I decide to merge `spark.sql.sources.timestampNTZTypeInference.enabled` into `spark.sql.timestampType` and let  `spark.sql.timestampType` control the schema inference behavior.

We can have followups to add data source options "inferTimestampNTZType" for CSV/JSON/partiton column like JDBC data source did.
Bug: Updating PySpark documentation to enhance usability
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/23 07:33
Affects Version/s: 3.4.0

Updates to the PySpark documentation web site:
 * Fixing typo on the Getting Started page (Version => Versions)
 * Capitalizing "In/Out" in the DataFrame Quick Start notebook
 * Adding "(Legacy)" to the Spark Streaming heading on the Spark Streaming page
 * Reorganizing the User Guide page to list PySpark guides first + minor language updates
Bug: Incorrect results or NPE when inserting null value into array using array_insert/array_append
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/23 23:18
Affects Version/s: 3.4.0

Example:
{noformat}
create or replace temp view v1 as
select * from values
(array(1, 2, 3, 4), 5, 5),
(array(1, 2, 3, 4), 5, null)
as v1(col1,col2,col3);

select array_insert(col1, col2, col3) from v1;
{noformat}
This produces an incorrect result:
{noformat}
[1,2,3,4,5]
[1,2,3,4,0] <== should be [1,2,3,4,null]
{noformat}
A more succint example:
{noformat}
select array_insert(array(1, 2, 3, 4), 5, cast(null as int));
{noformat}
This also produces an incorrect result:
{noformat}
[1,2,3,4,0] <== should be [1,2,3,4,null]
{noformat}
Another example:
{noformat}
create or replace temp view v1 as
select * from values
(array('1', '2', '3', '4'), 5, '5'),
(array('1', '2', '3', '4'), 5, null)
as v1(col1,col2,col3);

select array_insert(col1, col2, col3) from v1;
{noformat}
The above query throws a {{NullPointerException}}:
{noformat}
23/02/10 11:08:05 ERROR SparkSQLDriver: Failed in [select array_insert(col1, col2, col3) from v1]
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.LocalTableScanExec.$anonfun$unsafeRows$1(LocalTableScanExec.scala:44)
{noformat}
{{array_append}} has the same issue:
{noformat}
spark-sql> select array_append(array(1, 2, 3, 4), cast(null as int));
[1,2,3,4,0] <== should be [1,2,3,4,null]
Time taken: 3.679 seconds, Fetched 1 row(s)
spark-sql> select array_append(array('1', '2', '3', '4'), cast(null as string));
23/02/10 11:13:36 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.project_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
{noformat}
Bug: Updating PySpark documentation to support new users better
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/23 06:25
Affects Version/s: 3.4.0

This is the first of a series of updates to the PySpark documentation site to better guide new users on what to use and when as well as help improve discoverability of related pages/resources.
 * Add "Overview" to the top navigation bar to make it easy to get back to the main page (clicking the logo is not super discoverable)
 * Break architecture image into separate, clickable parts for easy navigation to information for each part
 * Added links to related topics under each area description
 * Added date and version to the page
Bug: Standardize __repr__ of CommonInlineUserDefinedFunction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/23 03:14
Affects Version/s: 3.4.0

As shown below, `f(df.id)` is evaluated to a Column with a super long representation in Connect, however, the vanilla PySpark returns `Column<'<lambda>(id)'>`. We shall Standardize __repr__ of CommonInlineUserDefinedFunction.

{code:python}
>>> f = udf(lambda x : x + 1)
>>> df.id
Column<'id'>
>>> f(df.id)
Column<'<lambda>(id), True, "string", 100, b'\x80\x05\x95\xe1\x01\x00\x00\x00\x00\x00\x00\x8c\x1fpyspark.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x01K\x02KCC\x08|\x00d\x01\x17\x00S\x00\x94NK\x01\x86\x94)\x8c\x01x\x94\x85\x94\x8c\x07<stdin>\x94\x8c\x08<lambda>\x94K\x01C\x00\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94uNNNt\x94R\x94\x8c$pyspark.cloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x16}\x94}\x94(h\x13h\r\x8c\x0c__qualname__\x94h\r\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x14\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\x11pyspark.sql.types\x94\x8c\nStringType\x94\x93\x94)\x81\x94\x86\x94.', f3.9'>
{code}
Bug: Support setting location for creating external in DataFrameWriterV2
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/23 07:54
Affects Version/s: 3.3.1

It not possible to set location when creating a table with DataFrameWriterV2 and therefore no external tables can be created with the new interface.

Proposed solution: create a method location in CreateTableWriter interface.
Bug: Support complex return types in DDL strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 05:51
Affects Version/s: 3.4.0

{code}
# Spark Connect
>>> spark.range(2).select(udf(lambda x: (x, x), "struct<x:integer, y:integer>")("id"))
...
AssertionError: returnType should be singular

# vanilla PySpark
>>> spark.range(2).select(udf(lambda x: (x, x), "struct<x:integer, y:integer>")("id"))
DataFrame[<lambda>(id): struct<x:int,y:int>]

{code}
Bug: Make suggestion in error message smarter
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/23 04:39
Affects Version/s: 3.4.0

Current suggestion message e.g. `UNRESOLVED_COLUMN.WITH_SUGGESTION` just pumps out all columns, so it would not very helpful for users.

This can be better to show most related columns only to user facing error message.
Bug: The code logic of the configmap max size validation lacks extra content
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Feb/23 09:55
Affects Version/s: 3.5.0

In each configmap, Spark adds an extra content in a fixed format,this extra content of the configmap is as belows:
  spark.kubernetes.namespace: default
  spark.properties: |
    #Java properties built from Kubernetes config map with name: spark-exec-b47b438630eec12d-conf-map
    #Wed Feb 08 20:10:19 CST 2023
    spark.kubernetes.namespace=default

But the max size validation code logic does not consider this part
Bug: Cleanup orphan sst and log files in RocksDB checkpoint directory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/23 07:54
Affects Version/s: 3.2.3

When RocksDB version.zip file get overwritten (e.g. concurrent task execution, task/stage/batch reattempts) or the zip file don't get uploaded successfully, the associated sst and log files don't get garbage collected.([https://github.com/databricks/runtime/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala|https://github.com/databricks/runtime/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala#L305-L309]) These files consume storage. We can clean up these SST files during periodic state store maintenance. The major concern is that sst files for ongoing version also appear to be "orphan" because they are uploaded before zip file, we have to be careful not to delete them.
Bug: Update Debugging PySpark documents to show error message properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 12:59
Affects Version/s: 3.4.0

The example in [https://spark.apache.org/docs/latest/api/python/development/debugging.html#debugging-pyspark] is outdated due to new PySpark error framework.

We should show proper example
Bug: Use FileSystem.exists in FileSystemBasedCheckpointFileManager.exists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/23 04:39
Affects Version/s: 3.5.0

Other methods in FileSystemBasedCheckpointFileManager already uses FileSystem.exists for all cases checking existence of the path. Use FileSystem.exists in FileSystemBasedCheckpointFileManager.exists, which is consistent with other methods in FileSystemBasedCheckpointFileManager.
Bug: distinct(count colname) with UNION ALL causes query analyzer bug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/23 08:52
Affects Version/s: 3.3.0

If you combine a UNION ALL with a count(distinct colname) you get a query analyzer bug.

 

This behaviour is introduced in 3.3.0.  The bug was not present in 3.2.1.

 

Here is a reprex in PySpark:

{{df_pd = pd.DataFrame([}}
{{    \{'surname': 'a', 'first_name': 'b'}}}
{{])}}
{{df_spark = spark.createDataFrame(df_pd)}}
{{df_spark.createOrReplaceTempView("input_table")}}

{{sql = """}}

{{SELECT }}
{{    (SELECT Count(DISTINCT first_name) FROM   input_table) }}
{{        AS distinct_value_count}}
{{FROM   input_table}}
{{UNION ALL}}
{{SELECT }}
{{    (SELECT Count(DISTINCT surname) FROM   input_table) }}
{{        AS distinct_value_count}}
{{FROM   input_table """}}

{{spark.sql(sql).toPandas()}}
Bug: Support left_outer join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 03:34
Affects Version/s: 3.4.0

```
>>> df = spark.range(1)
>>> df2 = spark.range(2)
>>> df.join(df2, how="left_outer")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/xinrong.meng/spark/python/pyspark/sql/connect/dataframe.py", line 438, in join
    plan.Join(left=self._plan, right=other._plan, on=on, how=how),
  File "/Users/xinrong.meng/spark/python/pyspark/sql/connect/plan.py", line 730, in __init__
    raise NotImplementedError(
NotImplementedError: 
                Unsupported join type: left_outer. Supported join types include:
                "inner", "outer", "full", "fullouter", "full_outer",
                "leftouter", "left", "left_outer", "rightouter",
                "right", "right_outer", "leftsemi", "left_semi",
                "semi", "leftanti", "left_anti", "anti", "cross",
```
Bug: Improve performance of HiveGenericUDTF by making inputProjection instantiate once
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 10:38
Affects Version/s: 3.4.0

{code:java}
+++ b/sql/hive/benchmarks/HiveUDFBenchmark-per-row-results.txt
@@ -0,0 +1,7 @@
+OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Hive UDTF benchmark:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+------------------------------------------------------------------------------------------------------------------------
+Hive UDTF dup 2                                    1574           1680         118          0.7        1501.1       1.0X
+Hive UDTF dup 4                                    2642           3076         588          0.4        2519.9       0.6X
+
diff --git a/sql/hive/benchmarks/HiveUDFBenchmark-results.txt b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
new file mode 100644
index 0000000000..8af8b6582c
--- /dev/null
+++ b/sql/hive/benchmarks/HiveUDFBenchmark-results.txt
@@ -0,0 +1,7 @@
+OpenJDK 64-Bit Server VM 1.8.0_352-bre_2022_12_13_23_06-b00 on Mac OS X 13.1
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Hive UDTF benchmark:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+------------------------------------------------------------------------------------------------------------------------
+Hive UDTF dup 2                                     712            789         101          1.5         678.7       1.0X
+Hive UDTF dup 4                                    1212           1294          78          0.9        1156.0       0.6X
+ {code}
over 2x performance gain via a benchmarking
Bug: Provide more details in ExecutorUpdated sent in Master.removeWorker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/23 10:44
Affects Version/s: 3.3.1

Currently field `message` in `ExecutorUpdated` sent in Master.removeWorker is always `Some("worker lost")`. We should provide more information in the message instead to better differentiate the cause of the worker removal.