av9ash/GitBugs huggingface dataset:
Bug: pyspark.pandas.DataFrame.drop drops dataframe if axis not specified
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/22 02:32
Affects Version/s: 3.3.0

import pyspark.pandas as pd
data = [\{"Category": 'A', "ID": 1, "Value": 121.44, "Truth": True},
        \{"Category": 'B', "ID": 2, "Value": 300.01, "Truth": False},
        \{"Category": 'C', "ID": 3, "Value": 10.99, "Truth": None},
        \{"Category": 'E', "ID": 4, "Value": 33.87, "Truth": True}
        ]
df = pd.DataFrame(data)
df.display()

--drops dataframe "Query returned no results"
df1=df.drop(["ID","Category"])
df1.display()

--works

df2=df.drop(["ID","Category"], 1)
df2.display()
Bug: Enable base image build in lint job and fix sparkr env
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 03:41
Affects Version/s: 3.4.0

Since sparkr 4.2.x has [below new change](https://github.com/r-devel/r-svn/blob/e6be1f6b14838016e78d6a91f48f21acec7fa4c4/doc/NEWS.Rd#L376):
  > Environment variables R_LIBS_USER and R_LIBS_SITE are both now set to the R system default if unset or empty, and can be set to NULL to indicate an empty list of user or site library directories.

lastest ubuntu pkg also sync this change and has some changes on /etc/R/Renviron ():
```
$ docker run -ti ghcr.io/yikun/apache-spark-github-action-image:sparkr-master-2569799176 cat /etc/R/Renviron | grep R_LIBS_SITE

R_LIBS_SITE=${R_LIBS_SITE:-'%S'}

$ docker run -ti ghcr.io/yikun/apache-spark-github-action-image:sparkr-master-2569799176 cat /etc/R/Renviron.site | grep R_LIBS_SITE

## edd Jul 2007  Now use R_LIBS_SITE, not R_LIBS
## edd Mar 2022  Now in Renviron.site reflecting R_LIBS_SITE
R_LIBS_SITE="/usr/local/lib/R/site-library/:${R_LIBS_SITE}:/usr/lib/R/library"
```

So, we add `R_LIBS_SITE` to ENV from `/etc/R/Renviron.site` to make sure search paths right for sparkr.

otherwise, even if we install the `lintr` will cause error like due to `R_LIBS_SITE` wrong set:
```
$ dev/lint-r
Loading required namespace: SparkR
Loading required namespace: lintr
Failed with error:  'there is no package called 'lintr''
Installing package into '/usr/lib/R/site-library'
(as 'lib' is unspecified)
Error in contrib.url(repos, type) :
  trying to use CRAN without setting a mirror
Calls: install.packages -> startsWith -> contrib.url
Execution halted
```

[1] https://cran.r-project.org/doc/manuals/r-devel/NEWS.html
[2] https://stat.ethz.ch/R-manual/R-devel/library/base/html/libPaths.html
Bug: Why generate WholeStagecodegen for single operator?
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 10/Jul/22 11:10
Affects Version/s: 3.3.0

WholeStagecodegen will have better performance in many cases. But it should not use WholeStagecodegen for single operator.

Below is a simple experiment.
{code:java}
test("range/filter should be combined") {
    val df = spark.range(10).filter("id = 1").selectExpr("id + 1")
    val plan = df.queryExecution.executedPlan
    assert(plan.find(_.isInstanceOf[WholeStageCodegenExec]).isDefined)
    assert(df.collect() === Array(Row(2)))
    df.explain(false)
    df.queryExecution.debug.codegen
  }{code}
 

If add 
{code:java}
override def supportCodegen: Boolean = false{code}
  in FilterExec.

 

The physical plan is 
{code:java}
== Physical Plan ==
*(2) Project [(id#0L + 1) AS (id + 1)#4L]
+- Filter (id#0L = 1)
   +- *(1) Range (0, 10, step=1, splits=2){code}
  
The performence is not good in this case.

How can disable WholeStagecodegen in these cases?
Bug: `tpcds-1g-gen` shouldn't execute If benchmark GA does not specify to execute TPCDSQueryBenchmark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/22 09:08
Affects Version/s: 3.4.0

I found that `{{{}Generate an input dataset for TPCDSQueryBenchmark with SF=1` {{}}}}always be executed even if I didn't specify to run `{{{}TPCDSQueryBenchmark`, f{}}}or example

[https://github.com/LuciferYang/spark/runs/7207453029?check_suite_focus=true]

I just want to run `{{{}MapStatusesConvertBenchmark{}}}`.

 

!image-2022-07-06-17-08-56-728.png!
Bug: SparkR build broken in branch-3.2 with Scala 2.13
Status: Open | Priority: Major | Resolution: 
Created: 08/Jul/22 02:48
Affects Version/s: 3.2.1

https://github.com/apache/spark/runs/7228058532?check_suite_focus=true

{code}
── 1. Failure (test_mllib_fpm.R:42:3): spark.fpGrowth ──────────────────────────
`expected_itemsets` not equivalent to `itemsets`.
Component “items”: Component 1: Component 1: 1 string mismatch
Component “items”: Component 2: Length mismatch: comparison on first 1 components
Component “items”: Component 2: Component 1: 1 string mismatch
Component “items”: Component 3: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Component 1: 1 string mismatch
Component “items”: Component 5: Length mismatch: comparison on first 1 components
Component “items”: Component 5: Component 1: 1 string mismatch
Component “freq”: Mean relative difference: 0.5454545
── 2. Failure (test_sparkSQL.R:190:5): structField type strings ────────────────
testField$dataType.toString() not equal to `expected`.
1/1 mismatches
x[1]: "Seq(StructField(a,StringType,true))"
y[1]: "StructType(StructField(a,StringType,true))"
── 3. Failure (test_sparkSQL_arrow.R:239:3): Arrow optimization - unsupported ty
`checkSchemaInArrow(structType("a STRUCT<a: INT>"))` did not throw an error.
{code}
Bug: Show proper error messages on the overflow errors of table insert
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/22 23:33
Affects Version/s: 3.3.0

In Spark 3.3, the error message of ANSI CAST is improved. However, the table insertion is using the same CAST expression:


{code:java}
> create table tiny(i tinyint);
> insert into tiny values (1000);
org.apache.spark.SparkArithmeticException[CAST_OVERFLOW]: The value 1000 of the type "INT" cannot be cast to "TINYINT" due to an overflow. Use `try_cast` to tolerate overflow and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
{code}
 

Showing the hint of `If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error` doesn't help at all. This PR is to fix the error message. After changes, the error message of this example will become:
{code:java}
org.apache.spark.SparkArithmeticException: [CAST_OVERFLOW_IN_TABLE_INSERT] Fail to insert a value of "INT" type into the "TINYINT" type column `i` due to an overflow. Use `try_cast` on the input value to tolerate overflow and return NULL instead.{code}
Bug: ORC uses Protobuf version vulnerable to CVE-2021-22569
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/22 05:07
Affects Version/s: 3.2.1

Our static analysis software detected vulnerability [CVE-2021-22569|https://github.com/protocolbuffers/protobuf/security/advisories/GHSA-wrvw-hg22-4m67], which comes from [ORC-1212] protobuf-java@2.5.0 has CVE-2021-22569 - ASF JIRA (apache.org).

Once ORC has addressed this vulnerability, Spark should upgrade to the next non-vulnerable version.
Bug: Improve error messages: step 3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/22 09:47
Affects Version/s: 3.4.0

1. https://github.com/apache/spark/pull/37171#discussion_r920796355
2. Remove unused error class INCONSISTENT_BEHAVIOR_CROSS_VERSION.FORMAT_DATETIME_BY_NEW_PARSER
3. Use the error class INDEX_ALREADY_EXISTS in the exception IndexAlreadyExistsException()
4. https://github.com/apache/spark/pull/38000#discussion_r1005438039
5. Remove unused NAMESPACE_ALREADY_EXISTS
6. Remove unused NAMESPACE_NOT_EMPTY
7. Remove unused NAMESPACE_NOT_FOUND
Bug: Request executor after kill executor, the number of executors is not as expected
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 11/Jul/22 09:17
Affects Version/s: 3.2.1

I used the killExecutors and requestExecutors function of SparkContext to dynamically adjust the resources, and found that the requestExecutors after killExecutors could not achieve the expected results.

Add unit tests in StandaloneDynamicAllocationSuite.scala 
{code:java}
test("kill executors first and then request") {
    sc = new SparkContext(appConf
      .set(config.EXECUTOR_CORES, 2)
      .set(config.CORES_MAX, 8))
    val appId = sc.applicationId
    eventually(timeout(10.seconds), interval(10.millis)) {
      val apps = getApplications()
      assert(apps.size === 1)
      assert(apps.head.id === appId)
      assert(apps.head.executors.size === 4) // 8 cores total
      assert(apps.head.getExecutorLimit === Int.MaxValue)
    }
    // sync executors between the Master and the driver, needed because
    // the driver refuses to kill executors it does not know about
    syncExecutors(sc)
    val executors = getExecutorIds(sc)
    assert(executors.size === 4)
    // kill 2 executors
    assert(sc.killExecutors(executors.take(3)))
    val apps = getApplications()
    assert(apps.head.executors.size === 1)
    // add 2 executors
    assert(sc.requestExecutors(3))
    assert(apps.head.executors.size === 4)
  } {code}
3 did not equal 4
Expected :4
Actual   :3
Bug: Column backticks are misplaced in the AnalysisException [UNRESOLVED_COLUMN] error message when using field with "."
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/22 22:13
Affects Version/s: 3.4.0

AnalysisException {{[UNRESOLVED_COLUMN.WITH_SUGGESTION]}} shows the wrong suggestion when a field with the "." separator is referenced without backticks.

The following code references a nested value {{{}`the`.`id`{}}}, that does not exist. The proposed / existing column names are not correctly wrapped in backticks:
{code:scala}
Seq((0)).toDF("the.id")
  .select("the.id").show()

org.apache.spark.sql.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `the`.`id` cannot be resolved. Did you mean one of the following? [`the`.`id`];
{code}
Instead of suggesting {{[`the`.`id`]}} in the *error message*, you would expect suggesting {{[`the.id`]}}.
Bug: Unpivot / melt function for SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/22 13:11
Affects Version/s: 3.4.0

With SPARK-38864 the Dataset API now has an unpivot / melt function. This should also be accessible through SQL statements, similar syntax as in [BigQuery|https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#unpivot_operator], [T-SQL|https://docs.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-ver15#unpivot-example], [Oracle|https://www.oracletutorial.com/oracle-basics/oracle-unpivot/].
Bug: Correctness issue when parsing dates with yyyyMMdd format in CSV and JSON
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 01:10
Affects Version/s: 3.4.0

In Spark 3.x, when reading CSV data like this:
{code:java}
name,mydate
1,2020011
2,20201203{code}
and specifying date pattern as "yyyyMMdd", dates are not parsed correctly with CORRECTED time parser policy.

For example,
{code:java}
val df = spark.read.schema("name string, mydate date").option("dateFormat", "yyyyMMdd").option("header", "true").csv("file:/tmp/test.csv")

df.show(false){code}
Returns:
{code:java}
+----+--------------+
|name|mydate        |
+----+--------------+
|1   |+2020011-01-01|
|2   |2020-12-03    |
+----+--------------+ {code}
and it used to return null instead of the invalid date in Spark 3.2 or below.

 

The issue appears to be caused by this PR: [https://github.com/apache/spark/pull/32959].

 

A similar issue can observed in JSON data source.

test.json
{code:java}
{"date": "2020011"}
{"date": "20201203"} {code}
 

Running commands
{code:java}
val df = spark.read.schema("date date").option("dateFormat", "yyyyMMdd").json("file:/tmp/test.json")
df.show(false) {code}
returns
{code:java}
+--------------+
|date          |
+--------------+
|+2020011-01-01|
|2020-12-03    |
+--------------+{code}
but before the patch linked in the description it used to show:
{code:java}
+----------+
|date      |
+----------+
|7500-08-09|
|2020-12-03|
+----------+{code}
which is strange either way. I will try to address it in the PR.
Bug: Dataset.repartition(N) may not create N partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/22 22:42
Affects Version/s: 3.3.0

Looks like there is a behavior change in Dataset.repartition in 3.3.0. For example, `spark.range(10, 0).repartition(5).rdd.getNumPartitions` returns 5 in Spark 3.2.0, but 0 in Spark 3.3.0.
Bug: Spark job failed with 10M rows data with Broken pipe error
Status: Open | Priority: Major | Resolution: 
Created: 12/Jul/22 07:35
Affects Version/s: 3.0.3

Spark job failed with 10M rows data with Broken pipe error. Same spark job was working previously with the settings "executor_cores": 1, "executor_memory": 1, "driver_cores": 1, "driver_memory": 1. where as the same job is failing with spark settings in 3.0.3 and 3.2.1.

Major symptoms (slowness, timeout, out of memory as examples): Spark job is failing with the error java.net.SocketException: Broken pipe (Write failed)


Here are the spark settings information which is working on Spark 3.0.3 and 3.2.1 : "executor_cores": 4, "executor_memory": 4, "driver_cores": 4, "driver_memory": 4.. The spark job doesn't consistently works with the above settings. Some times, need to increase the cores and memory.
Bug: The structural integrity of the plan is broken after UnwrapCastInBinaryComparison
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 14:06
Affects Version/s: 3.3.0

{code:scala}
sql("create table t1(a decimal(3, 0)) using parquet")
sql("insert into t1 values(100), (10), (1)")
sql("select * from t1 where a in(100000, 10, 0, 1.00)").show
{code}

{noformat}
After applying rule org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison in batch Operator Optimization before Inferring Filters, the structural integrity of the plan is broken.
java.lang.RuntimeException: After applying rule org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison in batch Operator Optimization before Inferring Filters, the structural integrity of the plan is broken.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.structuralIntegrityIsBrokenAfterApplyingRuleError(QueryExecutionErrors.scala:1325)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:229)
{noformat}
Bug: Use different error classes for numeric/interval arithmetic overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/22 04:49
Affects Version/s: 3.4.0

Currently, when  arithmetic overflow errors happen under ANSI mode, the error messages are like
[ARITHMETIC_OVERFLOW] long overflow. Use 'try_multiply' to tolerate overflow and return NULL instead. If necessary set spark.sql.ansi.enabled to "false" 

 

The "(except for ANSI interval type)" part is confusing. We should remove it for the numeric arithmetic operations and have a new error class for the interval division error: INTERVAL_ARITHMETIC_OVERFLOW
Bug: Use different error classes for numeric/interval divided by 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 05:54
Affects Version/s: 3.4.0

Currently, when numbers are divided by 0 under ANSI mode, the error message is like
{quote}[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "ansi_mode" to "false" (except for ANSI interval type) to bypass this error.{quote}
The "(except for ANSI interval type)" part is confusing.  We should remove it and have a new error class "INTERVAL_DIVIDED_BY_ZERO"
Bug: Expression transform error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 02:26
Affects Version/s: 3.2.1

{code:java}
spark.sql(
  """
    |select to_date(a) a, to_date(b) b from
    |(select  a, a as b from
    |(select to_date(a) a from
    | values ('2020-02-01') as t1(a)
    | group by to_date(a)) t3
    |union all
    |select a, b from
    |(select to_date(a) a, to_date(b) b from
    |values ('2020-01-01','2020-01-02') as t1(a, b)
    | group by to_date(a), to_date(b)) t4) t5
    |group by to_date(a), to_date(b)
    |""".stripMargin).show(){code}
result is (2020-02-01, 2020-02-01), (2020-01-01, 2020-01-01)
expected (2020-02-01, 2020-02-01), (2020-01-01, 2020-01-02)
Bug: Fix bug in existence DEFAULT value lookups for non-vectorized Parquet scans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/22 21:49
Affects Version/s: 3.4.0

How to reproduce:
{code:sql}
set spark.sql.parquet.enableVectorizedReader=false;
create table t(a int) using parquet;
insert into t values (42);
alter table t add column b int default 42;
insert into t values (43, null);
select * from t;
{code}
This should return two rows:

(42, 42) and (43, NULL)

But instead the scan misses the inserted NULL value, and returns the existence DEFAULT value of "42" instead:

(42, 42) and (43, 42).

 

This bug happens because the Parquet API calls one of these set* methods in ParquetRowConverter.scala whenever it finds a non-NULL value:

{code:scala}
private class RowUpdater(row: InternalRow, ordinal: Int)
extends ParentContainerUpdater {
  override def set(value: Any): Unit = row(ordinal) = value
  override def setBoolean(value: Boolean): Unit = row.setBoolean(ordinal, value)
  override def setByte(value: Byte): Unit = row.setByte(ordinal, value)
  override def setShort(value: Short): Unit = row.setShort(ordinal, value)
  override def setInt(value: Int): Unit = row.setInt(ordinal, value)
  override def setLong(value: Long): Unit = row.setLong(ordinal, value)
  override def setDouble(value: Double): Unit = row.setDouble(ordinal, value)
  override def setFloat(value: Float): Unit = row.setFloat(ordinal, value)
}
 {code}
 
But it never calls anything like "setNull()" when encountering a NULL value.

To fix the bug, we need to know how many columns of data were present in each row of the Parquet data, so we can differentiate between a NULL value and a missing column.
Bug: DS V2 aggregate push down can work with Top N or Paging (Sort with group expressions)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/22 04:06
Affects Version/s: 3.4.0

Currently, DS V2 aggregate push-down cannot work with Top N (order by ... limit ...) or Paging (order by ... limit ... offset ...).
If it can work with Top N or Paging, it will be better performance.
Bug: Regression due to AVRO-2035
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/22 06:59
Affects Version/s: 3.2.1

With the upgrade in Avro version to 1.9.0, for schema evolution Avro added https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default values in schemas by default) which is causing regressions when user upgrades their Spark verion.

Repro code:
{code:java}
import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
                        |{
                        |  "type": "record",
                        |  "name": "struct",
                        |  "fields": [
                        |    {"name": "id", "type": "long", "default": null}
                        |  ]
                        |}""".stripMargin

val df = spark.range(10).select(struct('id).as("struct"))
val avroStructDF = df.select(to_avro('struct, avroTypeStruct).as("avro"))
avroStructDF.select(from_avro('avro, avroTypeStruct)).show()
{code}

Hive mitigated it by disabling this feature altogether in https://issues.apache.org/jira/browse/HIVE-24797  
Spark-Hive integration also imported the above changes in https://issues.apache.org/jira/browse/SPARK-34512

Can we have a fix for all the senarios?
Bug: SkewJoin--Stream side skew in BroadcastJoin
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jul/22 06:54
Affects Version/s: 3.1.2

There is data skew in BroadcastJoin. Since LocalShuffle cannot cover all cases, we can refer to the solution of SortMergeJoin skew to consider solving the data skew problem in BroadcastJoin.
h3. senairo

!1.png!

!2.png!
h3. Effect

!4.png!

!3.png!
Bug: Memory leak when closing a  session of Spark Thrift Server
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/22 03:40
Affects Version/s: 3.1.3

we are using Spark Thrift Server as a distributed sql query engine, the queries are related to read some datasource table with a lot of files.

when we open multiple sessions, the Driver could be crashed with OOM.

We can reproduce it with the following steps:
 # start spark thrift server;
 # using beeline to open a new session;
 # create a new datasource table;
 # insert one row data into this table;
 # open another 5 new sessions, then using `select` command to scan this table
 # close all the 6 session
 # using jmap command `jmap -histo:live pid > pid.log` to print the dump of Driver

*Expected result:*

The cached FileStatus should be cleaned, the number of HdfsLocatedFileStatus object should be 0.

*Actural result:*

 ** The number of HdfsLocatedFileStatus object  is 6.

*!image-2022-07-26-11-54-06-826.png!*
Bug: Fix EliminateSorts remove global sort below the local sort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/22 03:17
Affects Version/s: 3.4.0

If a global sort below locol sort, we should not remove the global sort becuase the output partitioning can be affected.

This issue is going to worse since we pull out the V1 Write sort to logcial side.
Bug: Join‘ verbose string didn't contains JoinType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/22 07:23
Affects Version/s: 3.3.0

 Current verbose string don't have joinType


{code:java}
(5) BroadcastHashJoin [codegen id : 8]
Left keys [1]: [ss_sold_date_sk#3]
Right keys [1]: [d_date_sk#5]
Join condition: None
{code}

{code:java}
override def verboseStringWithOperatorId(): String = {
    val joinCondStr = if (condition.isDefined) {
      s"${condition.get}"
    } else "None"
    if (leftKeys.nonEmpty || rightKeys.nonEmpty) {
      s"""
         |$formattedNodeName
         |${ExplainUtils.generateFieldString("Left keys", leftKeys)}
         |${ExplainUtils.generateFieldString("Right keys", rightKeys)}
         |${ExplainUtils.generateFieldString("Join condition", joinCondStr)}
         |""".stripMargin
    } else {
      s"""
         |$formattedNodeName
         |${ExplainUtils.generateFieldString("Join condition", joinCondStr)}
         |""".stripMargin
    }
  }
{code}
Bug: Add Scan details to spark plan scan node in SparkUI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/22 01:06
Affects Version/s: 3.4.0

Hi,

For DSv2, the scan node in the spark plan on SparkUI simply shows "BatchScan" as opposed to "Scan <fileformat>".

Add a method "String name()" to the Scan interface, that "BatchScanExec" can invoke to set the node name the plan. This nodeName will be eventually used by "SparkPlanGraphNode" to display it in the header of the UI node.

 

DSv1

!Screen Shot 2022-07-27 at 6.00.27 PM.png|width=356,height=212!

 

DSv2

!Screen Shot 2022-07-27 at 6.00.50 PM.png|width=293,height=277!
Bug: Remove OptimizeLimitZero and merge it into EliminateLimits
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/22 12:24
Affects Version/s: 3.4.0

Currently, Spark Optimizer have two rule: OptimizeLimitZero and EliminateLimits.
In fact, OptimizeLimitZero is one case of eliminate limits.