av9ash/GitBugs huggingface dataset:
Bug: Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Apr/21 14:57
Affects Version/s: 3.2.0

HadoopVersionInfoSuite use a separate ivyPath to download the jars, we can use `

HiveClientBuilder.ivyPath` and  specify environment variables `SPARK_VERSIONS_SUITE_IVY_PATH` like `VersionsSuite` to avoid downloading jars repeatedly and speed up the test.
Bug: Investigate to prefer vectorized hash map in hash aggregate selectively
Status: Open | Priority: Minor | Resolution: 
Created: 27/Apr/21 07:40
Affects Version/s: 3.2.0

In hash aggregate, we always use row-based hash map as first level hash map in production, and use vectorized hash map in testing / benchmarking. However we do find in micro-benchmark that vectorized hash map is better than row-based hash map e.g. with single key - [https://github.com/apache/spark/pull/32357#discussion_r620914345] . So we should re-evaluate the decision to always use row-based hash map or not. And maybe come up with a more adaptive decision policy to choose which map to use depending on keys / values.
Bug: Some columns  in table ` Aggregated Metrics by Executor` of stage-detail page shows incorrectly.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Apr/21 12:01
Affects Version/s: 3.0.1

Some columns like 'Shuffle Read Size / Records', 'Output Size/ Records' etc in table ` Aggregated Metrics by Executor` of stage-detail page shouble be sorted as numerical-order instead of lexicographical-order.
Bug: mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Apr/21 06:34
Affects Version/s: 3.0.2


{code:java}
spark-sql> set spark.sql.adaptive.coalescePartitions.initialPartitionNum=1;
spark.sql.adaptive.coalescePartitions.initialPartitionNum	1
Time taken: 2.18 seconds, Fetched 1 row(s)
spark-sql> set mapred.reduce.tasks;
21/04/21 14:27:11 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
spark.sql.shuffle.partitions	1
Time taken: 0.03 seconds, Fetched 1 row(s)
spark-sql> set spark.sql.shuffle.partitions;
spark.sql.shuffle.partitions	200
Time taken: 0.024 seconds, Fetched 1 row(s)
spark-sql> set mapred.reduce.tasks=2;
21/04/21 14:31:52 WARN SetCommand: Property mapred.reduce.tasks is deprecated, automatically converted to spark.sql.shuffle.partitions instead.
spark.sql.shuffle.partitions	2
Time taken: 0.017 seconds, Fetched 1 row(s)
spark-sql> set mapred.reduce.tasks;
21/04/21 14:31:55 WARN SetCommand: Property mapred.reduce.tasks is deprecated, showing spark.sql.shuffle.partitions instead.
spark.sql.shuffle.partitions	1
Time taken: 0.017 seconds, Fetched 1 row(s)
spark-sql>
{code}
Bug: Accumulators Explore Page on Spark UI on History Server
Status: In Progress | Priority: Minor | Resolution: 
Created: 22/Apr/21 22:46
Affects Version/s: 2.4.4

Proposition for *Accumulators Explore Page* on *SparkUI*: The particular information for the accumulators will be located under a new tab that has an overview page with links to check for more details about the accumulators information by a particular name or stage.
Bug: Refractor org.apache.hive.service.cli.OperationType.getOperationType by using a hashMap.
Status: In Progress | Priority: Minor | Resolution: 
Created: 23/Apr/21 10:30
Affects Version/s: 3.0.1

SImply the *getOperationType* method in `org.apache.hive.service.cli.OperationType`.
 Introduce a *HashMap* to cache the existed enumeration types, so as to avoid a linear search in a for loop.
 `*OperationType.getOperationType*` can be called in OperationHandler.constructor:

 
{code:java}
 

public OperationHandle(TOperationHandle tOperationHandle, TProtocolVersion protocol) {
  super(tOperationHandle.getOperationId());
  this.opType = OperationType.getOperationType(tOperationHandle.getOperationType());
  this.hasResultSet = tOperationHandle.isHasResultSet();
  this.protocol = protocol;
}
 

{code}
`*OperationHandle* ` is widely used, It's better to improve the execution efficiency of `OperationType.getOperationType`
Bug: The DAG part of sql-execution page should can be hidden/expand instead of being displayed directly all the time.
Status: In Progress | Priority: Minor | Resolution: 
Created: 23/Apr/21 06:47
Affects Version/s: 3.0.1

1.if it is a particularly complex SQL, it would take a long time to render the svg, and the page will be stuck for a long time. 
2.sometimes we just want to look at the detail of execution plan, but DAG cannot be hidden, and it is often inconvenient to drag the scroll bar to view the execution plan information.
Bug: CombineConcats should call transformAllExpressions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Apr/21 06:57
Affects Version/s: 3.1.0

{code:java}
plan transformExpressions { ... }{code}
only applies the transformation node `plan` itself, but not its children. We should call transformAllExpressions instead of transformExpressions in CombineConcats.
Bug: When we switch between different stage-detail pages, the entry item in the newly-opened  page may be blank.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Apr/21 13:51
Affects Version/s: 3.0.1

spark: 3.0.1

reproduce step:

Assuming that we currently have two stages as shown below, note that the two stages contain different numbers of task(attrached as stages-list.png)
 # click the task with id 0, jump to stage-detail page, and then switch entries item to *All* option (attrached as first-step.png)
 #  then click the task with id 1, jump to stage-detail page, we will find that the entries shows empty. (attrached as second-step.png)
Bug: Introduce hybrid join for sort merge join and shuffled hash join in AQE
Status: Open | Priority: Minor | Resolution: 
Created: 21/Apr/21 22:50
Affects Version/s: 3.2.0

Per discussion in [https://github.com/apache/spark/pull/32210#issuecomment-823503243] , we can introduce some kind of {{HybridJoin}} operator in AQE, and we can choose to do shuffled hash join vs sort merge join for each task independently, e.g. based on partition size, task1 can do shuffled hash join, and task2 can do sort merge join, etc.
Bug: Make spark.sql.hive.version meaningful and not deprecated
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Apr/21 02:42
Affects Version/s: 3.2.0

Firstly let's take a look the definition and comment.
{code:java}
// A fake config which is only here for backward compatibility reasons. This config has no effect
// to Spark, just for reporting the builtin Hive version of Spark to existing applications that
// already rely on this config.
val FAKE_HIVE_VERSION = buildConf("spark.sql.hive.version")
  .doc(s"deprecated, please use ${HIVE_METASTORE_VERSION.key} to get the Hive version in Spark.")
  .version("1.1.1")
  .fallbackConf(HIVE_METASTORE_VERSION)
{code}

It is used for reporting the built-in Hive version but the current status is unsatisfactory, as it is could be changed in many ways e.g. --conf/SET syntax.

It is marked as deprecated but kept a long way until now. I guess it is hard for us to remove it and not even necessary.

On second thought, it's actually good for us to keep it to work with the `spark.sql.hive.metastore.version`. As when `spark.sql.hive.metastore.version` is changed, it could just be used to report the compiled hive version statically, it's useful when an error occurs in this case. So this parameter should be fixed to compiled hive version.
Bug: Revise outputpartitioning number in some SparkPlan
Status: In Progress | Priority: Minor | Resolution: 
Created: 19/Apr/21 12:04
Affects Version/s: 3.2.0

Revise the outputPartitioning number in some `SparkPlan`
 * UnionExec
 * RowDataSourceScanExec
 * LogicalRDD
 * RDDScanExec
 * ExternalRDDScanExec
 * LocalTableScanExec
 * ExpandExec
 * BroadcastNestedLoopJoinExec
 * CoGroupExec
 * DataSourceV2ScanExecBase

The mian change follow the logical
 1. if rdd is known, using `rdd.partitions.length`
 2. using `child.outputPartitioning.numPartitions`
Bug: Test ANSI interval literals
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Apr/21 10:17
Affects Version/s: 3.2.0

Add tests for YearMonthIntervalType and DayTimeIntervalType to LiteralExpressionSuite similar to CalendarIntervalType:
- https://github.com/apache/spark/blob/3f4c32b3ca2d0f20806f81422e086d035b2eb5ce/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala#L52
- https://github.com/apache/spark/blob/3f4c32b3ca2d0f20806f81422e086d035b2eb5ce/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala#L79
- https://github.com/apache/spark/blob/3f4c32b3ca2d0f20806f81422e086d035b2eb5ce/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala#L203-L204
Bug: Fix ugly indentation of multiple JSON records in a single split file generated by JacksonGenerator when pretty option is true
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Apr/21 06:05
Affects Version/s: 3.0.2

When writing multiple JSON records into a single split file with pretty option true, indentation will be broken except for the first JSON record.
{code:java}
// Run in the Spark Shell.
// Set spark.sql.leafNodeDefaultParallelism to 1 for the current master.
// Or set spark.default.parallelism for the previous releases.
spark.conf.set("spark.sql.leafNodeDefaultParallelism", 1)
val df = Seq("a", "b", "c").toDF
df.write.option("pretty", "true").json("/path/to/output")

# Run in a Shell
$ cat /path/to/output/*.json
{
  "value" : "a"
}
 {
  "value" : "b"
}
 {
  "value" : "c"
}
{code}
Bug: spark.jars.ivysettings should support local:// and hdfs:// schemes
Status: Open | Priority: Minor | Resolution: 
Created: 14/Apr/21 15:08
Affects Version/s: 3.2.0

During reviews of SPARK-34472, there was a desire to support local:// and hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See https://github.com/apache/spark/pull/31591#discussion_r598850998 and https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently this fails with the following error:

{code:java}
./bin/spark-shell --conf spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf spark.jars.ivySettings="local:///Use
rs/test/temp/ivysettings.xml"
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 {code}

We should make sure that this also works with {{SparkContext#addJar}} when running in cluster mode. At this point, only YARN resource manager supports managing {{ivysettings}} file in cluster mode. This can change based on SPARK-35073.
Bug: LIST FILES/JARS/ARCHIVES cannot handle multiple arguments properly when at least one path is quoted.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Apr/21 05:34
Affects Version/s: 3.0.2

`LIST FILES/JARS/ARCHIVES path1 path2 ...` cannot list all paths if at least one path is quoted.
An example here.

{code}
ADD FILE /tmp/test1;
ADD FILE /tmp/test2;

LIST FILES /tmp/test1 /tmp/test2;
file:/tmp/test1
file:/tmp/test2

LIST FILES /tmp/test1 "/tmp/test2";
file:/tmp/test2
{code}

In this example, the second `LIST FILES` doesn't show `file:/tmp/test1`.
Bug: Improve error msg if UTF8String concatWs length overflow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/21 09:36
Affects Version/s: 3.2.0

We encounter a very extreme case with expression `concat_ws`, and the error msg is
{code:java}
Caused by: java.lang.NegativeArraySizeException
	at org.apache.spark.unsafe.types.UTF8String.concatWs
{code}

Seems the `UTF8String.concat` has already done the length check, so it's better to add in `concatWs`.
Bug: Fix Incorrect assertion of "master/worker web ui available behind front-end reverseProxy" in MasterSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/21 07:53
Affects Version/s: 3.1.1

[https://github.com/apache/spark/blob/bfba7fadd2e65c853971fb2983bdea1c52d1ed7f/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala#L425]
{code:java}
// with LocalCluster, we have masters and workers in the same JVM, each overwriting
// system property spark.ui.proxyBase.
// so we need to manage this property explicitly for test
System.getProperty("spark.ui.proxyBase") should startWith  (s"$reverseProxyUrl/proxy/worker-") // Line 425
{code}
Line 425 in MasterSuite is considered as unused expression,  If we merge lines 424 and 425 into one：
{code:java}
System.getProperty("spark.ui.proxyBase") should startWith (s"$reverseProxyUrl/proxy/worker-")
{code}
this assertion will fail:
{code:java}
- master/worker web ui available behind front-end reverseProxy *** FAILED ***
  The code passed to eventually never returned normally. Attempted 45 times over 5.091914027 seconds. Last failure message: "http://proxyhost:8080/path/to/spark" did not start with substring "http://proxyhost:8080/path/to/spark/proxy/worker-". (MasterSuite.scala:405)

{code}
Bug: Failed to excute python/run-tests-with-coverage cmd
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Apr/21 12:57
Affects Version/s: 3.2.0

~/spark$ python/run-tests-with-coverage --testnames pyspark.sql.tests.test_arrow --python-executables=python

Running PySpark tests. Output is in /home/spark/spark/python/unit-tests.log

*rm: missing operand*
*Try 'rm --help' for more information.*


~/spark$ echo $?
123

 

We should add "-fr" to "rm"
Bug: Migrate First function from DeclarativeAggregate to TypedImperativeAggregate to improve performance
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Apr/21 23:17
Affects Version/s: 3.2.0

The main objective of this change is to improve performance in some cases.

We have three possibilities when we plan an aggregation. In the first case, with mutable primitive types, HashAggregate is used.

When we are not using these types we have two options. If the function implements TypedImperativeAggregate we use ObjectHashAggregate. Otherwise, we use SortAggregate that is less efficient.

In this PR I propose to migrate First function to implement TypedImperativeAggregate to take advantage of this feature (ObjectAggregateExec)

This Jira is related to SPARK-34464
Bug: Remove unnecessary sort in FileFormatWriter when only have one partition
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 01/Apr/21 13:05
Affects Version/s: 3.1.1

sort is a very resource-intensive operation, spark may add SortExec when writing partition file for writing sequentially, but i think sorting is unnecessary when only have one partition. 

 

related file: 
{code:java}
org.apache.spark.sql.execution.datasources.FileFormatWriter
{code}
related code:
{code:java}
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering = partitionColumns ++ bucketIdExpression ++ sortColumns
// the sort order doesn't matter
val actualOrdering = plan.outputOrdering.map(_.child)
val orderingMatched = if (requiredOrdering.length > actualOrdering.length) {
  false
} else {
  requiredOrdering.zip(actualOrdering).forall {
    case (requiredOrder, childOutputOrder) =>
      requiredOrder.semanticEquals(childOutputOrder)
  }
}{code}
Bug: support other numeric types besides Double in  QueryTest#checkAggregatesWithTol
Status: Open | Priority: Minor | Resolution: 
Created: 01/Apr/21 09:57
Affects Version/s: 3.1.1

This is a followup of SPARK-10641

it noly support Double type in QueryTest#checkAggregatesWithTol  and this pr will  support other numeric types.
Bug: Fix minor unit test in BasicWriteTaskStatsTrackerSuite
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 02/Apr/21 04:45
Affects Version/s: 3.2.0

This is to fix the minor typo in unit test of BasicWriteTaskStatsTrackerSuite ([https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BasicWriteTaskStatsTrackerSuite.scala#L152] ), where it should be a new file name, e.g. `f-3-3`, because the unit test expects 3 files in statistics ([https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BasicWriteTaskStatsTrackerSuite.scala#L160] ).
Bug: Failed to install pyspark[sql] (due to pyarrow error) on aarch64
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Apr/21 08:31
Affects Version/s: 3.2.0

~^$ pip install pyspark[sql]^~

~^Collecting pyarrow>=1.0.0^~
 ~^Using cached pyarrow-3.0.0.tar.gz (682 kB)^~
 ~^Installing build dependencies ... done^~
 ~^Getting requirements to build wheel ... done^~
 ~^Preparing wheel metadata ... done^~
 ~^// ... ...^~
 ~^Building wheels for collected packages: pyarrow^~
 ~^Building wheel for pyarrow (PEP 517) ... error^~
 ~^ERROR: Command errored out with exit status 1:^~
 ~^command: /root/venv/bin/python3.8 /tmp/tmpv35m1o0g build_wheel /tmp/tmpq0n5juib^~
 ~^cwd: /tmp/pip-install-sh0myu71/pyarrow^~
 ~^Complete output (183 lines):^~

~^– Running cmake for pyarrow^~
 ~^cmake -DPYTHON_EXECUTABLE=/root/venv/bin/python3.8 -DPython3_EXECUTABLE=/root/venv/bin/python3.8 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /tmp/pip-install-sh0myu71/pyarrow^~
 ~^error: command 'cmake' failed with exit status 1^~
 ~^----------------------------------------^~
 ~^ERROR: Failed building wheel for pyarrow^~
 ~^Failed to build pyarrow^~
 ~^ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly^~

 

The pip installation would be failed, due to the dependency pyarrow install failed.
Bug: spark.jars.xxx configs should be moved to config/package.scala
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Apr/21 15:25
Affects Version/s: 3.2.0

Currently {{spark.jars.xxx}} property keys (e.g. {{spark.jars.ivySettings}} and {{spark.jars.packages}}) are hardcoded in multiple places within Spark code across multiple modules. We should define them in {{config/package.scala}} and reference them in all other places.

This came up during reviews of SPARK-34472 at https://github.com/apache/spark/pull/31591#discussion_r584848624
Bug: Add row-based fast hash map into aggregate benchmark
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Apr/21 03:38
Affects Version/s: 3.2.0

`AggregateBenchmark` is only testing the performance for vectorized fast hash map, but not row-based hash map (which is used by default). We should add the row-based hash map into the benchmark.
Bug: Move InMemoryTable etc to org.apache.spark.sql.connector.catalog
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Apr/21 19:32
Affects Version/s: 3.2.0

Currently test classes such as {{InMemoryTable}} reside in {{org.apache.spark.sql.connector}} rather than {{org.apache.spark.sql.connector.catalog}}. We should move them to latter to match the interfaces they implement.
Bug: Fix minor exception messages of HashedRelation and HashJoin
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 17/Apr/21 06:08
Affects Version/s: 3.2.0

It seems that we miss classifying one `SparkOutOfMemoryError` in `HashedRelation`. Add the error classification for it. In addition, clean up two errors definition of `HashJoin` as they are not used.
Bug: Do not allow repartition by map
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Dec/24 22:21
Affects Version/s: 4.0.0

 

We allow users to repartition by a map column. This leads to incorrect results.
{code:java}
// Create a sequence of maps that all have the same element, but a different insertion order.
import scala.util.Random
val elements = Seq.tabulate(4)(i => i -> s"v$i") // 4 elements uses a scala.collection.immutable.Map$Map4, this retains the insertion order.
val maps = Seq.fill(10)(Random.shuffle(elements).toMap)

// Check if they are all the same in scala land.
assert(maps.distinct.size == 1)

// This fails, which is good.
maps.toDF.distinct.show()

// This should return a single partition. However it returns multiple partitions.
maps.toDF.repartition(4, $"value").groupBy(spark_partition_id()).show()

// +--------------------+-----+
// |SPARK_PARTITION_ID()|count|
// +--------------------+-----+
// |                   0|    2|
// |                   1|    4|
// |                   2|    2|
// |                   3|    2|
// +--------------------+-----+{code}
Bug: Fix KafkaMicroBatchSourceSuite to cover KafkaOffsetReaderConsumer with unit tests
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 13/Dec/24 11:15
Affects Version/s: 3.5.3

KafkaMicroBatchSourceSuite consists of set of different suites, where KafkaMicroBatchSourceSuiteBase based suite is defined. There are 4 implementations of this abstract class for now:
  1. `KafkaMicroBatchV1SourceSuite` - V1 source that supposes to use `KafkaOffsetReaderConsumer` as `KafkaOffsetReader`.
  2. `KafkaMicroBatchV2SourceSuite` - V2 source that supposes to use `KafkaOffsetReaderConsumer` as `KafkaOffsetReader`.
  3. `KafkaMicroBatchV1SourceWithAdminSuite` - V1 source that uses `KafkaOffsetReaderAdmin` as `KafkaOffsetReader`.
  4. `KafkaMicroBatchV2SourceWithAdminSuite` - V2 source that uses `KafkaOffsetReaderAdmin` as `KafkaOffsetReader`.
But `KafkaMicroBatchV1SourceSuite` and `KafkaMicroBatchV2SourceSuite` are still running based on `KafkaOffsetReaderAdmin`, as `USE_DEPRECATED_KAFKA_OFFSET_FETCHING` is `false` be default. By switching it to `true` in `beforeAll`, we make sure that corresponding `KafkaOffsetReader` is in use.
Bug: Add RPAD pushdown support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/24 16:29
Affects Version/s: 3.5.4

Make {{RPAD/LPAD}} function translatable (via {{{}V2ExpressionBuilder{}}}) and it's pushdown supported (via {{V2ExpressionSQLBuilder}} as a {{{}visitSQLFunction{}}}).
h3. CONTEXT:

When trying to compare CHAR(...) and a literal, Spark injects a RPAD function to make the CHAR parameter the same length. This behaviour is per ANSI standard.
h3. THE PROBLEM:

RPAD is not only an unsupported, but an untranslatable function. This prevents CHAR + literal comparison pushdown.
h3. THE SOLUTION:

Support RPAD pushdowns and implement it when needed