av9ash/GitBugs huggingface dataset:
Bug: Upgrade Jetty version to 9.4.44
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/22 14:32
Affects Version/s: 3.2.0

Upgrade Jetty version to 9.4.44.v20210927 in current Spark master to bring-in the fixes for the [jetty#6973|https://github.com/eclipse/jetty.project/issues/6973] issue.
Bug: Use loops to refactor methods UnsafeRow#isFixedLength and UnsafeRow#isMutable
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Jan/22 05:53
Affects Version/s: 3.3.0

Methods UnsafeRow#isFixedLength and UnsafeRow#isMutable use tail recursion now , we can rewrite them with looping, which will be considerably faster
Bug: Spark dynamic allocation ignores pending jobs
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jan/22 14:52
Affects Version/s: 3.2.0

When running spark with dynamic allocation on kubernetes, everything seems to work fine for a single job. If there are pending tasks, executors are requested according to the configurations just fine. The problem happens when there are pending jobs, it seems that whatever is on queue to be executed is ignored by the dynamic allocation system. We're hitting this corner case because, in this particular situation, several jobs with just one(or just a few) task are submitted in parallel. The behavior we're seeing is that only the first executor is added, regardless how many pending tasks there are in other jobs in the same context. I'm strapped for time right now, but please let me know if you need some code to reproduce this or if I wasn't clear enough with the explanation
Bug: Expose Arrow Vector from ArrowColumnVector
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jan/22 01:39
Affects Version/s: 3.3.0

In some cases we need to work with Arrow Vectors behind ColumnVector using Arrow APIs. For example, some Spark extension libraries need to consume Arrow Vectors. For now, it is impossible as the Arrow Vector is private member in ArrowColumnVector. We need to expose the Arrow Vector from ArrowColumnVector.
Bug: Upgrade dropwizard metrics from 4.2.2 to 4.2.7
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/22 09:31
Affects Version/s: 3.3.0

dropwizard metrics has released 5 versions after 4.2.2：
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.3]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.4]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.5]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.6]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.7]

 

And after 4.2.5 version, codahale metrics supports build with JDK 17 (https://github.com/dropwizard/metrics/pull/2180)
Bug: Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/22 06:40
Affects Version/s: 3.3.0

As of Spark 3.3.0, session_window supports not only TimestampType but also TimestampNTZType but the API docs doesn't mention TimestampNTZType.
Bug: In start-thriftserver.sh arguments, "--hiveconf xxx" should have higher precedence over "--conf spark.hadoop.xxx", or any other hadoop configurations
Status: Open | Priority: Minor | Resolution: 
Created: 25/Jan/22 01:34
Affects Version/s: 2.4.8

By convention, An Apache Hive server will read configuration options from different sources with different precedence, and the precedence of "–hiveconf" options in command line options should only be lower than those set by using the {*}set command (see [https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration] for detail){*}. It should be higher than hadoop configuration, or any of the configuration files on the server (including, but not limited to hive-site.xml and core-site.xml)

This convention is clearly not maintained very well by Apache Spark thrift server. As demonstrated in the following example: If I start this server with diverging option values on "hive.server2.thrift.port":

 

```
./sbin/start-thriftserver.sh \
--conf spark.hadoop.hive.server2.thrift.port=10001 \
--hiveconf hive.server2.thrift.port=10002
```

 

"–conf"/port 10001 will be preferred over "–hiveconf"/port 10002:

 

```

Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /home/xxx/spark-2.4.7-bin-hadoop2.7-scala2.12/conf/:/home/xxx/spark-2.4.7-bin-hadoop2.7-scala2.12/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --conf spark.hadoop.hive.server2.thrift.port=10001 --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server spark-internal --hiveconf hive.server2.thrift.port=10002
========================================
...
22/01/24 17:32:18 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10001 with 5...500 worker threads

```

 

replacing "--conf" line with an entry in core-site.xml makes no difference.

I doubt if this divergence from conventional hive server behaviour is deliberate. Thus I'm calling the precedence of hive configuration options to be set to be on par or maximally similar to that of an Apache Hive server of the same version. To my knowledge, it should be:

 

SET command > --hiveconf > hive-site.xml > hive-default.xml > --conf > core-site.xml >. core-default.xml
Bug: Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/22 03:39
Affects Version/s: 3.2.0

Add scalastyle check to disable internal use of scala.concurrent.ExecutionContext.Implicits.global. 
The reason is that user queries can also use this thread pool, causing competing in resource and starvation. Spark-internal APIs should thus not use the global thread pool.
Bug: Typing incompatibilities with numpy==1.22.x
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/22 15:18
Affects Version/s: 3.3.0

When type checked against {{numpy==1.22}} mypy detects following issues:


{code:python}
python/pyspark/mllib/linalg/__init__.py:412: error: Argument 2 to "norm" has incompatible type "Union[float, str]"; expected "Union[None, float, Literal['fro'], Literal['nuc']]"  [arg-type]
python/pyspark/mllib/linalg/__init__.py:457: error: No overload variant of "dot" matches argument types "ndarray[Any, Any]", "Iterable[float]"  [call-overload]
python/pyspark/mllib/linalg/__init__.py:457: note: Possible overload variant:
python/pyspark/mllib/linalg/__init__.py:457: note:     def dot(a: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], b: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], out: None = ...) -> Any
python/pyspark/mllib/linalg/__init__.py:457: note:     <1 more non-matching overload not shown>
python/pyspark/mllib/linalg/__init__.py:707: error: Argument 2 to "norm" has incompatible type "Union[float, str]"; expected "Union[None, float, Literal['fro'], Literal['nuc']]"  [arg-type]
{code}
Bug: Upgrade commons-collections3 to commons-collections4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/22 08:42
Affects Version/s: 3.3.0

Apache commons-collections 3.x is a Java 1.3 compatible version, and it does not use Java 5 generics.  

Apache commons-collections4 4.4 is an upgraded version and it built by Java 8
Bug: Fix the UT of checking norm in KMeans & BiKMeans
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jan/22 08:20
Affects Version/s: 3.3.0

In KMeansSuite and BisectingKMeansSuite, there are some unused lines:

 
{code:java}
model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 {code}
 

For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful;

For euclidean distance, the norm checking is meaningless;
Bug: PartitioningAwareFileIndex->basePath incorrectly contains the partition filters
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 18/Jan/22 16:52
Affects Version/s: 3.2.0

PartitioningAwareFileIndex.getBasePath method returns paths that contain the partitioning directories. This violates the definition of the basePath per FileIndex, i.e. the parent directory of a file path with all the partitioning directories being stripped off. 

This PR fixes the issue by separating the notion of the partitioningPaths and the basePaths in the PartitioningAwareFileIndex. The basePaths are derived by removing from the partitioningPaths any partitioning columns with the aid of the PartitioningSchema.
Bug: Upgrade Netty from 4.1.72 to 4.1.73
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/22 19:25
Affects Version/s: 3.3.0

Netty has a new release that upgrades log4j to 2.17.1. Although I didn't find obvious dependence on log4j via netty in my search of Spark's codebase, it would be good to pick up this specific version. The version Spark currently depends on is 4.1.72, which depends on log4j 2.15. Several CVE's have been fixed in log4j between 2.15 and 2.17.1.

Besides this dependency update, several minor bugfixes have been made in this release.
Bug: Move SpecificParquetRecordReaderBase.listDirectory to TestUtils
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jan/22 09:24
Affects Version/s: 3.3.0

The `SpecificParquetRecordReaderBase.listDirectory` method is used to return the list of files at `path` recursively and the result will skips files that are ignored normally by MapReduce.

This method is only used by tests in Spark now and it also includes non parquet related test cases, such as `OrcColumnarBatchReaderSuite`.
Bug: Update annotations to resolve issues detected with mypy==0.931
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/22 21:11
Affects Version/s: 3.3.0

The following new issues are detected when type checked with {{mypy==0.931}}


{code}
python/pyspark/pandas/base.py:879: error: "Sequence[Any]" has no attribute "tolist"  [attr-defined]
python/pyspark/sql/tests/test_pandas_udf_typehints_with_future_annotations.py:268: error: Incompatible return value type (got "floating[Any]", expected "float")  [return-value]
python/pyspark/sql/tests/test_pandas_udf_typehints.py:265: error: Incompatible return value type (got "floating[Any]", expected "float")  [return-value]

{code}
Bug: Fix flaky test: AdaptiveQueryExecSuite with Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/22 08:19
Affects Version/s: 3.3.0

Use maven test `AdaptiveQueryExecSuite` with Scala-2.13, the following exceptions will occur with a very small probability:
{code:java}
AdaptiveQueryExecSuite

- Logging plan changes for AQE *** FAILED ***
  java.util.ConcurrentModificationException: mutation occurred during iteration
  at scala.collection.mutable.MutationTracker$.checkMutations(MutationTracker.scala:43)
  at scala.collection.mutable.CheckedIndexedSeqView$CheckedIterator.hasNext(CheckedIndexedSeqView.scala:47)
  at scala.collection.StrictOptimizedIterableOps.filterImpl(StrictOptimizedIterableOps.scala:225)
  at scala.collection.StrictOptimizedIterableOps.filterImpl$(StrictOptimizedIterableOps.scala:222)
  at scala.collection.mutable.ArrayBuffer.filterImpl(ArrayBuffer.scala:43)
  at scala.collection.StrictOptimizedIterableOps.filterNot(StrictOptimizedIterableOps.scala:220)
  at scala.collection.StrictOptimizedIterableOps.filterNot$(StrictOptimizedIterableOps.scala:220)
  at scala.collection.mutable.ArrayBuffer.filterNot(ArrayBuffer.scala:43)
  at org.apache.spark.SparkFunSuite$LogAppender.loggingEvents(SparkFunSuite.scala:288)
  at org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.$anonfun$new$152(AdaptiveQueryExecSuite.scal{code}
Bug: Add trash feature to FileCommitProtocol.deleteWithJob
Status: In Progress | Priority: Minor | Resolution: 
Created: 13/Jan/22 08:27
Affects Version/s: 3.2.0

Currently function implementation of `FileCommitProtocol.deleteWithJob` was just calling fs.delete(). In our production, We often need to retrieve old data to analyze data problems. So I think  we better support trash feature for this.
Bug: Refactor TestUtils#configTestLog4j method to use log4j2 api
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jan/22 09:20
Affects Version/s: 3.3.0

SPARK-37795 Add a scalastyle rule to ban `org.apache.log4j` imports, but there is still one place to retain the imports of `org.apache.log4j`.
Bug: composite field name like `field name` doesn't work with Aggregate push down
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jan/22 06:00
Affects Version/s: 3.2.0


{code:java}
sql("SELECT SUM(`field name`) FROM h2.test.table")

org.apache.spark.sql.catalyst.parser.ParseException: 
extraneous input 'name' expecting <EOF>(line 1, pos 9)

	at org.apache.spark.sql.catalyst.parser.ParseErrorListener$.syntaxError(ParseDriver.scala:212)
	at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
	at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
	at org.antlr.v4.runtime.DefaultErrorStrategy.reportUnwantedToken(DefaultErrorStrategy.java:377)
	at org.antlr.v4.runtime.DefaultErrorStrategy.singleTokenDeletion(DefaultErrorStrategy.java:548)
	at org.antlr.v4.runtime.DefaultErrorStrategy.recoverInline(DefaultErrorStrategy.java:467)
	at org.antlr.v4.runtime.Parser.match(Parser.java:206)
	at org.apache.spark.sql.catalyst.parser.SqlBaseParser.singleMultipartIdentifier(SqlBaseParser.java:519)
{code}
Bug: Expose PeriodicCheckpointer/PeriodicRDDCheckpointer/PeriodicGraphCheckpointer to public for reusability
Status: Open | Priority: Minor | Resolution: 
Created: 09/Jan/22 02:18
Affects Version/s: 3.2.0

Currently, the {*}PeriodicCheckpointer{*}, *PeriodicRDDCheckpointer* and the *PeriodicGraphCheckpointer* are all private classes. It would help customers' fine grained management on RDD persistency/checkpoint lifecycle by exposing all these classes to public.
Bug: Broadcast Join throws HintErrorLogger for joins with multiple tables
Status: Open | Priority: Minor | Resolution: 
Created: 04/Jan/22 20:29
Affects Version/s: 3.1.2

Following query throws HintErrorLogger Warnings in v3.1.2. 

 
{code:java}
// code placeholder
val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " +
              " L1.v1 AS L1V1 " +
              " L4.* " +
              " FROM L1 " +
              " INNER JOIN L2 ON L2.id = L1.id " +
              " INNER JOIN L3 ON L3.id = L1.id " +
              " LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 "
            {code}
 

Following is the warning it thorws during runtime:
{code:java}
WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code}
 

The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4).

I have set \{{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3.

 

Let me know if you need more info.
Bug: Pyspark Pandas API: Cross and Conditional Merging
Status: In Progress | Priority: Minor | Resolution: 
Created: 01/Jan/22 02:24
Affects Version/s: 3.2.0

Pandas currently supports a `how="cross"` merge which provides a cartesian product of the left/right tables. This can be achieved by doing a `spark.sql.dataframe.join(..., on=None, how="inner")`.

Additionally, I am currently in the middle of adding conditional merging in the pandas API (see PR here: [https://github.com/pandas-dev/pandas/pull/42964|https://github.com/pandas-dev/pandas/pull/42964).]). This is much easier to achieve in spark, since the functionality is already available, and we can trivially expose it in the pyspark pandas API. Due to the demand  of this functionality (countless SO/pandas issues either asking how to do this, or asking questions that would be solved by this), I think that this would be worth adding even before it makes it into the core pandas API.

Will open a PR, which includes both, shortly.
Bug: Fix the comments on SQLQueryTestSuite.scala/ThriftServerQueryTestSuite.scala to more explicit
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Jan/22 02:44
Affects Version/s: 3.3.0

Currently, the comments on `SQLQueryTestSuite.scala` says:
{code:java}
 * To re-generate golden files for entire suite, run:
 * {{{
 *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *SQLQueryTestSuite" 
 * }}} {code}
But we have only `org.apache.spark.sql.SQLQueryTestSuite` for `*SQLQueryTestSuite`, so we'd better to recommend to use `org.apache.spark.sql.SQLQueryTestSuite` explicitly.

Also same for `ThriftServerQueryTestSuite.scala`.
Bug: Clean up deprecation compilation warning related to log4j2
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Jan/22 08:43
Affects Version/s: 3.3.0

[WARNING] [Warn] /spark-source/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala:64: [deprecation @ org.apache.spark.util.logging.DriverLogger.addLogAppender.fa | origin=org.apache.logging.log4j.core.appender.FileAppender.createAppender | version=] method createAppender in class FileAppender is deprecated
[WARNING] [Warn] /spark-source/core/src/test/scala/org/apache/spark/SparkFunSuite.scala:268: [deprecation @ org.apache.spark.SparkFunSuite.LogAppender.<init> | origin=org.apache.logging.log4j.core.appender.AbstractAppender.<init> | version=] constructor AbstractAppender in class AbstractAppender is deprecated
Bug: improve structured streaming window of calculated
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Jan/22 23:27
Affects Version/s: 3.2.0

Structed Streaming computes window by intermediate result windowId, and windowId computes window by CaseWhen.

We can use Flink's method of calculating window to write it, which is more easy to understand, simple and efficient
Bug: Backout agg build time metrics from sort aggregate
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Jan/22 23:41
Affects Version/s: 3.3.0

This is a followup of https://issues.apache.org/jira/browse/SPARK-37564 . I realize the agg build time metrics for sort aggregate is actually not correctly recorded. We don't have a hash build phase for sort aggregate, so there is really no way to measure so-called build time for sort aggregate. So here I make the change to back out the change introduced in [https://github.com/apache/spark/pull/34826] for agg build time metric.
Bug: Remove useless and duplicated configuration in ParquetFileFormat builderReader
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/Jan/22 03:11
Affects Version/s: 3.2.0

Currently, we called `ParquetWriteSupport.setSchema(requiredSchema, hadoopConf)` when ParquetFileFormat buildReaderWithPartition, and this helper method set `org.apache.spark.sql.parquet.row.attributes` and `parquet.writer.version`, actually,  the former is already set and latter is useless when parquetRead
Bug: Rename `MEMORY_MODE` in `VectorizedParquetRecordReader` to `memoryMode` 
Status: In Progress | Priority: Trivial | Resolution: 
Created: 25/Jan/22 03:49
Affects Version/s: 3.3.0

`MEMORY_MODE` is a `final` field, not a `static final` field,  the naming should be CamelCase
Bug: Clean up duplicated planner logic for window operator
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/Jan/22 00:15
Affects Version/s: 3.3.0

Both `WindowExec.scala` and `WindowInPandasExec.scala` have some duplicated logic regarded to query planning (`output`, `requiredChildDistribution/Ordering`, `outputPartitioning/Ordering`). We can move these logic into their common parent `WindowExecBase` to reduce duplication.
Bug: Improve test case inside SQL\Catalyst\Catalog
Status: In Progress | Priority: Trivial | Resolution: 
Created: 23/Jan/22 15:30
Affects Version/s: 3.2.0

Test case "{*}basic create and list partitions{*}" inside *ExternalCatalogSuite.scala* can be simplified by replacing set up code with a test utility function from the same suite: *newBasicCatalog()*
Bug: Use type match to simplify TestUtils#withHttpConnection
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Jan/22 09:28
Affects Version/s: 3.3.0

{code:java}
if (connection.isInstanceOf[HttpsURLConnection]) {
      ....
      connection.asInstanceOf[HttpsURLConnection].setSSLSocketFactory(sslCtx.getSocketFactory())
      connection.asInstanceOf[HttpsURLConnection].setHostnameVerifier(verifier) {code}
Bug: ORC read benchmark should enable vectorization for nested column
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Jan/22 01:10
Affects Version/s: 3.3.0

We added an initial struct benchmark for ORC in [https://github.com/apache/spark/pull/35100] . However the benchmark does not enable vectorization for nested column. For comparison between vectorized and non-vectorized, we should enable vectorization for nested column.
Bug: Aggregation with DISTINCT gives wrong results when dealing with literals
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 25/Jul/24 06:16
Affects Version/s: 3.0.0

Aggregation with *DISTINCT* gives wrong results when dealing with literals. It appears that this bug affects all (or most) released versions of Spark.

 

For example:
{code:java}
select count(distinct 1) from t{code}
returns 1, while the correct result should be 0.

 

For reference:
{code:java}
select count(1) from t{code}
returns 0, which is the correct and expected result.

 

In these examples, suppose that *t* is a table with any columns).
Bug: Create a unified SQL Scala interface shared by regular SQL and Connect.
Status: Open | Priority: Major | Resolution: 
Created: 17/Jul/24 00:55
Affects Version/s: 4.0.0

*Motivation*

Current the scala sql/core and connect API share the same API; connect implements a subset of the functionality of the sql/core API. The compatibility of the two implementations is enforced by MiMa checks.

While this sort of works for application development, it is not ideal for a couple of reasons:
 * An application developer needs to pick against which API they are going to develop while setting up their project (they need to select the correct dependencies). While it is true, that they can this change later, it does put a mental burden on de the developer. A much preferred solution would be to defer binding to an implementation until you run the code.
 * (Minor) the current setup confuses IDEs, and is more of a pain to work with especially for Spark developers.
 * Developing and maintaining Spark API is more difficult because of the added burden of working with MiMa and/or adding the same API in more places.
 * Connect testing is fairly anaemic. We have seen a couple of cases where connect behaves slightly different, and this could have been detected if connect was able to leverage Spark SQLs extensive testing.

*Goals*
 * Create a truly shared Scala API with two implementations. The goal is *not* to replace/simplify/reduce the current sql/core API we all love, the interface will only support the API shared between the implementations. An implementation can provide additional functionality (e.g. RDD centric methods for the sql/core implementation).
 * The common interface should cover all API supported by the current Connect Scala client.
 * Maintain as much binary compatibility with previous Spark releases as possible

*Design Notes*
 * We are going to try to make the interface very connect centric. Where possible we will implement functionality using the connect API.
 * .... TBD