av9ash/GitBugs huggingface dataset:
Bug: Spark: Getting SQLException: Unsupported type -102 reading from Oracle
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Mar/23 05:05
Affects Version/s: 3.3.2

 
{code:java}
Exception in thread "main" org.apache.spark.SparkSQLException: Unrecognized SQL type -102
    at org.apache.spark.sql.errors.QueryExecutionErrors$.unrecognizedSqlTypeError(QueryExecutionErrors.scala:832)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:225)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:308)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:308)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.getQueryOutputSchema(JDBCRDD.scala:70)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:242)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:37)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
 
{code}
oracle driver
{code:java}
<dependency>
    <groupId>com.oracle.database.jdbc</groupId>
    <artifactId>ojdbc8</artifactId>
    <version>21.9.0.0</version>
</dependency>{code}
 

oracle sql:

 
{code:java}
CREATE TABLE "ORDERS" 
   (    "ORDER_ID" NUMBER(9,0) NOT NULL ENABLE, 
    "ORDER_DATE" TIMESTAMP (3) WITH LOCAL TIME ZONE NOT NULL ENABLE, 
    "CUSTOMER_NAME" VARCHAR2(255) NOT NULL ENABLE, 
    "PRICE" NUMBER(10,5) NOT NULL ENABLE, 
    "PRODUCT_ID" NUMBER(9,0) NOT NULL ENABLE, 
    "ORDER_STATUS" NUMBER(1,0) NOT NULL ENABLE, 
     PRIMARY KEY ("ORDER_ID")
  USING INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "LOGMINER_TBS"  ENABLE, 
     SUPPLEMENTAL LOG DATA (ALL) COLUMNS
   ) SEGMENT CREATION IMMEDIATE 
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "LOGMINER_TBS"
 
{code}
[~yao]
Bug: PySpark type hint returns Any for methods on GroupedData
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/23 16:44
Affects Version/s: 3.3.0

Since upgrading to PySpark 3.3.x, type hints for
{code:java}
df.groupBy(...).count(){code}
are now returning Any instead of DataFrame, causing type inference issues downstream. This used to be correctly typed prior to 3.3.x.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2006
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:45
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2006* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2007
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:46
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2007* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2010
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:49
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2010* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Upgrade protobuf-java to 3.22.2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/23 02:50
Affects Version/s: 3.5.0

* [https://github.com/protocolbuffers/protobuf/releases/tag/v22.1]
 * [https://github.com/protocolbuffers/protobuf/releases/tag/v22.2]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2013
Status: Open | Priority: Minor | Resolution: 
Created: 17/Mar/23 12:51
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2013* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Delayed scheduling doesn’t work in some situations in local mode if different localities present in loaded files leading to tasks getting stuck
Status: Open | Priority: Minor | Resolution: 
Created: 26/Mar/23 12:23
Affects Version/s: 3.3.2

I stumbled on the following issue when running spark in local mode where part of the loaded files were present in the same host as the spark and others not.

Symptom: Some task in larger job would consistently get stuck without no immediately clear errors in logs. My hope/expectation would have been that even if some tasks would have failed to complete on during some expected time the job would have retried the task or failed completely with some exception and not just get stuck forever.

Workaround:
Setting spark.locality.wait.node to 0s seemed to fix the getting stuck in my environment.

Potential root cause:
I managed to reproduce the issue with the spark codebase by adding a test case to FileSourceStrategySuite, which is trying to read two files to a table where another is located in the same host as the local spark executor and another in some other host. https://github.com/apache/spark/commit/c23db78863c7342ae7b7bc3922a200a523e45538

While digging into the issue with the debugger I finally noticed that the LocalSchedulerBackend is missing the reviveThread present in CoarseGrainedSchedulerBackend, which forces the periodic calling of resourceOffsers in TaskSchedulerImpl and not just in taskUpdates.

Potential fix:
Add the revive thread also to LocalSchedulerBackend.
I don’t really have understanding of the codebase whether simply adding the revive thread to LocalSchedulerBackend could have some unwanted side effects.

Questions/Observations:
Should delayed scheduling work at all in local mode?
This issue probably effect also the case where instead of local file there is file which is rack local to the executor and then some non rack local file, which are being loaded.
Bug: Add exit code in SparkListenerApplicationEnd
Status: Open | Priority: Minor | Resolution: 
Created: 28/Mar/23 14:27
Affects Version/s: 3.3.2

When an application ends, the {{SparkListener}} receives a final event called {{{}SparkListenerApplicationEnd{}}}.

This event currently only includes a timestamp, but it would be beneficial to also include the exitCode of the application. This additional information would provide insight into whether the application succeeded or failed.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2004
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:41
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2004* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2008
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 12:46
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2008* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Expose VectorTypes API for DataSourceV2 Batch Scans
Status: Open | Priority: Minor | Resolution: 
Created: 13/Mar/23 18:33
Affects Version/s: 3.3.2

SparkPlan's vectorType's attribute can be used to [specialize codegen|https://github.com/apache/spark/blob/5556cfc59aa97a3ad4ea0baacebe19859ec0bcb7/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala#L151] however [BatchScanExecBase|https://github.com/apache/spark/blob/6b6bb6fa20f40aeedea2fb87008e9cce76c54e28/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala] does not override this so we DSv2 sources do not get any benefit of concrete class dispatch.

This proposes adding an override to BatchScanExecBase which delegates to a new default method on [PartitionReaderFactory|https://github.com/apache/spark/blob/f1d42bb68d6d69d9a32f91a390270f9ec33c3207/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/PartitionReaderFactory.java] to expose vectoryTypes:

{{
default Optional<Iterable<String>> getVectorTypes()

{ return Optional.empty(); } }}
Bug: Generalize handling of metadata attributes in FileSourceStrategy
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Mar/23 11:38
Affects Version/s: 3.4.1

A first step towards allowing file format implementations to inject custom metadata fields into plans is to make the handling of metadata attributes in `FileSourceStrategy` more generic.

Today in `FileSourceStrategy` , the lists of constant and generated metadata fields are created manually, checking for known generated fields on one hand and considering the remaining fields as constant metadata fields. We need instead to introduce a way of declaring metadata fields as generated or constant directly in `FileFormat` and propagate that information to `FileSourceStrategy`.
Bug: in spark-py docker images, arrowkeys do not work in (scala) spark-shell
Status: Open | Priority: Minor | Resolution: 
Created: 14/Mar/23 12:19
Affects Version/s: 3.1.3

i tested this for 3.1.3 and 3.3.1 from [https://hub.docker.com/r/apache/spark-py/tags]

while it works for pyspark, it does not for the scala spark-shell.

it seems this is due to scala REPL using {{jline}} for input management.
 * creating a \{{.inputrc}} file with mapping for the arrow keys. this wouldn't work
 * finally, building and running from {{dev/create-release/spark-rm/Dockerfile}} with jline installed as of the Dockerfile, things worked.

likely not limited to the {{spark-py}} images only.

i'd do a PR, but am unsure if this is even the right Dockerfile to contribute to in order to fix the docker hub images...

{code:sh}
diff --git a/dev/create-release/spark-rm/Dockerfile b/dev/create-release/spark-rm/Dockerfile
--- dev/create-release/spark-rm/Dockerfile
+++ dev/create-release/spark-rm/Dockerfile
@@ -71,9 +71,9 @@
   $APT_INSTALL nodejs && \
   # Install needed python packages. Use pip for installing packages (for consistency).
   $APT_INSTALL python-is-python3 python3-pip python3-setuptools && \
   # qpdf is required for CRAN checks to pass.
-  $APT_INSTALL qpdf jq && \
+  $APT_INSTALL qpdf jq libjline-java && \
   pip3 install $PIP_PKGS && \
   # Install R packages and dependencies used when building.
   # R depends on pandoc*, libssl (which are installed above).
   # Note that PySpark doc generation also needs pandoc due to nbsphinx

{code}
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2005
Status: Open | Priority: Minor | Resolution: 
Created: 17/Mar/23 12:41
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2005* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Use SecureRandom, instead of Random in security sensitive contexts
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Mar/23 16:15
Affects Version/s: 3.2.3

Most uses of Random in spark are either in test cases or where we need a pseudo random number which is repeatable.
The following are usages where moving from Random to SecureRandom would be useful

a) HttpAuthUtils.createCookieToken
b) ThriftHttpServlet.RAN
Bug: CVE-2020-13936 request for upgrading version of Velocity
Status: Open | Priority: Minor | Resolution: 
Created: 23/Mar/23 02:44
Affects Version/s: 3.2.3

An attacker that is able to modify Velocity templates may execute arbitrary Java code or run arbitrary system commands with the same privileges as the account running the Servlet container. This applies to applications that allow untrusted users to upload/modify velocity templates running Apache Velocity Engine versions up to 2.2.

The current version of Velocity that spark relies on is 1.5, should we upgrade to version 2.3?
Bug: SELECT * LIKE 'pattern' FROM ....
Status: Open | Priority: Minor | Resolution: 
Created: 24/Mar/23 17:53
Affects Version/s: 3.5.0

SparkSQL supports *regex_column_names.*
[https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select.html]
However, support depends on a config: spark.sql.parser.quotedRegexColumnNames
The reason is that it overloads proper identifier names.

Here we propose a cleaner, compatible API:

SELECT * LIKE 'pattern' ...

The semantic should follow common regular expression patterns used for the LIKE operator with the caveat that it should obey identifier case insensitivity setting.
Bug: Subexpression elimination support shortcut expression
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/23 15:19
Affects Version/s: 3.5.0

The subexpression may not need to eval even if it appears more than once.
e.g., {{{}if(or(a, and(b, b))){}}}, the expression {{b}} would be skipped if {{a}} is true.
Bug: Avoid evaluate more than once for the variables from the left side in the FullOuter SMJ condition
Status: In Progress | Priority: Minor | Resolution: 
Created: 22/Mar/23 14:12
Affects Version/s: 3.4.0

Codegen issue for FullOuter SMJ,  for example
{code}
    val df1 = spark.range(5).select($"id".as("k1"))
    val df2 = spark.range(10).select($"id".as("k2"))
    df1.join(df2.hint("SHUFFLE_MERGE"),
        $"k1" === $"k2" % 3 && $"k1" + 3 =!= $"k2" && $"k1" + 5 =!= $"k2", "full_outer")
{code}
the join condition *$"k1" + 3 =!= $"k2" && $"k1" + 5 =!= $"k2"* both will evaluate the variable *k1* and caused the codegen failed.
Bug: Upgrade GCS connector to 2.2.11.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/23 20:10
Affects Version/s: 3.4.0

Upgrade the [GCS Connector|https://github.com/GoogleCloudDataproc/hadoop-connectors/tree/v2.2.11/gcs] bundled in the Spark distro from version 2.2.7 to 2.2.11. The new release contains multiple bug fixes and enhancements discussed in the [Release Notes|https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.11/gcs/CHANGES.md]. Notable changes include:
* Improved socket timeout handling.
* Trace logging capabilities.
* Fix bug that prevented usage of GCS as a [Hadoop Credential Provider|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html].
* Dependency upgrades.
* Support OAuth2 based client authentication.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_2003
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Mar/23 12:35
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2003* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: spark binaries are not added to $PATH in public docker hub spark-py
Status: Open | Priority: Minor | Resolution: 
Created: 14/Mar/23 12:22
Affects Version/s: 3.1.3

i tested this for 3.1.3 and 3.3.1 from https://hub.docker.com/r/apache/spark-py/tags.
for the user 185 `/opt/spark/bin` is not added to {{$PATH}} which hampers the user experience