av9ash/GitBugs huggingface dataset:
Bug: Prevent Multiple SortOrders for an Expression
Status: Open | Priority: Minor | Resolution: 
Created: 10/Dec/23 03:28
Affects Version/s: 4.0.0

Hello everyone,

This is my first contribution to the project. I welcome any feedback and edits to improve this pull request.Currently, it's possible to create redundant sort expressions in both Scala and Python APIs, leading to potentially incorrect and confusing SQL statements. For example:

Scala:
{code:java}
spark.range(10).orderBy($"id".desc.asc).show(){code}
Python:
{code:java}
spark.range(10).orderBy(f.desc('id'), ascending=False).show(){code}
 
Such usage generates SQL like order by id DESC NULLS LAST DESC NULLS LAST, causing non-descriptive error messages.

 

I look forward to your feedback and thank you for considering this contribution.
Bug: Optimize shuffle reading when both sort and combine are used.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Dec/23 09:48
Affects Version/s: 4.0.0

After the shuffle reader obtains the block, it will first perform a combine operation, and then perform a sort operation. It is known that both combine and sort may generate temporary files, so the performance may be poor when both sort and combine are used. In fact, combine operations can be performed during the sort process, and we can avoid the combine spill file.

 

I did not find any direct api to construct the shuffle which both sort and combine is used. But I can do like following code, here is a wordcount, and the output words is sorted.
{code:java}
sc.textFile(input).flatMap(_.split(" ")).map(w => (w, 1)).
reduceByKey(_ + _, 1).
asInstanceOf[ShuffledRDD[String, Int, Int]].setKeyOrdering(Ordering.String).
collect().foreach(println) {code}
Bug: Add ability to create databases/schemas via Catalog API
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/23 18:15
Affects Version/s: 3.5.0

As of Spark 3.5, the only way to create a database is via SQL. The Catalog API should offer an equivalent.

Perhaps something like:
{code:python}
spark.catalog.createDatabase(
    name: str,
    existsOk: bool = False,
    comment: str = None,
    location: str = None,
    properties: dict = None,
)
{code}

If {{schema}} is the preferred terminology, then we should use that instead of {{database}}.
Bug: Remove the check for `c>=0` from `ExternalCatalogUtils#needsEscaping`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Dec/23 07:32
Affects Version/s: 4.0.0

 
{code:java}
def needsEscaping(c: Char): Boolean = {
  c >= 0 && c < charToEscape.size() && charToEscape.get(c)
} {code}
 

 

The numerical range of Char in Scala is from 0 to 65,535, so `c>=0` is always true.
Bug: In Function DataSourceStrategy.translateFilterWithMapping, we need transfer cast expression to  data source for filtering
Status: Resolved | Priority: Minor | Resolution: Abandoned
Created: 13/Dec/23 11:49
Affects Version/s: 3.5.0

Considering this Situation:
 We create a partition table that created by source which is extends TableProvider, if we select data from  some specific partitions, choose partition dataType differ from table partition type leads partition can not be pushed down .
Bug: Remove the legacy datetime rebasing SQL configs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Dec/23 16:06
Affects Version/s: 4.0.0

Remove already deprecated SQL configs (alternatives to other configs):
- spark.sql.legacy.parquet.int96RebaseModeInWrite
- spark.sql.legacy.parquet.datetimeRebaseModeInWrite
- spark.sql.legacy.parquet.int96RebaseModeInRead
- spark.sql.legacy.parquet.datetimeRebaseModeInRead
- spark.sql.legacy.avro.datetimeRebaseModeInWrite
- spark.sql.legacy.avro.datetimeRebaseModeInRead
Bug: Exception when ordering by UDT in interpreted mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Dec/23 17:01
Affects Version/s: 3.3.3

In interpreted mode, ordering by a UDT will result in an exception. For example:
{noformat}
import org.apache.spark.ml.linalg.{DenseVector, Vector}

val df = Seq.tabulate(30) { x =>
  (x, x + 1, x + 2, new DenseVector(Array((x/100.0).toDouble, ((x + 1)/100.0).toDouble, ((x + 3)/100.0).toDouble)))
}.toDF("id", "c1", "c2", "c3")

df.createOrReplaceTempView("df")

// this works
sql("select * from df order by c3").collect

sql("set spark.sql.codegen.wholeStage=false")
sql("set spark.sql.codegen.factoryMode=NO_CODEGEN")

// this gets an error
sql("select * from df order by c3").collect
{noformat}
The second {{collect}} action results in the following exception:
{noformat}
org.apache.spark.SparkIllegalArgumentException: Type UninitializedPhysicalType does not support ordered operations.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.orderedOperationUnsupportedByDataTypeError(QueryExecutionErrors.scala:348)
	at org.apache.spark.sql.catalyst.types.UninitializedPhysicalType$.ordering(PhysicalDataType.scala:332)
	at org.apache.spark.sql.catalyst.types.UninitializedPhysicalType$.ordering(PhysicalDataType.scala:329)
	at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:60)
	at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:39)
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter$RowComparator.compare(UnsafeExternalRowSorter.java:254)
{noformat}
Note: You don't get an error if you use {{show}} rather than {{collect}}. This is because {{show}} will implicitly add a {{limit}}, in which case the ordering is performed by {{TakeOrderedAndProject}} rather than {{UnsafeExternalRowSorter}}.
Bug: Simplify the code to generate the Spark tarball `filename` in the `HiveExternalCatalogVersionsSuite`.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Dec/23 06:54
Affects Version/s: 4.0.0

{code:java}
val filename = VersionUtils.majorMinorPatchVersion(version) match {
  case Some((major, _, _)) if major > 3 => s"spark-$version-bin-hadoop3.tgz"
  case Some((3, minor, _)) if minor >= 3 => s"spark-$version-bin-hadoop3.tgz"
  case Some((3, minor, _)) if minor < 3 => s"spark-$version-bin-hadoop3.2.tgz"
  case Some((_, _, _)) => s"spark-$version-bin-hadoop2.7.tgz"
  case None => s"spark-$version-bin-hadoop2.7.tgz"
} {code}
Currently, the minimum tested version is Spark 3.3, so there is no need for complex case matching anymore.
Bug: Clean up initValue vs zeroValue semantics in SQLMetrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Dec/23 00:31
Affects Version/s: 3.5.0

The semantics of initValue and _zeroValue in SQLMetrics is a little bit confusing, since they effectively mean the same thing. Changing it to the following would be clearer, especially in terms of defining what an "invalid" metric is.
 
proposed definitions:
 
initValue is the starting value for a SQLMetric. If a metric has value equal to its initValue, then it should be filtered out before aggregating with SQLMetrics.stringValue().
 
zeroValue defines the lowest value considered valid. If a SQLMetric is invalid, it is set to zeroValue upon receiving any updates, and it also reports zeroValue as its value to avoid exposing it to the user programatically (concern previouosly addressed in SPARK-41442).

For many SQLMetrics, we use initValue = -1 and zeroValue = 0 to indicate that the metric is by default invalid. At the end of a task, we will update the metric making it valid, and the invalid metrics will be filtered out when calculating min, max, etc. as a workaround for SPARK-11013.
Bug: Fix the output name of pyspark.sql.functions.user and session_user
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Dec/23 07:00
Affects Version/s: 4.0.0

{code}
scala> spark.range(1).select(user()).show()
+--------------+
|current_user()|
+--------------+
|  hyukjin.kwon|
+--------------+
{code}
Bug: Allocate capacity of array list of TColumns by columns size in TRowSet generation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Dec/23 08:21
Affects Version/s: 4.0.0

Background:

ArrayList is created for TColumn value collections in RowSetUtils for TRowSet generation. Currently, they are created with Java's default capacity of 16, rather than by the number of columns, which could cause array copying in assembling each TColumn collections when the column number exceeds the default capacity.

 

Suggested solution:

Allocate enough capacity by columns size for assembling array list of TColumns in TRowSet generation.
Bug: If Hadoop is not installed and configured, can the Spark cluster read and write OBS in standalone mode?
Status: Open | Priority: Minor | Resolution: 
Created: 08/Dec/23 01:40
Affects Version/s: 3.4.1

If Hadoop is not deployed, PySpark APIs read data from OBS buckets and convert the data to RDD. How can I achieve it?

The following code reports an error: No FileSystem for scheme "obs",Can Spark read and write OBS without Hadoop installation and configuration?

And I'm not familiar with pyspark. Is the code wrong?
{code:java}
// code placeholder
from pyspark import SparkConf
from pyspark.sql import SparkSession

conf = SparkConf()
conf.set("spark.app.name", "read and write OBS")
conf.set("spark.security.credentials.hbase.enabled", "true")
conf.set("spark.hadoop.fs.obs.access.key", ak)
conf.set("spark.hadoop.fs.obs.secret.key", sk)
conf.set("spark.hadoop.fs.obs.endpoint", "http://xxx")
spark = SparkSession.builder.config(conf=conf).getOrCreate()

df = spark.read.json('obs://bucket_name/xxx.json')
df.coalesce(2).write.json("obs://bucket_name/", "overwrite") {code}
Bug: Test string representation of TestResult (pyspark.mllib.stat.test)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Dec/23 00:20
Affects Version/s: 4.0.0

https://app.codecov.io/gh/apache/spark/blob/master/python%2Fpyspark%2Fmllib%2Fstat%2Ftest.py#L28
Bug: Test SparkPandasNotImplementedError (pyspark.pandas.exceptions)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Dec/23 00:08
Affects Version/s: 4.0.0

https://app.codecov.io/gh/apache/spark/blob/master/python%2Fpyspark%2Fpandas%2Fexceptions.py
Bug: Add restriction parameters to dynamic partitions for writing datasource
Status: Open | Priority: Trivial | Resolution: 
Created: 11/Dec/23 01:34
Affects Version/s: 3.5.0

Add restriction parameters to dynamic partitions for writing datasource. 

`InsertIntoHiveTable 'limits the number of single dynamic partition writes, while' InsertIntoHadoopFsRelationCommand 'does not limit the number of single dynamic partitions and should be aligned with it
Bug: Fix `dev-run-integration-tests.sh` to use 17 for `JAVA_VERSION`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Dec/23 19:16
Affects Version/s: 4.0.0

The file dev-run-integration-tests.sh ([https://github.com/apache/spark/blob/master/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh)] hardcodes the Java version as Java 8. However, Spark requires Java 17 in multiple locations like the kvstore module.

When you try to run the integration test, you get the below error. This can be overriden by running 
{code:java}
dev-run-integration-tests.sh  --java-version 17 {code}
but it would be nice if the script was up to date.
Bug: docker based spark release script should use correct CRAN repo.
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 21/Jul/20 07:51
Affects Version/s: 2.4.6

While running, dev/create-release/do-release-docker.sh script, it is failing with following errors

{code}
[root@kyok-test-1 ~]# tail docker-build.log 
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 r-base : Depends: r-base-core (>= 4.0.2-1.1804.0) but it is not going to be installed
          Depends: r-recommended (= 4.0.2-1.1804.0) but it is not going to be installed
 r-base-dev : Depends: r-base-core (>= 4.0.2-1.1804.0) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates apt-transport-https &&   echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/' >> /etc/apt/sources.list &&   gpg --keyserver keyserver.ubuntu.com --recv-key E298A3A825C0D65DFD57CBB651716619E084DAB9 &&   gpg -a --export E084DAB9 | apt-key add - &&   apt-get clean &&   rm -rf /var/lib/apt/lists/* &&   apt-get clean &&   apt-get update &&   $APT_INSTALL software-properties-common &&   apt-add-repository -y ppa:brightbox/ruby-ng &&   apt-get update &&   $APT_INSTALL openjdk-8-jdk &&   update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java &&   $APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev     pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev &&   ln -s -T /usr/share/java/ivy.jar /usr/share/ant/lib/ivy.jar &&   curl -sL https://deb.nodesource.com/setup_4.x | bash &&   $APT_INSTALL nodejs &&   $APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip &&   pip install --upgrade pip && hash -r pip &&   pip install setuptools &&   pip install $BASE_PIP_PKGS &&   pip install $PIP_PKGS &&   cd &&   virtualenv -p python3 /opt/p35 &&   . /opt/p35/bin/activate &&   pip install setuptools &&   pip install $BASE_PIP_PKGS &&   pip install $PIP_PKGS &&   $APT_INSTALL r-base r-base-dev &&   $APT_INSTALL texlive-latex-base texlive texlive-fonts-extra texinfo qpdf &&   Rscript -e "install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" &&   Rscript -e "devtools::install_github('jimhester/lintr')" &&   $APT_INSTALL ruby2.3 ruby2.3-dev mkdocs &&   gem install jekyll --no-rdoc --no-ri -v 3.8.6 &&   gem install jekyll-redirect-from -v 0.15.0 &&   gem install pygments.rb' returned a non-zero code: 100

{code}
Bug: DSv2 SQL overwrite incorrectly uses static plan with hidden partitions
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jul/20 20:45
Affects Version/s: 3.0.0

The v2 analyzer rule {{ResolveInsertInto}} tries to detect when a static overwrite and a dynamic overwrite would produce the same result and will choose to use static overwrite in that case. It will only use a dynamic overwrite if there is a partition column without a static value and the SQL mode is set to dynamic.

{code:lang=scala}
val dynamicPartitionOverwrite = partCols.size > staticPartitions.size &&
          conf.partitionOverwriteMode == PartitionOverwriteMode.DYNAMIC
{code}

The problem is that {{partCols}} are the names of only partitions that are in the column list (identity partitions) and does not include hidden partitions, like {{days(ts)}}. As a result, this doesn't detect hidden partitions and use dynamic overwrite. Static overwrite is used instead; when a table has only hidden partitions, the static filter drops all table data.

This is a correctness bug because Spark will overwrite more data than just the set of partitions being written to in dynamic mode. The impact is limited because this rule is only used for SQL queries (not plans from DataFrameWriters) and only affects tables with hidden partitions.
Bug: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 17/Jul/20 12:04
Affects Version/s: 3.0.0

 when using hive on spark engine:

    FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session

hadoop version: 2.7.3 / hive version: 3.1.2 / spark version 3.0.0
Bug: Cartesian Product Hint cause data error
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 08/Jul/20 06:12
Affects Version/s: 3.0.0

{code:java}
spark-sql> select * from test4 order by a asc;
1 2
Time taken: 1.063 seconds, Fetched 4 row(s)20/07/08 14:11:25 INFO SparkSQLCLIDriver: Time taken: 1.063 seconds, Fetched 4 row(s)
spark-sql>select * from test5 order by a asc
1 2
2 2
Time taken: 1.18 seconds, Fetched 24 row(s)20/07/08 14:13:59 INFO SparkSQLCLIDriver: Time taken: 1.18 seconds, Fetched 24 row(s)spar
spark-sql>select /*+ shuffle_replicate_nl(test4) */ * from test4 join test5 where test4.a = test5.a order by test4.a asc ;
1 2 1 2
1 2 2 2
Time taken: 0.351 seconds, Fetched 2 row(s)
20/07/08 14:18:16 INFO SparkSQLCLIDriver: Time taken: 0.351 seconds, Fetched 2 row(s){code}
Bug: nullability of GetArrayStructFields is incorrect
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jul/20 08:21
Affects Version/s: 1.6.3

The following should be `Array([WrappedArray(1, null)])` instead of `Array([WrappedArray(1, 0)])`
{code:java}
import scala.collection.JavaConverters._
import org.apache.spark.sql.Row
import org.apache.spark.sql.types.{ArrayType, StructType}

val innerStruct = new StructType().add("i", "int", nullable = true)
val schema = new StructType().add("arr", ArrayType(innerStruct, containsNull = false))
val df = spark.createDataFrame(List(Row(Seq(Row(1), Row(null)))).asJava, schema)
df.select($"arr".getField("i")).collect
{code}
Bug: Build and run the Spark with test cases in Github Actions
Status: Resolved | Priority: Critical | Resolution: Done
Created: 09/Jul/20 13:35
Affects Version/s: 2.4.6

Last week and onwards, the Jenkins machines became very unstable for some reasons. 
  - Apparently, the machines became extremely slow. Almost all tests can't pass.
  - One machine (worker 4) started to have the corrupt .m2 which fails the build.
  - Documentation build fails time to time for an unknown reason in Jenkins machine specifically.

Almost all PRs are basically blocked by this instability currently.

This JIRA aims to run the tests in Github Actions.
  - To avoid depending on few persons who can access to the cluster. 
  - To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce.
  - To control the environment more flexibly.
  - Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost.
Bug: .m2 repository corruption happens
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 02/Jul/20 07:57
Affects Version/s: 2.4.8

Build task on Jenkins-worker4 often fails with dependency problem.
[https://github.com/apache/spark/pull/28971#issuecomment-652570066]
[https://github.com/apache/spark/pull/28971#issuecomment-652611025]
[https://github.com/apache/spark/pull/28971#issuecomment-652690849] [https://github.com/apache/spark/pull/28971#issuecomment-652611025]
[https://github.com/apache/spark/pull/28942#issuecomment-652842960]
[https://github.com/apache/spark/pull/28942#issuecomment-652835679]

These can be related to .m2 corruption.
Bug: Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies
Status: Open | Priority: Major | Resolution: 
Created: 22/Jul/20 07:30
Affects Version/s: 3.1.0

Spark has a lot of dependencies, many of them very common (e.g. Guava, Jackson). Also, versions of these dependencies are not updated as frequently as they are released upstream, which is totally understandable and natural, but which also means that often Spark has a dependency on a lower version of a library, which is incompatible with a higher, more recent version of the same library. This incompatibility can manifest in different ways, e.g as classpath errors or runtime check errors (like with Jackson), in certain cases.

 

Spark does attempt to "fix" versions of its dependencies by declaring them explicitly in its {{pom.xml}} file. However, this approach, being somewhat workable if the Spark-using project itself uses Maven, breaks down if another build system is used, like Gradle. The reason is that Maven uses an unconventional "nearest first" version conflict resolution strategy, while many other tools like Gradle use the "highest first" strategy which resolves the highest possible version number inside the entire graph of dependencies. This means that other dependencies of the project can pull a higher version of some dependency, which is incompatible with Spark.

 

One example would be an explicit or a transitive dependency on a higher version of Jackson in the project. Spark itself depends on several modules of Jackson; if only one of them gets a higher version, and others remain on the lower version, this will result in runtime exceptions due to an internal version check in Jackson.

 

A widely used solution for this kind of version issues is publishing of a "bill of materials" descriptor (see here: [https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html] and here: [https://docs.gradle.org/current/userguide/platforms.html#sub:bom_import]). This descriptor would contain all versions of all dependencies of Spark; then downstream projects will be able to use their build system's support for BOMs to enforce version constraints required for Spark to function correctly.

 

One example of successful implementation of the BOM-based approach is Spring: [https://www.baeldung.com/spring-maven-bom#spring-bom]. For different Spring projects, e.g. Spring Boot, there are BOM descriptors published which can be used in downstream projects to fix the versions of Spring components and their dependencies, significantly reducing confusion around proper version numbers.
Bug: Add batch write support on LevelDB to improve performance of HybridStore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/20 16:42
Affects Version/s: 3.0.1

The idea is to improve the performance of HybridStore by adding batch write support to LevelDB. https://issues.apache.org/jira/browse/SPARK-31608 introduces HybridStore. HybridStore will write data to InMemoryStore at first and use a background thread to dump data to LevelDB once the writing to InMemoryStore is completed. In the comments section of [https://github.com/apache/spark/pull/28412], Mridul Muralidharan mentioned using batch writing can improve the performance of this dumping process and he wrote the code of writeAll().

I did the comparison of the HybridStore switching time between one-by-one write and batch write on an HDD disk. When the disk is free, the batch-write has around 25% improvement, and when the disk is 100% busy, the batch-write has 7x - 10x improvement.

when the disk is at 0% utilization:

 
||log size, jobs and tasks per job||original switching time, with write()||switching time with writeAll()||
|133m, 400 jobs, 100 tasks per job|16s|13s|
|265m, 400 jobs, 200 tasks per job|30s|23s|
|1.3g, 1000 jobs, 400 tasks per job|136s|108s|

 

when the disk is at 100% utilization:
||log size, jobs and tasks per job||original switching time, with write()||switching time with writeAll()||
|133m, 400 jobs, 100 tasks per job|116s|17s|
|265m, 400 jobs, 200 tasks per job|251s|26s|

I also ran some write related benchmarking tests on LevelDBBenchmark.java and measured the total time of writing 1024 objects.

when the disk is at 0% utilization:

 
||Benchmark test||with write(), ms||with writeAll(), ms ||
|randomUpdatesIndexed|213.060|157.356|
|randomUpdatesNoIndex|57.869|35.439|
|randomWritesIndexed|298.854|229.274|
|randomWritesNoIndex|66.764|38.361|
|sequentialUpdatesIndexed|87.019|56.219|
|sequentialUpdatesNoIndex|61.851|41.942|
|sequentialWritesIndexed|94.044|56.534|
|sequentialWritesNoIndex|118.345|66.483|

 

when the disk is at 50% utilization:
||Benchmark test||with write(), ms||with writeAll(), ms||
|randomUpdatesIndexed|230.386|180.817|
|randomUpdatesNoIndex|58.935|50.113|
|randomWritesIndexed|315.241|254.400|
|randomWritesNoIndex|96.709|41.164|
|sequentialUpdatesIndexed|89.971|70.387|
|sequentialUpdatesNoIndex|72.021|53.769|
|sequentialWritesIndexed|103.052|67.358|
|sequentialWritesNoIndex|76.194|99.037|