av9ash/GitBugs huggingface dataset:
Bug: Count on distinct asterisk not equals to the count with column names provided
Status: Reopened | Priority: Minor | Resolution: 
Created: 14/Jun/22 11:58
Affects Version/s: 3.1.3

Hi everyone,

We came across a case when count distinct with asterisk produce incorrect result comparing to count distinct when all columns provided.

Example provide below:
{noformat}
scala> val df = Seq(
     |     (1655172,1463032,"PHON","US",null,1),
     |     (1655172,1061329,"DESK","AU",null,3),
     |     (1655172,1334977,"MOBILE","US",null,23),
     |     (1655172,1165470,"PHON","CR",null,12),
     |     (1655172,1021215,"PHON","CA","USD",11)).toDF
df: org.apache.spark.sql.DataFrame = [_1: int, _2: int ... 4 more fields]

scala> df.printSchema
root
 |-- _1: integer (nullable = false)
 |-- _2: integer (nullable = false)
 |-- _3: string (nullable = true)
 |-- _4: string (nullable = true)
 |-- _5: string (nullable = true)
 |-- _6: integer (nullable = false)

scala> df.createOrReplaceTempView("a_table")

scala> spark.sql("select count(1), count(distinct(*)), count(distinct(_1, _2, _3, _4, _5, _6)) from a_table").show(false)
+--------+--------------------------------------+----------------------------------------------------------------------------+
|count(1)|count(DISTINCT _1, _2, _3, _4, _5, _6)|count(DISTINCT named_struct(_1, _1, _2, _2, _3, _3, _4, _4, _5, _5, _6, _6))|
+--------+--------------------------------------+----------------------------------------------------------------------------+
|5       |1                                     |5                                                                           |
+--------+--------------------------------------+----------------------------------------------------------------------------+

{noformat}
We understand that this is somehow related to null values but in our understanding asterisk should mimic same behavior as all columns provided.

If there is any documentation about this It would be nice to read.

Any help would be appreciated. 

Michael
Bug: Upgrade hive-storage-api to 2.7.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/22 05:41
Affects Version/s: 3.2.1

HIVE-25190: Fix many small allocations in BytesColumnVector

 
{code:java}
Caused by: java.lang.RuntimeException: Overflow of newLength. smallBuffer.length=1073741824, nextElemLength=408101
	at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.increaseBufferSpace(BytesColumnVector.java:311)
	at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:182)
	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.setColumn(WriterImpl.java:179)
	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.setColumn(WriterImpl.java:268)
	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.setColumn(WriterImpl.java:223)
	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:294)
	at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:105)
	at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:157)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:176)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:86)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:93)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:312)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1534)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:319) {code}
Bug: Add AQE invalid plan check
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jun/22 02:49
Affects Version/s: 3.3.0

AQE logical optimization rules can lead to invalid physical plans as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes.

Logical optimizations, on the other hand, are not (and should not be) aware of such restrictions. So a general solution here is to check for invalid plans and throw exceptions, which can be caught by AQE replanning process. And if such an exception is captured, AQE can void the current replanning result and keep using the latest valid plan.
Bug: Support ports definition in executor pod template
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/22 16:08
Affects Version/s: 3.4.0

*Description:*

Spark on K8S allows to open additional ports for custom purposes on the driver pod via the pod template, but ignores the port specification in the executor pod template. Port specifications from the pod template should be preserved (and extended) for both drivers and executors.

*Scenario:*

I want to run functionality in the executor that exposes data on an additional port. In my case, this is monitoring data exposed by Spark's JMX metrics sink via the JMX prometheus exporter java agent https://github.com/prometheus/jmx_exporter -- the java agent opens an extra port inside the container, but for prometheus to detect and scrape the port, it must be exposed in the K8S pod resource.
(More background if desired: This seems to be the "classic" Spark 2 way to expose prometheus metrics. Spark 3 introduced a native equivalent servlet for the driver, but for the executor, only a rather limited set of metrics is forwarded via the driver, and that also follows a completely different naming scheme. So the JMX + exporter approach still turns out to be more useful for me, even in Spark 3)

Expected behavior:

I add the following to my pod template to expose the extra port opened by the JMX exporter java agent

spec:
  containers:
  - ...
    ports:
    - containerPort: 8090
      name: jmx-prometheus
      protocol: TCP

Observed behavior:

The port is exposed for driver pods but not for executor pods


*Corresponding code:*

driver pod creation just adds ports
[https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala] (currently line 115)

val driverContainer = new ContainerBuilder(pod.container)
...
  .addNewPort()
...
  .addNewPort()

while executor pod creation replaces the ports
[https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala] (currently line 211)

val executorContainer = new ContainerBuilder(pod.container)
...
  .withPorts(requiredPorts.asJava)


The current handling is incosistent and unnecessarily limiting. It seems that the executor creation could/should just as well preserve pods from the template and add extra required ports.


*Workaround:*

It is possible to work around this limitation by adding a full sidecar container to the executor pod spec which declares the port. Sidecar containers are left unchanged by pod template handling.
As all containers in a pod share the same network, it does not matter which container actually declares to expose the port.
Bug: Fix possible null pointer in MySQLDialect listIndexes
Status: Closed | Priority: Minor | Resolution: Invalid
Created: 30/Jun/22 07:01
Affects Version/s: 3.4.0

h3. when:
{quote}val indexComment = rs.getString("Index_comment") {color:#172b4d}*return null*{color}
{quote}
h3. then:
{quote}if (indexComment.nonEmpty) properties.put("COMMENT", indexComment) {color:#172b4d}*throw NullPointerException*{color}
{quote}
h3. finally:
{quote}MySQLDialect.listIndexes {color:#172b4d}return incorrect result(ingore exist index).{color}
{quote}
Bug: SHOW CREATE TABLE should suggest 'AS SERDE' for Hive tables with unsupported serde configurations
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/22 01:52
Affects Version/s: 3.0.0

If you run `SHOW CREATE TABLE` against a Hive table which uses an unsupported Serde configuration, Spark will return an error message like
{code:java}
org.apache.spark.sql.AnalysisException: Failed to execute SHOW CREATE TABLE against table rcFileTable, which is created by Hive and uses the following unsupported serde configuration
 SERDE: org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe INPUTFORMAT: org.apache.hadoop.hive.ql.io.RCFileInputFormat OUTPUTFORMAT: org.apache.hadoop.hive.ql.io.RCFileOutputFormat {code}
which is confusing to end users.

In this situation, I think the error should suggest `SHOW CREATE TABLE ... AS SERDE` to users (similar to other error messages in this code path).
Bug: History server page and API are using inconsistent conditions to filter running applications
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jun/22 14:53
Affects Version/s: 3.2.1

When opening summary page, history server follows this logic:
* If there's completed/incomplete application, page will add script in response, using AJAX to call the REST API to get the filtered list.
* If there's no such application, page will only return a message telling nothing found.

Issue is that page and REST API are using different conditions to filter applications. In HistoryPage, an application is considered as completed as long as the last attempt is completed. But in ApplicationListResource, all attempts should be completed. This brings inconsistency and will cause issue in a corner case.

In driver, event queues have capacity to protect memory. When there's too many events, some of them will be dropped and the event log file will be incomplete. For an application with multiple attempts, there's possibility that the last attempt is completed, but the previous attempts is considered as incomplete due to loss of application end event.

For this type of application, page thinks it is completed, but the API thinks it is still running. When opening summary page:
* When checking completed applications, page will call script, but API returns nothing.
* When checking incomplete applications, page returns nothing.

So the user won't be able to see this app in history server.
Bug: Fix TPCDSQueryBenchmark Measuring Performance of Wrong Query Results
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/22 20:41
Affects Version/s: 3.0.3

GenTPCDSData uses the schema defined in `TPCDSSchema` that contains char(N). When GenTPCDSData generates parquet, that pads spaces for strings whose lengths are < N.

When TPCDSQueryBenchmark reads data from parquet generated by GenTPCDSData, it uses schema from the parquet file and keeps the paddings. Due to the extra spaces, string filter queries of TPC-DS fail to match. For example, q13 query results are all nulls and returns too fast because string filter does not meet any rows.

Therefore, TPCDSQueryBenchmark is benchmarking with wrong query results and that is inflating some performance results.

I am exploring two possible solutions now
1. Call `CREATE TABLE tableName schema USING parquet LOCATION path` before reading. This is what Spark TPC-DS unit tests are doing
2. Change char to string in the schema. This is what [databricks data generator|https://github.com/databricks/spark-sql-perf] is doing

TPCDSQueryBenchmark was ported from databricks/spark-sql-perf in https://issues.apache.org/jira/browse/SPARK-35192

History related char issue [https://lists.apache.org/thread/rg7pgwyto3616hb15q78n0sykls9j7rn]
Bug: Extend EdgePartition1D with Destination based Strategy
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Jun/22 13:52
Affects Version/s: 3.4.0

Currently, the *[EdgePartition1D|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/PartitionStrategy.scala#L101]*  use only the source vertex ID.

If doing topological like algorithm on graph, directly partition based on destination vertex ID would improve the performance significantly.

The proposal here is either adding a new case object like *EdgePartition1DDst* or update the *EdgePartition1D* such that user can configure the src or dst directly.
Bug: add state utils to StreamTest to check states during streaming queries
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Jun/22 13:40
Affects Version/s: 3.3.0

The framework for implementing tests for streaming queries ([StreamTest|https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala]) doesn't have any StreamActions that allows to check the state for a specific key during a stateful streaming query. 

For this purpose, two new Stream Actions need to be implemented:
{code:java}
case class CheckEmptyState[K](expectedKey: K) extends StreamAction
case class CheckNonEmptyState[K, V](expectedKey: K, expectedState: V) extends StreamAction{code}
The CheckEmptyState StreamAction will check if doesn't exists a state for the specified key.

The CheckNonEmptyState StreamAction will check if there is a state of type V, for the specified key, which is equals to the specified state.
Bug: Allow all Reader or Writer settings to be provided as options
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jun/22 21:03
Affects Version/s: 3.3.0

Almost all Reader or Writer settings can be provided via individual calls to `.option()` or by providing a map to `.options()`.

There are notable exceptions, though, like:
 * read/write format
 * write mode
 * write partitionBy, bucketBy, and sortBy

These settings must be provided via dedicated method calls.

Why not make it so that _all_ settings can be provided as options? Is there a design reason not to do this?

Any given DataFrameReader or DataFrameWriter (along with the streaming equivalents) should be able to "export" all of its settings as a map of options, and then in turn be reconstituted entirely from that map of options.
{code:python}
reader1 = spark.read.option("format", "parquet").option("path", "/data")
options = reader.getOptions()
reader2 = spark.read.options(options)

# reader1 and reader2 are configured identically
data1 = reader1.load()
data2 = reader2.load()
data1.collect() == data2.collect(){code}
Some of these special directives like `bucketBy` will need to be broken up to fit the structure of an option.
{code:python}
# instead of this
data.write.bucketBy(42, "name")
# someone would be able to do this
data.write.option("bucketByCount", "42").option("bucketByColumn", "name"){code}
Complex read or write options aren't new to Spark. We already use them to configure [detailed JDBC connections|https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html], for example.

So why would someone care about this? Mainly, because they want to capture read or write settings declaratively somewhere, like in a YAML file, and use them to drive a Spark job's behavior.

Right now we are 80% of the way there. Unless there is a design obstacle in the way, it seems good to go the last 20% and just make it standard for every and any read or write directive to be supported via the option interface.
Bug: Upgrade RoaringBitmap from 0.9.28 to 0.9.30
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jun/22 06:03
Affects Version/s: 3.4.0

[https://github.com/RoaringBitmap/RoaringBitmap/compare/0.9.28...0.9.30]

[fix previousValue value smaller than first value |https://github.com/RoaringBitmap/RoaringBitmap/commit/ad116feb97edd76b86cce6927bbab94f431bc7a4]
Bug: Add relevance score for nDCG evaluation in MLLIB
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/22 10:12
Affects Version/s: 3.2.1

The current implementation of ndcgAt function treats relevance score as binary (as written [Document|[https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.mllib.evaluation.RankingMetrics.html#pyspark.mllib.evaluation.RankingMetrics.ndcgAt]|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.mllib.evaluation.RankingMetrics.html#pyspark.mllib.evaluation.RankingMetrics.ndcgAt]).

However, it is better to extend this to accept a user-defined relevance score to calculate nDCG more flexibly.
Bug: Reuse orcSchema when push down Orc predicates
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/22 09:24
Affects Version/s: 3.4.0

`OrcUtils.readCatalystSchema` method will trigger a file IO to read `orcSchema` when Orc pushes down predicates.

After SPARK-37463:

-  For `OrcFileFormat`, we can reuse [a read `orcSchema`](https://github.com/apache/spark/blob/cc0bf563b8caea21da5692f05e34b5f77e002ab9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala#L146) to save this file read IO when Orc pushes down predicates

- For `OrcPartitionReaderFactory`  we can achieve the same goal by moving `OrcFile.createReader` before `pushDownPredicates`.
Bug: Write event logs in a continuous manner
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jun/22 13:49
Affects Version/s: 3.3.0

We are using Spark Thrift Server as a service to run Spark SQL queries.

Currently Spark writes *eventlogs* once the job is done/ gracefully killed.

This means that in case of Spark thrift Server when it runs as one job in case of unexpected error ( i.e. OOM ) we cannot access the logs after, as it doesn't write them due to the unexpected shut down.

We would like to suggest as an improvement the functionality for sparks *to write the event logs to a cloud storage in a continuous manner.*
Bug: “SPARK_HOME” usage in bin/docker-image-tool.sh might be explained
Status: Open | Priority: Minor | Resolution: 
Created: 23/Jun/22 04:16
Affects Version/s: 3.3.0

Hello, I am using docker-image-tool.sh on building docker images for k8s, as described in here:
[https://spark.apache.org/docs/latest/running-on-kubernetes.html#user-identity] 

 

I found out the “SPARK_HOME” env variable is utilized on build step, like this:
{code:java}
// https://github.com/apache/spark/blob/master/bin/docker-image-tool.sh#L177-L179

(cd $(img_ctx_dir base) && docker build $NOCACHEARG "${BUILD_ARGS[@]}" \
-t $(image_ref spark) \
-f "$BASEDOCKERFILE" .){code}
 

It means that if the user set the SPARK_HOME to other path on the local, and runs this script in another spark directory,

it would perform build on the directory where SPARK_HOME is set and would not work as intended.

 

Since this might become hard to debug, and this script usage is suggested to common users in public doc,

I thought further explanation on “SPARK_HOME” or “is_dev_build” usage in this script might be needed, like this:
{code:java}
// docker-image-tool.sh
 
Usage: $0 [options] [command]
Builds or pushes the built-in Spark Docker image.
…
Examples:
- Build image in current directory
  env SPARK_HOME=. $0 build {code}
Thank you.
Bug: V2SessionCatalog should not throw NoSuchDatabaseException in loadNamespaceMetadata
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/22 18:39
Affects Version/s: 3.3.0

DROP NAMESPACE IF EXISTS

{table}

 

if a catalog doesn't overrides `namespaceExists` it by default uses `loadNamespaceMetadata` and in case a `db` not exists loadNamespaceMetadata throws a `NoSuchDatabaseException` which is not catched and we see failures even with `if exists` clause. One such use case we observed was in iceberg table a post test clean up was failing with `NoSuchDatabaseException` now.

 

Found {color:#000000}V2SessionCatalog `{color}{color:#00627a}loadNamespaceMetadata{color}{color:#000000}` {color}was also throwing the same unlike `{color:#000000}JDBCTableCatalog`{color}

ref a stack trace :
{quote}Database 'db' not found
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'db' not found
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireDbExists(SessionCatalog.scala:219)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getDatabaseMetadata(SessionCatalog.scala:284)
at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadNamespaceMetadata(V2SessionCatalog.scala:247)
at org.apache.iceberg.spark.SparkSessionCatalog.loadNamespaceMetadata(SparkSessionCatalog.java:97)
at org.apache.spark.sql.connector.catalog.SupportsNamespaces.namespaceExists(SupportsNamespaces.java:98)
at org.apache.spark.sql.execution.datasources.v2.DropNamespaceExec.run(DropNamespaceExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
{quote}
Bug: The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/22 11:05
Affects Version/s: 3.3.0

The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats, example:

spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create

*before*

gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.snappy.parquet

*after*

gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.zstd.parquet ...
Bug: Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/22 12:37
Affects Version/s: 3.4.0

{{ExpressionSet ++}} method in the master branch a little slower than the branch-3.3 with Scala-2.13

 

For example, write a microbenchmark as follows and run with Scala 2.13:
{code:java}
val valuesPerIteration = 100000
    val benchmark = new Benchmark("Test ExpressionSet ++ ", valuesPerIteration, output = output)
    val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
    val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
    val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)

    benchmark.addCase("Test ++") { _: Int =>
      for (_ <- 0L until valuesPerIteration) {
        initialSet ++ setToAddWithSameDeterministicExpression
      }
    }

    benchmark.run() {code}
*branch-3.3 result:*
 
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
Test ExpressionSet ++ :                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Test ++                                              14             16           4          7.2         139.1       1.0X
 {code}
 
*master result :*
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
Test ExpressionSet ++ :                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Test ++                                              16             19           5          6.1         163.9       1.0X
 {code}