av9ash/GitBugs huggingface dataset:
Bug: Murmur3 and xxHash64 implementations do not produce the correct results on big-endian platforms
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Sep/20 13:45
Affects Version/s: 3.0.1

The Murmur3 and xxHash64 implementations in Spark do not produce the correct results on big-endian systems. This causes test failures on my target platform (s390x).

These hash functions require that multi-byte chunks be interpreted as integers encoded in *little-endian* byte order. This requires byte reversal when using multi-byte unsafe operations on big-endian platforms.

I have a PR ready for discussion and review.
Bug: Fix the "Relation: view text" test in DataSourceV2SQLSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Sep/20 20:21
Affects Version/s: 3.1.0

The existing code just defines a function literal and doesn't execute it:
{code:java}
test("Relation: view text") {
  val t1 = "testcat.ns1.ns2.tbl"
  withTable(t1) {
    withView("view1") { v1: String =>
      sql(s"CREATE TABLE $t1 USING foo AS SELECT id, data FROM source")
      sql(s"CREATE VIEW $v1 AS SELECT * from $t1")
      checkAnswer(sql(s"TABLE $v1"), spark.table("source"))
    }
  }
}
{code}
Bug: Struct field names should not change after normalizing floats
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Sep/20 04:37
Affects Version/s: 3.0.2

This ticket aims at fixing a minor bug when normalizing floats for struct types;
{code}
scala> import org.apache.spark.sql.execution.aggregate.HashAggregateExec
scala> val df = Seq(Tuple1(Tuple1(-0.0d)), Tuple1(Tuple1(0.0d))).toDF("k")
scala> val agg = df.distinct()
scala> agg.explain()
== Physical Plan ==
*(2) HashAggregate(keys=[k#40], functions=[])
+- Exchange hashpartitioning(k#40, 200), true, [id=#62]
   +- *(1) HashAggregate(keys=[knownfloatingpointnormalized(if (isnull(k#40)) null else named_struct(col1, knownfloatingpointnormalized(normalizenanandzero(k#40._1)))) AS k#40], functions=[])
      +- *(1) LocalTableScan [k#40]

scala> val aggOutput = agg.queryExecution.sparkPlan.collect { case a: HashAggregateExec => a.output.head }
scala> aggOutput.foreach { attr => println(attr.prettyJson) }
### Final Aggregate ###
[ {
  "class" : "org.apache.spark.sql.catalyst.expressions.AttributeReference",
  "num-children" : 0,
  "name" : "k",
  "dataType" : {
    "type" : "struct",
    "fields" : [ {
      "name" : "_1",
                ^^^
      "type" : "double",
      "nullable" : false,
      "metadata" : { }
    } ]
  },
  "nullable" : true,
  "metadata" : { },
  "exprId" : {
    "product-class" : "org.apache.spark.sql.catalyst.expressions.ExprId",
    "id" : 40,
    "jvmId" : "a824e83f-933e-4b85-a1ff-577b5a0e2366"
  },
  "qualifier" : [ ]
} ]

### Partial Aggregate ###
[ {
  "class" : "org.apache.spark.sql.catalyst.expressions.AttributeReference",
  "num-children" : 0,
  "name" : "k",
  "dataType" : {
    "type" : "struct",
    "fields" : [ {
      "name" : "col1",
                ^^^^
      "type" : "double",
      "nullable" : true,
      "metadata" : { }
    } ]
  },
  "nullable" : true,
  "metadata" : { },
  "exprId" : {
    "product-class" : "org.apache.spark.sql.catalyst.expressions.ExprId",
    "id" : 40,
    "jvmId" : "a824e83f-933e-4b85-a1ff-577b5a0e2366"
  },
  "qualifier" : [ ]
} ]
{code}
Bug: Add spark.sql.maxMetadataStringLength config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Sep/20 04:29
Affects Version/s: 3.1.0

Add a new config `spark.sql.maxMetadataStringLength`. This config aims to limit metadata value length, e.g. file location.

Found that metadata has been abbreviated by `...` when tried to add some test in `SQLQueryTestSuite`. That caused we can't replace the location value by `className` since the `className` has been abbreviated.
Bug: Tests should fail if errors happen when generating expr code
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Sep/20 03:01
Affects Version/s: 3.1.0

I noticed that the code generation of SafeProjection failed in the existing test (https://issues.apache.org/jira/browse/SPARK-32828) but it passed because FALLBACK was set at CODEGEN_FACTORY_MODE (by default) in SharedSparkSession. To get aware of these failures, I think its worth setting CODEGEN_ONLY at CODEGEN_FACTORY_MODE.
Bug: Add sinkParameter to check sink options robustly in DataStreamReaderWriterSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Sep/20 11:24
Affects Version/s: 2.4.6

{code:java}
org.apache.spark.sql.streaming.test.DataStreamReaderWriterSuite.SPARK-32832: later option should override earlier options for start()

java.util.NoSuchElementException: key not found: path
{code}
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128498/testReport/org.apache.spark.sql.streaming.test/DataStreamReaderWriterSuite/SPARK_32832__later_option_should_override_earlier_options_for_start__/
Bug: Default options for metadata & nullable when creating StructType
Status: In Progress | Priority: Minor | Resolution: 
Created: 11/Sep/20 00:47
Affects Version/s: 3.0.1

When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571] function. This function then internally calls [StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430] for each field in the list. This function requires that you must have all keys populated:
 * name
 * type
 * nullable
 * metadata

 

name and type most definitely have to be required. However, if you take a look at the constructor for [StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395] there are default arguments for nullable and metadata. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
Bug: Change the number of partitions to zero when a range is empty with WholeStageCodegen disabled or falled back
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Sep/20 11:10
Affects Version/s: 3.1.0

If WholeStageCodegen effects, the number of partitions of an empty range will be changed to zero. But it doesn't changed when WholeStageCodegen is disabled or falled back.
Bug: Cast from a derived user-defined type to a base type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Sep/20 07:28
Affects Version/s: 3.1.0

This ticket targets at fixing an existing bug below in `UserDefinedTypeSuite`;This PR intends to fix a existing bug below in `UserDefinedTypeSuite`;
{code:java}
[info] - SPARK-19311: UDFs disregard UDT type hierarchy (931 milliseconds)16:22:35.936 WARN org.apache.spark.sql.catalyst.expressions.SafeProjection: Expr codegen error and falling back to interpreter modeorg.apache.spark.SparkException: Cannot cast org.apache.spark.sql.ExampleSubTypeUDT@46b1771f to org.apache.spark.sql.ExampleBaseTypeUDT@31e8d979. at org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeCastFunction(Cast.scala:891) at org.apache.spark.sql.catalyst.expressions.CastBase.doGenCode(Cast.scala:852) at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:147)    ...{code}
Bug: Add test case for get null columns using SparkGetColumnsOperation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Sep/20 03:52
Affects Version/s: 3.1.0

In Spark 3.0.0, the SparkGetColumnsOperation can not recognize NULL columns but now we can because the side effect of https://issues.apache.org/jira/browse/SPARK-32696
Bug: while parsing the data from parquet to case class
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 09/Sep/20 07:46
Affects Version/s: 2.4.3

Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableDouble cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableLong

 
I am trying to parse the the parquet file to dataset using case class with encoders. 

this is very much known error , but don't see the resolution.

could you please help on this.

 

[https://stackoverflow.com/questions/51300978/spark-error-reading-parquet?rq=1]

 similar like this :: https://issues.apache.org/jira/browse/SPARK-17477
Bug: Replace IN predicate of continuous range with boundary checks
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Sep/20 09:36
Affects Version/s: 3.1.0

This expression 
{code:java}
select a from t where a in (1, 2, 3, 3, 4){code}
can be translated to 
{code:java}
select a from t where a >= 1 and a <= 4 {code}
This would speed up parquet row group filter (currently or(or(or(or(or(eq(x, 1), eq(x, 2)), eq(x, 3), eq(x, 4.....)))) and make query more compact
Bug: Improve performance for some slow DPP tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Sep/20 09:54
Affects Version/s: 3.1.0

The whole `DynamicPartitionPruningSuite` takes about 2 min on my laptop (either AE on or off). The slowest tests are `test("simple inner join triggers DPP with mock-up tables")` and `test("cleanup any DPP filter that isn't pushed down due to expression id clashes")`, which totally take about 1 min.

We can reuse existing test tables or use smaller tables to reduce the cost. After that, the two tests takes only about 1 sec in total, leading to 2x speedup for the suite.
Bug: Spark Streaming
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 22/Sep/20 08:08
Affects Version/s: 2.4.5

Hey there,

I'm using this spark streaming job which integrated with Kafka (and manage its offsets commitions at Kafka itself),

The problem is when I have a failure I want to repeat the work on  those offset ranges (that something went wrong with them) , therefore I catch the exception and NOT commit (with commitAsync) this range.

However I notice it keeps proceeding (without any commit made).

moreover I removed later all the commitAsync calls and I the stream keep proceeding!

I guess there might be any inner cache or something that helps the streaming job to consume the entries from Kafka.

 

Could you please advice?
Bug: Encoders::bean doc incorrectly states maps are not supported
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 11/Sep/20 22:22
Affects Version/s: 2.4.6

The documentation for the bean method in the Encoders class currently states:
{quote}collection types: only array and java.util.List currently, map support is in progress
{quote}
But map support appears to work properly and has been available since 2.1.0 according to SPARK-16706.  Documentation should be updated to match what is / is not actually supported (Set, Queue, etc?).
Bug: Add bucket scan info in explain output of FileSourceScanExec
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Sep/20 07:08
Affects Version/s: 3.1.0

As a followup from discussion in [https://github.com/apache/spark/pull/29804#discussion_r493229395] , currently for `FileSourceScanExec` explain output, there's no information to indicate whether the table is read as bucketed table. And if not read as bucketed table, what's the reason behind it. Add this info into `FileSourceScanExec` explain output, can help users/developers understand query plan more easily without spending a lot of time debugging why table is not read as bucketed table.
Bug: Docker buildx now requires --push
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Sep/20 00:06
Affects Version/s: 3.0.0

The buildx command has been updated and now requires --push to be added to ensure the image is pushed. To fix this please edit `./bin/docker-image-tool.sh` and verify that your images are pushed with the latest docker buildx
Bug: Add DisableUnnecessaryBucketedScan rule to AQE queryStagePreparationRules
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Sep/20 17:11
Affects Version/s: 3.1.0

As a followup comment from [https://github.com/apache/spark/pull/29804#issuecomment-700650620] , will add the physical plan rule DisableUnnecessaryBucketedScan into AQE AdaptiveSparkPlanExec.queryStagePreparationRules. This mostly needs some more unit tests crafting.
Bug: FeatureHasher does not check categoricalCols in inputCols
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Sep/20 10:16
Affects Version/s: 2.3.0

doc related to {{categoricalCols}}:
{code:java}
Numeric columns to treat as categorical features. By default only string and boolean columns are treated as categorical, so this param can be used to explicitly specify the numerical columns to treat as categorical. Note, the relevant columns must also be set in inputCols. {code}
 

However, the check to make sure {{categoricalCols}} in {{inputCols}} was never implemented:

for example, in 2.4.7 and current master(3.1.0):
{code:java}
scala> import org.apache.spark.ml.feature._
import org.apache.spark.ml.feature._
scala> import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.ml.linalg.{Vector, Vectors}

scala> val df = Seq((2.0, 1, "foo"),(3.0, 2, "bar")).toDF("real", "int", "string")
df: org.apache.spark.sql.DataFrame = [real: double, int: int ... 1 more field]
scala> val n = 100
n: Int = 100
scala> val hasher = new FeatureHasher().setInputCols("int", "string").setCategoricalCols(Array("real")).setOutputCol("features").setNumFeatures(n) 
hasher: org.apache.spark.ml.feature.FeatureHasher = featureHasher_fbe05968b33f
scala> hasher.transform(df).show
+----+---+------+--------------------+
|real|int|string|            features|
+----+---+------+--------------------+
| 2.0|  1|   foo|(100,[2,39],[1.0,...|
| 3.0|  2|   bar|(100,[2,42],[2.0,...|
+----+---+------+--------------------+

{code}
 

CategoricalCols "real" is not in inputCols ("int", "string").

 

I think there are two options:

1, remove this comment  "Note, the relevant columns must also be set in inputCols. ", since this requirement seems unnecessary;

2, add a check to make sure all CategoricalCols are in inputCols.
Bug: No need for some big-endian specific code paths in {On,Off}HeapColumnVector
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Sep/20 08:51
Affects Version/s: 3.0.1

There is no need for a separate code path for big-endian platforms in putFloats and putDoubles in OnHeapColumnVector and OffHeapColumnVector. Since SPARK-26985 was fixed the values have been copied in native byte order so the code required to perform this operation can be the same on both little- and big-endian platforms.
Bug: Avoid per-row join type check in stream-stream join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 11/Sep/20 05:32
Affects Version/s: 3.1.0

In stream-stream join (`StreamingSymmetricHashJoinExec`), we can optimize to avoid per row check for join type ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L486-L492] ), to create the method before the loop of reading row. Similar optimization (i.e. create auxiliary method/variable per different join type beforehand) has been done in batch join world (`SortMergeJoinExec`, `ShuffledHashJoinExec`). This is a minor change.
Bug: Investigate UTF8_BINARY regression
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Nov/24 17:19
Affects Version/s: 4.0.0

The regression was detected in the recent benchmark. This issue aims to track the regression in order to fix it before Apache Spark 4.0.0 release.

 

https://github.com/apache/spark/pull/48721#discussion_r1828031983
Bug: Re-enable `SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession are isolated`
Status: Open | Priority: Critical | Resolution: 
Created: 01/Nov/24 14:17
Affects Version/s: 3.5.2

https://github.com/apache/spark/actions/runs/10915451051/job/30295259985
This test case needs a refactor to use only 2 threads instead of 3, because having 3 threads is not guaranteed in CI.
Bug: (RCE) Command Execution Vulnerability in Spark
Status: Open | Priority: Critical | Resolution: 
Created: 06/Nov/24 10:05
Affects Version/s: 3.5.3

Spark allow user to deploy jar packet with`extraJavaOptions` ,

but spark has incomplete check of it, leading to command execution.

 

using below command can see the /abc file was created: 

spark-submit --class JavaWordCount --master yarn --deploy-mode client --conf spark.executor.extraJavaOptions="\`touch\$IFS/abc\`" test.jar
Bug: Enable QUERY_TAG for SQL Session in Spark SQL
Status: Open | Priority: Critical | Resolution: 
Created: 13/Nov/24 20:18
Affects Version/s: 3.5.3

As Spark SQL becomes more powerful for both analytics and ELT (with big T), we see more tools are generating and executing SQL to transform data.

*Session* is a very important mechanism for lineage and usage/cost tracking, especially for the multi-statement ELT cases. *Tagging* a {color:#ff0000}series{color} of query statements with the higher level business *context* (such as project, flow_name, job_name, batch_id, start_data_dt, end_data_dt, owner, cost_group, ...) can provide tremendous observability improvement without much overhead. It is not efficient to collect and analyze the scattered query UUID and try to group them together to reconstruct the SESSION. But it is quite easy to allow the SQL client to set the tags when the session is established.
 * Presto has *Session Properties*
 * Trino has {*}X-Trino-Session{*}, *X-Trino-Client-Info* and *X-Trino-Client-Tags* to carry a list of K/V
 * Snowflake has *QUERY_TAG* to make observability much easier and efficient
 * Redshift supports tagging for query as well

It will be great that Spark SQL can set a paved path/recipe for the workload/cost analysis/observability based on the session QUERY_TAG, so that the whole community can follow instead reinventing the wheel.
Bug: Support high precision DECIMAL256 in SparkSQL and Dataframe
Status: Open | Priority: Critical | Resolution: 
Created: 13/Nov/24 02:38
Affects Version/s: 3.5.3

Financial services can't use DOUBLE to handle money values, and DECIMAL(38,x) is not sufficient for [Crypto|https://cypherpunk.org/2019/10/06/cryptocurrency-units/] because we need to handle DECIMAL(54,24) or DECIMAL(60,24) {color:#ff0000}*without*{color} precision loss.

Originally, long time ago, SparkSQL supported Java BigDecimal for some time, but later it reverted the support to be *compatible* with the HIVE standard (only up to Decimal(38,x)).
{quote}[https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala] fully uses BigInteger and BigDecimal already.
{quote}
----
Postgres can handle financial data with very high precision. If Databricks can support this feature, it will be a great differentiator (compared with other Big Data analytical systems) for the entire Financial Service sector.

 
The request is to support Java BigDecimal and/or Decimal256 (Apache Arrow supports it already) in Spark Tungsten Dataframe, SQL and Unity Catalog.

 
 * [/java/format/src/main/java/org/apache/arrow/flatbuf/Decimal.java|https://github.com/apache/arrow/blob/main/java/format/src/main/java/org/apache/arrow/flatbuf/Decimal.java]
 ** [/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java|https://github.com/apache/arrow/blob/main/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java]
 ** [/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java#L46|https://github.com/apache/arrow/blob/main/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java#L46]
 ** [/java/vector/src/test/java/org/apache/arrow/vector/TestDecimal256Vector.java|https://github.com/apache/arrow/blob/main/java/vector/src/test/java/org/apache/arrow/vector/TestDecimal256Vector.java]
Bug: Improve SparkFatalException Propagation when OutOfMemoryError occurs on BroadcastExchangeExec building small table to broadcast
Status: Open | Priority: Major | Resolution: 
Created: 21/Nov/24 22:03
Affects Version/s: 3.4.4

When BroadcastHashJoin builds small table to broadcast by BroadcastExchangeExec, if OutOfMemoryError has occurred on driver, BroadcastExchangeExec throws SparkFatalException which wraps SparkException. However, SparkException' s cause property comes as null by missing actual cause which is `java.lang.OutOfMemoryError: Java heap space` in following example. Actual cause info is also useful to propagate Throwable.getCause / getCause.* properties to clients. Repro test case has been added.
Bug: Reorder BindParameters and ResolveIdentifierClause rules to converge to fixpoint faster
Status: Open | Priority: Major | Resolution: 
Created: 14/Nov/24 17:11
Affects Version/s: 4.0.0

The current relative order of rules is:
 # ResolveRelations
 # ResolveIdentifierClause
 # BindParameters

These rules interact in exactly the opposite way as they are listed:
 * BindParameters always operates on leafs
 * ResolveIdentifierClause turns constant expressions (typically parameters) into UnresolvedRelations
 * ResolveRelations is needed to progress resolved identifier clauses
 
We propose to move BindParameters and ResolveIdentifierClause up.
Bug: SubqueryBroadcast is not reused in the SparkPlanGraph
Status: Open | Priority: Major | Resolution: 
Created: 20/Nov/24 19:30
Affects Version/s: 3.5.3

_emphasized text_Currently, the SparkPlanGraph construction ignores that SubqueryBroadcast can be used as a subquery. This results in duplicating graph nodes.

I uploaded two screenshots to show the difference between the current UI graph Vs. the expected.
Bug: date_add/date_sub functions' second parameter support long type
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/24 09:53
Affects Version/s: 3.4.3

[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "date_add(2023-01-14, (FLOOR(((datediff(2024-09-08, 2023-01-14) / 7) - 1)) * 7))" due to data type mismatch: Parameter 2 requires the ("INT" or "SMALLINT" or "TINYINT") type, however "(FLOOR(((datediff(2024-09-08, 2023-01-14) / 7) - 1)) * 7)" has the type "BIGINT"

The second parameter value was returned by anther function, and it was supported by spark version 2.4
Bug: Internal error from EXECUTE IMMEDIATE and spark.sql w/ parameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/24 12:26
Affects Version/s: 4.0.0

When spark.sql() is invoked with parameters (named or positionals), and tries to execute a query via EXECUTE IMMEDIATE using parameters too, the call fails w/ on an assert:

{code:scala}
spark.sql("execute immediate ‘select ?' using 1", Map("param1" -> "1"))
{code}

The output:

{code:java}
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
	at org.apache.spark.SparkException$.internalError(SparkException.scala:107)
	at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:623)
	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:636)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:258)
{code}