av9ash/GitBugs huggingface dataset:
Bug: Use error classes in the compilation errors of grouping
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 17:28
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* groupingIDMismatchError
* groupingColInvalidError
* groupingSizeTooLargeError
* groupingMustWithGroupingSetsOrCubeOrRollupError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Use error classes in the execution errors of casting
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jan/22 17:34
Affects Version/s: 3.3.0

Migrate the following errors in QueryExecutionErrors:
* failedToCastValueToDataTypeForPartitionColumnError
* invalidInputSyntaxForNumericError
* cannotCastToDateTimeError
* invalidInputSyntaxForBooleanError
* nullLiteralsCannotBeCastedError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryExecutionErrorsSuite.
Bug: Need to update Partition URI after renaming table in InMemoryCatalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/22 12:09
Affects Version/s: 3.3.0

After renaming a partitioned table, select from the new table from InMemoryCatalog will get an empty result.

The following checkAnswer will fail as the result is empty.
{code:java}
sql(s"create table foo(i int, j int) using PARQUET partitioned by (j)")
sql("insert into table foo partition(j=2) values (1)")
sql(s"alter table foo rename to bar")
checkAnswer(spark.table("bar"), Row(1, 2)) {code}
To fix the bug, we need to update Partition URI after renaming a table in InMemoryCatalog
Bug: Fix ColumnVectorUtils.populate to handle CalendarIntervalType correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 07:12
Affects Version/s: 3.3.0

`ColumnVectorUtils.populate()` does not handle CalendarInterval type correctly - [https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java#L93-L94] . The CalendarInterval type is in the format of (months: int, days: int, microseconds: long) ([https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java#L58] ). However, the function above misses `days` field, and sets `microseconds` field in wrong position.

 

`ColumnVectorUtils.populate()` is used by Parquet ([https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java#L258] ) and ORC ([https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java#L171] )vectorized reader to read partition column. So technically Spark can potentially produce wrong result if reading table with CalendarInterval partition column. However I also notice Spark explicitly disallows writing data with CalendarInterval type ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L586] ), so it might not be a big deal for users. But it's worth to fix anyway.

 

Caveat: I found the bug when reading through the related code path, and I don't have experience in production for partition column with CalendarInterval type. I think it should be an obvious fix unless anyone more experienced could find some historical context. The code was introduced a long time ago where I couldn't find any more info why it was implemented as it is ([https://github.com/apache/spark/pull/11435] )
Bug: Spark should not dedup the groupingExpressions when the first child of Union has duplicate columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 09:16
Affects Version/s: 3.3.0

When the first child of Union has duplicate columns like select a, a from t1 union select a, b from t2, spark only use the first column to aggregate the results, which would make the results incorrect, and this behavior is inconsistent with other engines like PostgreSQL, MySQL. We could alias the attribute of the first child of union to resolve this, or you could argue that this is the feature of Spark SQL.

sample query:
select
a,
a
from values (1, 1), (1, 2) as t1(a, b)
UNION
SELECT
a,
b
from values (1, 1), (1, 2) as t2(a, b)

result is
(1,1)

result from PostgreSQL and MySQL
(1,1)
(1,2)
Bug: Structured streaming not working in history server when using LevelDB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/22 12:43
Affects Version/s: 3.1.2

In [SPARK-31953|https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68], structured streaming support is added to history server. However this does not work when spark.history.store.path is set to save app info using LevelDB.

This is because one of the keys of StreamingQueryData, runId,  is UUID type, which is not supported by LevelDB. When replaying event log file in history server, StreamingQueryStatusListener will throw an exception when writing info to the store, saying "java.lang.IllegalArgumentException: Type java.util.UUID not allowed as key.".

Example event log is provided in attachments. When opening it in history server with spark.history.store.path set to somewhere, no structured streaming info is available.
Bug: Use error classes in the parsing errors of properties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 17:03
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* cannotCleanReservedNamespacePropertyError
* cannotCleanReservedTablePropertyError
* invalidPropertyKeyForSetQuotedConfigurationError
* invalidPropertyValueForSetQuotedConfigurationError
* propertiesAndDbPropertiesBothSpecifiedError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Use error classes in the parsing errors of partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 16:58
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* emptyPartitionKeyError
* partitionTransformNotExpectedError
* descColumnForPartitionUnsupportedError
* incompletePartitionSpecificationError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Improve the implement of UnevaluableAggregate
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/Jan/22 09:53
Affects Version/s: 3.3.0

Currently, Spark provides UnevaluableAggregate to replace function with another function which must be exists in build-in functions, so UnevaluableAggregate make Spark could reuse the implement of build-in function.
However, the sub classes of UnevaluableAggregate looks cumbersome.
Bug: Make RuntimeReplaceable works for AggregateFunction
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Jan/22 03:47
Affects Version/s: 3.3.0

Currently, Spark provides RuntimeReplaceable to replace function with another function. The last function must be exists in build-in functions, so RuntimeReplaceable make Spark could reuse the implement of build-in function.
But RuntimeReplaceable not works for aggregate function.
Bug: Spark 3.2.1 breaks binary compatibility with Spark 3.2.0
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Jan/22 13:40
Affects Version/s: 3.2.1

[PR 35243|https://github.com/apache/spark/pull/35243] introduced a new parameter to class `Invoke` with a default value (`isDeterministic: Boolean = true`). Existing Spark libraries (such as [frameless|https://github.com/typelevel/frameless]) that invoke [Invoke|https://github.com/typelevel/frameless/blob/29961d549e332dddf5cd711ef699dde7460cc48a/dataset/src/main/scala/frameless/RecordEncoder.scala#L154] directly expect a method with 7 parameters, and the new version expects 8. If Frameless would recompile with Spark 3.2.1, the updated library will NOT be binary compatible with Spark 3.2.0. Adding default parameters to existing methods [should be avoided|https://github.com/jatcwang/binary-compatibility-guide#dont-adding-parameters-with-default-values-to-methods].

One way forward would be to revert the change in the constructor and introduce a second constructor or a companion method that takes all the 8 parameters.
Bug: Implement vectorized  DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/22 19:27
Affects Version/s: 3.2.0

SPARK-36879 implements the DELTA_BINARY_PACKED encoding which is for integer values, but does not implement the DELTA_BYTE_ARRAY encoding which is for string values. DELTA_BYTE_ARRAY encoding also requires the DELTA_LENGTH_BYTE_ARRAY encoding. Both these encodings need vectorized versions as the current implementation simply calls the non-vectorized Parquet library methods.
Bug: Add task partition id in metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 16:37
Affects Version/s: 3.3.0

There is no partition id in current metrics, it makes difficult to trace stage metrics, such as stage shuffle read, especially when there are stage retries. It is also impossible to check task metrics between different applications.
{code:java}
class TaskData private[spark](
    val taskId: Long,
    val index: Int,
    val attempt: Int,
    val launchTime: Date,
    val resultFetchStart: Option[Date],
    @JsonDeserialize(contentAs = classOf[JLong])
    val duration: Option[Long],
    val executorId: String,
    val host: String,
    val status: String,
    val taskLocality: String,
    val speculative: Boolean,
    val accumulatorUpdates: Seq[AccumulableInfo],
    val errorMessage: Option[String] = None,
    val taskMetrics: Option[TaskMetrics] = None,
    val executorLogs: Map[String, String],
    val schedulerDelay: Long,
    val gettingResultTime: Long) {code}
Adding partitionId in Task Data can not only make us easy to trace task metrics, also can make it possible to collect metrics for actual stage outputs, especially when stage retries.
Bug: Support SQL split_part function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/22 00:39
Affects Version/s: 3.3.0

`split_part()` is a commonly supported function by other systems such as Postgres and some other systems. The Spark equivalent  is `element_at(split(arg, delim), part)`



h5. Function Specificaiton

h6. Syntax

{code:java}
split_part(str, delimiter, partNum)
{code}

h6. Arguments
{code:java}
str: string type
delimiter: string type
partNum: Integer type
{code}

h6. Note
{code:java}
1. This function splits `str` by `delimiter` and return requested part of the split (1-based). 
2. If any input parameter is NULL, return NULL.
3. If the index is out of range of split parts, returns empty stirng.
4. If `partNum` is 0, throws an error.
5. If `partNum` is negative, the parts are counted backward from the end of the string
6. when delimiter is empty, str is considered not split thus there is just 1 split part. 
{code}

h6. Examples
{code:java}
> SELECT _FUNC_('11.12.13', '.', 3);
13
> SELECT _FUNC_(NULL, '.', 3);
NULL
> SELECT _FUNC_('11.12.13', '', 1);
'11.12.13'
{code}
Bug: SQL UI shows accurate Metrics with stage retries
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/22 16:42
Affects Version/s: 3.4.0

At present, the SQL UI page can show us the metrics of sql execution. However the metrics are not accurate with task or stage retries. This PR is used to provide accurate Metrics in the UI page, mainly based on SPARK-37831 .
Bug: Use relativePath for K8s remote file test in BasicTestsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 09:51
Affects Version/s: 3.3.0

*BEFORE*
{code:java}
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds)
[info]   The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452)
... {code}
*AFTER*
{code:java}
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds){code}
Bug: old columns should not be available after select or drop
Status: Open | Priority: Major | Resolution: 
Created: 18/Jan/22 15:59
Affects Version/s: 3.0.1

 
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.functions import col as col
spark = SparkSession.builder.appName('available_columns').getOrCreate()
df = spark.range(5).select((col("id")+10).alias("id2"))
assert df.columns==["id2"] #OK
try:
    df.select("id")
    error_raise = False
except:
    error_raise = True
assert error_raise #OK
df = df.drop("id") #should raise an error
df.filter(col("id")!=2).count() #returns 4, should raise an error
{code}
Bug: Undefined link function causing error in GLM that uses Tweedie family
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Jan/22 20:49
Affects Version/s: 3.1.2

I am trying to use the GLM regression with a Tweedie distribution so I can model insurance use cases. I have set up a very simple example adapted from the docs:


{code:python}
    def create_fake_losses_data(self):
        df = self._spark.createDataFrame([
            ("a", 100.0, 12, 1, Vectors.dense(0.0, 0.0)),
            ("b", 0.0, 12, 1, Vectors.dense(1.0, 2.0)),
            ("c", 0.0, 12, 1, Vectors.dense(0.0, 0.0)),
            ("d", 2000.0, 12, 1, Vectors.dense(1.0, 1.0)), ], ["user", "label", "offset", "weight", "features"])
        logging.info(df.collect())
        setattr(self, 'fake_data', df)
        try:
            glr = GeneralizedLinearRegression(
                family="tweedie", variancePower=1.5, linkPower=-1, offsetCol='offset')
            glr.setRegParam(0.3)
            model = glr.fit(df)
            logging.info(model)
        except Py4JJavaError as e:
            print(e)
        return self
{code}

This causes the following error:

*py4j.protocol.Py4JJavaError: An error occurred while calling o99.toString.
: java.util.NoSuchElementException: Failed to find a default value for link*
        at org.apache.spark.ml.param.Params.$anonfun$getOrDefault$2(params.scala:756)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.ml.param.Params.getOrDefault(params.scala:756)
        at org.apache.spark.ml.param.Params.getOrDefault$(params.scala:753)
        at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:41)
        at org.apache.spark.ml.param.Params.$(params.scala:762)
        at org.apache.spark.ml.param.Params.$$(params.scala:762)
        at org.apache.spark.ml.PipelineStage.$(Pipeline.scala:41)
        at org.apache.spark.ml.regression.GeneralizedLinearRegressionModel.toString(GeneralizedLinearRegression.scala:1117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)


I was under the assumption that the default value for link is None, if not defined otherwise.
Bug: Introduce a new interface on streaming data source to notify the latest seen offset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/22 11:03
Affects Version/s: 3.3.0

We figure out the case of streaming data source that knowing the latest seen offset when restarting query would be handy and useful to implement some feature. One useful case is enabling the data source to track the offset by itself, for the case where the external storage of data source is not exposing any API to provide the latest available offset.

We will propose a new interface on streaming data source, which indicates Spark to give the latest seen offset whenever the query is being restarted. For the first start of the query, the initial offset of the data source should be retrieved from calling initialOffset.
Bug: Spark MLlib FPGrowth not working with 40+ items in Frequent Item set
Status: Open | Priority: Major | Resolution: 
Created: 26/Jan/22 14:52
Affects Version/s: 3.2.0

We have been using Spark FPGrowth and it works well with millions of transactions (records) when the frequent items in the Frequent Itemset is less than 25. Beyond 25 it runs into computational limit. For 40+ items in the Frequent Itemset the process never return.

To reproduce, you can create a simple data set of 3 transactions with equal items (40 of them) and run FPgrowth with 0.9 support, the process never completes. Below is a sample data I have used to narrow down the problem:
|I1|I2|I3|I4|I5|I6|I7|I8|I9|I10|I11|I12|I13|I14|I15|I16|I17|I18|I19|I20|I21|I22|I23|I24|I25|I26|I27|I28|I29|I30|I31|I32|I33|I34|I35|I36|I37|I38|I39|I40|
|I1|I2|I3|I4|I5|I6|I7|I8|I9|I10|I11|I12|I13|I14|I15|I16|I17|I18|I19|I20|I21|I22|I23|I24|I25|I26|I27|I28|I29|I30|I31|I32|I33|I34|I35|I36|I37|I38|I39|I40|
|I1|I2|I3|I4|I5|I6|I7|I8|I9|I10|I11|I12|I13|I14|I15|I16|I17|I18|I19|I20|I21|I22|I23|I24|I25|I26|I27|I28|I29|I30|I31|I32|I33|I34|I35|I36|I37|I38|I39|I40|

 

While the computation grows (2^n -1) with each item in Frequent Itemset, it surely should be able to handle 40 or more items in a Frequest Itemset

 

Is this a FPGrowth implementation limitation,

are there any tuning parameters that I am missing? Thank you.
Bug: DS V2 supports list namespaces of MySQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/22 10:52
Affects Version/s: 3.3.0

Currently, JDBCTableCatalog.scala query namespaces by below.
         val rs = conn.getMetaData.getSchemas(null, db)
         while (rs.next()) {
           if (rs.getString(1) == db) return true;
         }
         false
But cannot get any information when using MySQL JDBC driver.
Bug: Use error classes in the compilation errors of casting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 17:20
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* upCastFailureError
* unsupportedAbstractDataTypeForUpCastError
* cannotUpCastAsAttributeError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Use error classes in the parsing errors of lateral join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 16:52
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* lateralJoinWithNaturalJoinUnsupportedError
* lateralJoinWithUsingJoinUnsupportedError
* unsupportedLateralJoinTypeError
* invalidLateralJoinRelationError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Incorrect query ordering with flatMap() and distinct()
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 28/Jan/22 16:42
Affects Version/s: 3.0.2

I have a Dataset of non-unique identifiers that I can use with {{Dataset::flatMap()}} to create multiple rows with sub-identifiers for each id. When I run the code below, the {{limit(2)}} call is placed _after_ the call to {{flatMap()}} in the optimized logical plan. This unexpectedly yields only 2 rows, when I would expect it to yield 6.

{code:java}
StructType idSchema = DataTypes.createStructType(List.of(DataTypes.createStructField("id", DataTypes.LongType, false)));
StructType flatMapSchema = DataTypes.createStructType(List.of(
    DataTypes.createStructField("id", DataTypes.LongType, false),
    DataTypes.createStructField("subId", DataTypes.LongType, false)
));Dataset<Row> inputDataset = context.sparkSession().createDataset(
    LongStream.range(0,5).mapToObj((id) -> RowFactory.create(id)).collect(Collectors.toList()),
    RowEncoder.apply(idSchema)
);
return inputDataset
    .distinct()
    .limit(2)
    .flatMap((Row row) -> {
        Long id = row.getLong(row.fieldIndex("id"));        return LongStream.range(6,8).mapToObj((subid) -> RowFactory.create(id, subid)).iterator();
    }, RowEncoder.apply(flatMapSchema));  {code}
When run, the above code produces something like:
||id||subID||
|0|6|
|0|7|

But I would expect something like:
||id||subID||
|1|6|
|1|7|
|1|8|
|0|6|
|0|7|
|0|8|
Bug: Add OUTLIER_NO_FALLBACK executor roll policy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/22 23:33
Affects Version/s: 3.3.0

Currently executor rolling will always kill one executor every {{{}spark.kubernetes.executor.rollInterval{}}}. This may not be optimal in cases where the executor metric isn't an outlier compared to other executors. There is a cost associated with killing executors (ramp-up time for new executors for example) which applications may not want to incur for non-outlier executors.

 

This ticket would add the ability to only kill executors if they are outliners via the introduction of a new roll policy.
Bug: AQE can change bhj to smj if no extra shuffle introduce
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 04:41
Affects Version/s: 3.3.0

An example to reproduce the bug.

{code:java}
create table t1 as select 1 c1, 2 c2;
create table t2 as select 1 c1, 2 c2;
create table t3 as select 1 c1, 2 c2;

set spark.sql.adaptive.autoBroadcastJoinThreshold=-1;

select /*+ merge(t3) */ * from t1
left join (
select c1 as c from t3
) t3 on t1.c1 = t3.c
left join (
select /*+ repartition(c1) */ c1 from t2
) t2 on t1.c1 = t2.c1;

{code}

The key to produce this bug is that a bhj convert to smj/shj without introducing extra shuffe and AQE does not think the join can be planned as bhj.
Bug: Hive script transform with order by and limit will return fake rows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/22 22:58
Affects Version/s: 3.1.2

For example:

{noformat}
create or replace temp view t as
select * from values
(1),
(2),
(3)
as t(a);

select transform(a)
USING 'cat' AS (a int)
FROM t order by a limit 10;
{noformat}
This returns:
{noformat}
NULL
NULL
NULL
1
2
3
{noformat}
Without {{order by}} and {{limit}}, the query returns:
{noformat}
1
2
3
{noformat}
Spark script transform does not have this issue. That is, if {{spark.sql.catalogImplementation=in-memory}}, Spark does not return fake rows.