av9ash/GitBugs huggingface dataset:
Bug: Add option for k-fold cross-validation to CrossValidator
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Jul/20 09:33
Affects Version/s: 3.1.0

*What changes were proposed in this pull request?*

I have added a `method` parameter to `CrossValidator.scala` to allow the user to choose between repeated random sub-sampling cross-validation (current behavior) and _k_-fold cross-validation (optional new behavior). The default method is random sub-sampling cross-validation.

If _k_-fold cross-validation is chosen, the new behavior is as follows:
 # Instead of splitting the input dataset into _k_ training and validation sets, I split them into _k_ folds; for each fold of training, one of the _k_ splits is selected for validation, and the others are unioned together for training.
 # Instead of caching each training and validation set _k_ times, I cache each of the folds once.
 # Instead of waiting for every model to finish training on fold _n_ before moving on to fold _n+1_, new fold/model combinations will be trained as soon as resources are available.
 # Instead of creating one `Future` per model for each fold in series, all `Future`s for each fold & parameter grid pair are created and trained in parallel.
 # A new `Int` parameter is added to the `Future` (now `Future[Int, Double]` instead of `Future[Double]`) in order to keep track of which `Future` belongs to which parameter grid.

*Why are the changes needed?*

These changes allow the user to choose between repeated random sub-sampling cross-validation (current behavior) and _k_-fold cross-validation (optional new behavior). These changes:
 1. allow the user to choose between two types of cross-validation.
 2. (If _k_-fold is chosen) only require caching the entire dataset once (instead of _k_ times in repeated random sub-sampling cross-validation, as it does now).
 3. (if _k_-fold is chosen) free resources to train new model/fold combinations as soon as the previous one finishes. Currently, a model can only train one fold at a time. If _k_-fold is chosen, the added functionality will allow the `fit` to train multiple folds at once for the same model, and, in the case of a grid search, allow it to train multiple model/fold combinations at once, without needing to wait for the slowest model to fit the first fold before moving onto the second.

*Does this PR introduce _any_ user-facing change?*

Yes. This PR introduces the `setMethod` method to `CrossValidator`. If the `method` parameter is not set, the behavior will be the same as it has always been.

*How was this patch tested?*

Unit tests will be added.
Bug: Make `mvn clean` remove `metastore_db` and `spark-warehouse`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/20 12:14
Affects Version/s: 3.1.0

Now we support two version of build-in hive and there are some test generated meta data not in target dir like `spark-warehouse`, `metastore_db`, they don't cleanup automatically when we run `mvn clean` command.

So if we run `mvn clean test -pl sql/hive -am -Phadoop-2.7 -Phive -Phive-1.2 ` , the `metastore_db` dir will created and meta data will remains after test complete.

Then we need manual cleanup `metastore_db` directory to ensure `mvn clean test -pl sql/hive -am -Phadoop-2.7 -Phive` command can succeed because the residual metastore data is not compatible.

`spark-warehouse` will also cause test failure in some data residual scenarios too.
Bug: Remove empty children of union
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jul/20 12:37
Affects Version/s: 3.1.0

Empty relation children of a union can be removed.
e.g. the plan of
{noformat}
SELECT c FROM t UNION ALL SELECT c FROM t WHERE FALSE{noformat}
is currently:
{noformat}
== Physical Plan ==
Union
:- *(1) Project [value#219 AS c#222]
:  +- *(1) LocalTableScan [value#219]
+- LocalTableScan <empty>, [c#224]{noformat}
but it could be improved as: 
{noformat}
== Physical Plan ==
*(1) Project [value#219 AS c#222]
+- *(1) LocalTableScan [value#219]{noformat}
Bug: Redirect to the history page when accessed to /history without appliation id
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jul/20 21:35
Affects Version/s: 3.1.0

 

In the current master, when we access to /history on the HistoryServer with without application id, status code 400 will be returned.

I wonder it's better to redirect to the history page instead for the better UX.
Bug: Support ANSI dialect in MAKE_DATE and MAKE_TIMESTAMP 
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jul/20 11:05
Affects Version/s: 3.1.0

The MAKE_DATE and MAKE_TIMESTAMP functions should throw an exception for invalid parameters in the ANSI dialect (spark.sql.ansi.enabled is true) instead of NULLs.
Bug: Support pass 'spark.yarn.priority' through the SparkSubmit on Yarn modes
Status: Closed | Priority: Minor | Resolution: Won't Do
Created: 08/Jul/20 08:35
Affects Version/s: 3.1.0

SPARK-29603 support configure 'spark.yarn.priority' on Yarn deploy modes, and should we support pass this configuration through the SparkSubmit command line
Bug: Print column name when throws ClassCastException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jul/20 08:52
Affects Version/s: 2.4.3

We have a demo like this:
{code:java}
drop table if exists cast_exception_test;
create table cast_exception_test(c1 int, c2 string) partitioned by (dt string) stored as orc;
insert into table cast_exception_test partition(dt='2020-04-08') values('1', 'jeff_1');
{code}
Now we query this partition no problem both spark and hive, but if we change the type of partition's field, spark will throw exception, but hive will not:
{code:java}
alter table cast_exception_test change column c1 c1 string;
-- hive correct, but spark throws ClassCastException
select * from cast_exception_test where dt='2020-04-08';
{code}
exception like this:
{code:java}
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text    at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)    at org.apache.spark.sql.hive.HiveInspectors$$anonfun$unwrapperFor$23.apply(HiveInspectors.scala:547)    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$21$$anonfun$apply$15.apply(TableReader.scala:515)    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$21$$anonfun$apply$15.apply(TableReader.scala:515)    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:532)    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:522)    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:256)    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:838)    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:838)    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:326)    at org.apache.spark.rdd.RDD.iterator(RDD.scala:290)    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)    at org.apache.spark.scheduler.Task.run(Task.scala:121)    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)
{code}
Bug: spark-ml must support one hot encoded output labels for classification
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jul/20 04:24
Affects Version/s: 2.4.0

In any classification algorithm, for target labels that have no ordinal relationship, it is advised to one hot encode the target labels. Refer here:

[https://stackoverflow.com/questions/51384911/one-hot-encoding-of-output-labels/53291690#53291690]

[https://www.linkedin.com/pulse/why-using-one-hot-encoding-classifier-training-adwin-jahn/]

spark-ml is not supporting the one hot encoded target labels. When I try, i get the below error:

IllegalArgumentException: u'requirement failed: Column label_ohe must be of type numeric but was actually of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.'

So it will be nice if OHE is supported for target labels
Bug: Re-use GetTimestamp in ParseToDate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jul/20 10:32
Affects Version/s: 3.1.0

Replace the combination of expressions SecondsToTimestamp and UnixTimestamp by GetTimestamp in ParseToDate. This will allow to eliminate unnecessary parsing overhead in: string -> timestamp -> long (seconds) -> timestamp -> date. After the changes, the chain will be: string -> timestamp -> date.
Bug: .alias() not working after pivot if there is only one aggregation
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jul/20 05:26
Affects Version/s: 2.4.0

When there is only one aggregation used after pivoting, the renaming is not working with alias() function.  The same works when there is more than one aggregation. Refer this stack overflow discussion for more details  : [https://stackoverflow.com/questions/62750017/pyspark-strange-behavior-of-alias-function-when-used-in-agg-after-pivot]

It would be better to have a consistent behavior irrespective of the number of aggregations
Bug: Use createDirectory instead of mkdir
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jul/20 01:46
Affects Version/s: 3.1.0

Use Files.createDirectory() to create local directory instead of File.mkdir() in DiskBlockManager.
Many times, we will see such error log information like "Failed to create local dir in xxxxxx". But there is no clear information indicating why the directory creation failed.
When Files.createDirectory() fails to create a local directory, it can give specific error information for subsequent troubleshooting(also throws IOException).
Bug: Provide options for offset-based semantics when using structured streaming from a file stream source
Status: Open | Priority: Minor | Resolution: 
Created: 02/Jul/20 09:13
Affects Version/s: 3.1.0

Implement the following options while performing structured streaming from a file data source:
{code:java}
startingOffsetsByTimestamp
endingOffsetsByTimestamp
startingOffsets
endingOffsets
{code}
These options currently exist when using structured streaming from a Kafka data source. 

*Please see comments from the below PR for details.*
[https://github.com/apache/spark/pull/28841]

*Example from usage with Kafka data source*
 [http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries]
Bug: Repartition leads to out of memory
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jul/20 00:29
Affects Version/s: 2.4.4

 We  have an application that does aggregation on 7 columns. In order to avoid shuffles we thought of doing repartition on those 7 columns. It works well with 1 to 4tb of data. When it gets over 4Tb, it  fails with OOM or disk space.

 

Do we have a better  approach to reduce the shuffle ? For our biggest dataset, the spark job never ran with repartition.  We are out of options.

 

We do have a 24 node cluster with r5.24X machines and 1TB of disk.  Our shuffle partition is set to 6912.
Bug: Not accurate Log Info  in  BroadcastExchangeExec.scala
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Jul/20 12:11
Affects Version/s: 3.0.0

s"Cannot broadcast the table that is larger than 8GB: $\{dataSize >> 30} GB")  is not accurate info , because  8GB  is not accurate.
Bug: Add overload for slice that accepts Columns or Int
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 17/Jul/20 01:30
Affects Version/s: 3.1.0

Add an overload for org.apache.spark.sql.functions.slice with the following signature:


{code:scala}
def slice(x: Column, start: Any, length: Any): Column = ???
{code}
Bug: Disallow empty part col values in partition spec before static partition writing
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 31/Jul/20 10:55
Affects Version/s: 3.1.0

When writing to the current static partition, the partition field is empty, and an error will be reported when all tasks are completed.
We can prevent such behavior before submitting the task.

 
{code:java}
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for key d is null or empty;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
    at org.apache.spark.sql.hive.HiveExternalCatalog.getPartitionOption(HiveExternalCatalog.scala:1212)
    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getPartitionOption(ExternalCatalogWithListener.scala:240)
    at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:276)
{code}
Bug: Add code-gen for shuffled hash join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jul/20 03:20
Affects Version/s: 3.1.0

We added shuffled hash join codegen internally in our fork, and seeing obvious improvement in benchmark compared to current non-codegen code path. Creating this Jira to add this support. Shuffled hash join codegen is very similar to broadcast hash join codegen. So this is a simple change.
Bug: Add handling for unique key in non-codegen hash join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jul/20 03:16
Affects Version/s: 3.1.0

`HashRelation` has two separate code paths for unique key look up and non-unique key look up E.g. in its subclass `UnsafeHashedRelation`([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L144-L177]), unique key look up is more efficient as it does not have extra `Iterator[UnsafeRow].hasNext()/next()` overhead per row.

`BroadcastHashJoinExec` has handled unique key vs non-unique key separately in code-gen path ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala#L289-L321]). But the non-codegen path for broadcast hash join and shuffled hash join do not separate it yet, so adding the support here.
Bug: Coalesce bucketed tables for shuffled hash join if applicable
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Jul/20 22:55
Affects Version/s: 3.1.0

Based on a follow up comment in PR [#28123|https://github.com/apache/spark/pull/28123], where we can coalesce buckets for shuffled hash join as well. The note here is we only coalesce the buckets from shuffled hash join stream side (i.e. the side not building hash map), so we don't need to worry about OOM when coalescing multiple buckets in one task for building hash map.
Bug: [EXAMPLES] Make BinaryClassificationMetricsExample consistently print the metrics on driver's stdout
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jul/20 12:03
Affects Version/s: 2.4.6

Some RDDs in this example (e.g., precision, recall) call println without calling collect.
If the job is under local mode, it sends the data to the driver node and prints the metrics on the driver's stdout.
However if the job is under cluster mode, the job prints the metrics on the executor's stdout.
It seems inconsistent compared to the other metrics nothing to do with RDD (e.g., auPRC, auROC) since these metrics always output the result on the driver's stdout.
All of the metrics should output its result on the driver's stdout.
Bug: Provide an explanation error message when calling require
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jul/20 00:42
Affects Version/s: 3.0.0

When requirement checks inside `MLibUtils.fastSquaredDistance` are not met a non useful exception is thrown

```

Caused by: java.lang.IllegalArgumentException: requirement failedCaused by: java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:212) at org.apache.spark.mllib.util.MLUtils$.fastSquaredDistance(MLUtils.scala:508) at org.apache.spark.mllib.clustering.EuclideanDistanceMeasure$.fastSquaredDistance(DistanceMeasure.scala:232) at org.apache.spark.mllib.clustering.EuclideanDistanceMeasure.isCenterConverged(DistanceMeasure.scala:190) at org.apache.spark.mllib.clustering.KMeans$$anonfun$runAlgorithm$4.apply(KMeans.scala:336) at org.apache.spark.mllib.clustering.KMeans$$anonfun$runAlgorithm$4.apply(KMeans.scala:334) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at scala.collection.MapLike$MappedValues.foreach(MapLike.scala:245) at org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:334) at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:251) at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:233)

```

This is happening in the following lines. Please add more useful information to the error message when the requirements are not met  [https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala#L537-L538]

 

Example:

```

require(v2.size == n, s"Both vectors should have same length, found v1 is $n while v2 is ${v2.size}")
 require(norm1 >= 0.0 && norm2 >= 0.0, "Both norms should be greater or equal to 0.0, found norm1=${norm1}, norm2=${norm2}")

```
Bug: change file locations for use db and refresh table
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Jul/20 16:15
Affects Version/s: 3.1.0

docs/sql-ref-syntax-aux-refresh-table.md -> docs/sql-ref-syntax-aux-cache-refresh-table.md
docs/sql-ref-syntax-qry-select-usedb.md -> docs/sql-ref-syntax-ddl-usedb.md
Bug: Revive HashClusteredDistribution and apply to stream-stream join
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 07/Feb/22 07:04
Affects Version/s: 3.3.0

SPARK-35703 removed HashClusteredDistribution and replaced its usages with ClusteredDistribution.

While this works great for non stateful operators, we still need to have a separate requirement of distribution for stateful operator, because the requirement of ClusteredDistribution is too relaxed while the requirement of physical partitioning on stateful operator is quite strict.

In most cases, stateful operators must require child distribution as HashClusteredDistribution, with below major assumptions:
 # HashClusteredDistribution creates HashPartitioning and we will never ever change it for the future.
 # We will never ever change the implementation of {{partitionIdExpression}} in HashPartitioning for the future, so that Partitioner will behave consistently across Spark versions.
 # No partitioning except HashPartitioning can satisfy HashClusteredDistribution.

 

We should revive HashClusteredDistribution (with probably renaming specifically with stateful operator) and apply the distribution to the all stateful operators.

SPARK-35703 only touched stream-stream join, which means stream-stream join hasn't been broken in actual releases. Let's aim the partial revert of SPARK-35703 in this ticket, and have another ticket to deal with other stateful operators, which have been broken for their introduction (2.2+).
Bug: Upgrade Dependencies in spark-sql Java library
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 18/Feb/22 18:04
Affects Version/s: 3.2.1

There are numerous vulnerabilities in *spark-sql* Java library.

Can we pls upgrade the transitive dependencies to mitigate these vulnerabilities.

 

+*Maven Dependency*+
{quote}{{<dependency>}}
{{  <groupId>org.apache.spark</groupId>}}
{{  <artifactId>spark-sql_2.13</artifactId>}}
{{  <version>3.2.1</version>}}
{{</dependency>}}
{quote}
 

+*Vulnerabilities*+
{quote}{{Found 17 vulnerabilities (12 High, 4 Medium, 1 Low)}}
{{---------------------------------------------------------------}}
{{| SEVERITY  |  LIBRARY                      |  ID             |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  commons-compress-1.20.jar    |  CVE-2021-35515 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  commons-compress-1.20.jar    |  CVE-2021-35516 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  commons-compress-1.20.jar    |  CVE-2021-35517 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  commons-compress-1.20.jar    |  CVE-2021-36090 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  gson-2.8.6.jar               |  WS-2021-0419   |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2019-17571 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2020-9493  |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2021-4104  |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23302 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23305 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  log4j-1.2.17.jar             |  CVE-2022-23307 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| HIGH      |  netty-all-4.1.68.Final.jar   |  WS-2020-0408   |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| MEDIUM    |  jackson-core-2.12.3.jar      |  WS-2021-0616   |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| MEDIUM    |  jackson-databind-2.12.3.jar  |  WS-2021-0616   |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| MEDIUM    |  netty-all-4.1.68.Final.jar   |  CVE-2021-43797 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| MEDIUM    |  protobuf-java-2.5.0.jar      |  CVE-2021-22569 |}}
{{|---------- | ----------------------------- | ----------------|}}
{{| LOW       |  log4j-1.2.17.jar             |  CVE-2020-9488  |}}
{{---------------------------------------------------------------}}
{quote}