av9ash/GitBugs huggingface dataset:
Bug: Validating sha512 files for releases does not comply with Apache Project documentation
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 30/Jan/20 15:27
Affects Version/s: 2.3.0

Spark release artifacts currently use gpg generated sha512 files.  These are not in the same format as ones generated via sha512sum, and can not be validated easily using sha512sum.  As the [Apache Project instructions|https://www.apache.org/info/verification.html] for how to validate a release explicitly instructs the use of sha512sum compatible tools, there is no documentation for successfully validating a downloaded release using the sha512 file.

 

This subject has been [brought up before|https://www.mail-archive.com/dev@spark.apache.org/msg20140.html] and an issue to change to using sha512sum compatible formats was closed as WontFix with little discussion; even if switching isn't warranted, at the least instructions for validating using the sha512 files should be documented.
Bug: Executor can't use spark.executorEnv.HADOOP_USER_NAME to change the user accessing to hdfs
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Jan/20 10:31
Affects Version/s: 2.4.3

Currently, we can specify hadoop user by setting HADOOP_USER_NAME on driver when submit a job. However it's invalid to executor by setting spark.executorEnv.HADOOP_USER_NAME.
Bug: Add SPARK_APPLICATION_ID into driver pod env
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jan/20 19:32
Affects Version/s: 3.0.0

This should be a minor improvement.

The use case is we want to look up environment variables and create application folder and redirect driver logs to application folder.  Executors has it and we want to make a change to driver as well. 

 
{code:java}
Limits:
 cpu: 1024m
 memory: 896Mi
 Requests:
 cpu: 1
 memory: 896Mi
Environment:
 SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
 SPARK_LOCAL_DIRS: /var/data/spark-9c315655-aba4-47fb-821c-30268d02af7e
 SPARK_CONF_DIR: /opt/spark/conf{code}
 

[https://github.com/apache/spark/blob/afe70b3b5321439318a456c7d19b7074171a286a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala#L73-L79]

We need SPARK_APPLICATION_ID inside the pod to organize logs
Bug: allow driver to consume a fractional core
Status: Open | Priority: Minor | Resolution: 
Created: 18/Jan/20 12:20
Affects Version/s: 3.1.0

see https://stackoverflow.com/questions/56781927/apache-spark-standalone-scheduler-why-does-driver-need-a-whole-core-in-cluste

this is to make it possible for a driver to use 0.2 cores rather than a whole core

Standard CPUs, no GPUs
Bug: spark worker graceful shutdown
Status: Open | Priority: Minor | Resolution: 
Created: 22/Jan/20 22:45
Affects Version/s: 3.1.0

I am not talking about spark streaming! just regular batch jobs using spark-submit that may try to read large csv (100+gb) then write it out as parquet. In an autoscaling cluster would be nice to be able to scale down (ie terminate) ec2s without slowing down active spark applications.

for example:
1. start spark cluster with 8 ec2s
2. submit 6 spark apps
3. 1 spark app completes, so 5 apps still running
4. cluster can scale down 1 ec2 (to save $) but don't want to make the existing apps running on the (soon to be terminated) ec2 have to make its csv read, RDD processing steps.etc start from the beginning on different ec2's executors. Instead want to have a 'graceful shutdown' command so that the 8th ec2 does not accept new spark-submit apps to it (ie don't start new executors on it) but finish the ones that have already launched on it, then exit the worker pid. then the ec2 can be terminated


I thought stop-slave.sh could do this but looks like it just kills the pid
Bug: Add more metrics to the all stages page
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Jan/20 15:09
Affects Version/s: 3.1.0

The web UI page for individual stages has many useful metrics to diagnose poorly performing stages, e.g. spilled bytes or GC time. Identifying those stages among hundreds or thousands of stages is cumbersome, as you have to click through all stages on the all stages page. The all stages page should host more metrics from the individual stages page like
 - Peak Execution Memory
 - Spill (Memory)
 - Spill (Disk)
 - GC Time

These additional metrics would make the page more complex, so showing them should be optional. The individual stages page hides some metrics under !image-2020-01-28-16-12-49-807.png! . Those new metrics on the all stages page should also be made optional in the same way.

!image-2020-01-28-16-13-36-174.png!

Existing metrics like
 - Input
 - Output
 - Shuffle Read
 - Shuffle Write

could be made optional as well and active by default. Then users can remove them if they want but get the same view as now by default.

The table extends as additional metrics get checked / unchecked:
!image-2020-01-28-16-15-20-258.png!

Sorting the table by metrics allows to find the stages with highest GC time or spilled bytes.
Bug: Detect equijoins better
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Jan/20 15:34
Affects Version/s: 3.1.0

The following 2 query produce different plans, as the second one is not recognised as equijoin.

{noformat}
SELECT * FROM t1 FULL OUTER JOIN t2 ON t1.c2 = 2 AND t2.c2 = 2 AND t1.c = t2.c

SortMergeJoin [c#225], [c#236], FullOuter, ((c2#226 = 2) AND (c2#237 = 2))
:- *(2) Sort [c#225 ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(c#225, 5), true, [id=#101]
:     +- *(1) Project [_1#220 AS c#225, _2#221 AS c2#226]
:        +- *(1) LocalTableScan [_1#220, _2#221]
+- *(4) Sort [c#236 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(c#236, 5), true, [id=#106]
      +- *(3) Project [_1#231 AS c#236, _2#232 AS c2#237]
         +- *(3) LocalTableScan [_1#231, _2#232]
{noformat}

{noformat}
SELECT * FROM t1 FULL OUTER JOIN t2 ON t1.c2 = 2 AND t2.c2 = 2

BroadcastNestedLoopJoin BuildRight, FullOuter, ((c2#226 = 2) AND (c2#237 = 2))
:- *(1) Project [_1#220 AS c#225, _2#221 AS c2#226]
:  +- *(1) LocalTableScan [_1#220, _2#221]
+- BroadcastExchange IdentityBroadcastMode, [id=#146]
   +- *(2) Project [_1#231 AS c#236, _2#232 AS c2#237]
      +- *(2) LocalTableScan [_1#231, _2#232]
{noformat}

We could detect the implicit equalities from the join condition.
Bug: Regression in the wide schema benchmark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jan/20 17:17
Affects Version/s: 3.0.0

New results of WideSchemaBenchmark generated in the PR: https://github.com/apache/spark/pull/27078 show regressions up to 2 times.
Before:
{code}
2500 select expressions                        103 /  107          0.0   102962705.0       0.1X
{code}
https://github.com/apache/spark/pull/27078/files#diff-8d27bbf2f73a68bf0c2025f0702f7332L11
After:
{code}
2500 select expressions                             211            214           4          0.0   210927791.0       0.0X
{code}
https://github.com/apache/spark/pull/27078/files#diff-8d27bbf2f73a68bf0c2025f0702f7332R11
Bug: DataFrameStatFunctions.approxQuantile doesn't work with TABLE.COLUMN syntax
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jan/20 19:24
Affects Version/s: 2.4.4

The DataFrameStatFunctions.approxQuantile doesn't work with fully qualified column name (i.e TABLE_NAME.COLUMN_NAME) which is often the way you refer to the column when working with joined dataframes having ambiguous column names.


See code below for example.
{code:java}

import scala.util.Random
val l = (0 to 1000).map(_ => Random.nextGaussian() * 1000)
val df1 = sc.parallelize(l).toDF("num").as("tt1")
val df2 = sc.parallelize(l).toDF("num").as("tt2")
val dfx = df2.crossJoin(df1)

dfx.stat.approxQuantile("tt1.num", Array(0.1), 0.0)
// throws: java.lang.IllegalArgumentException: Field "tt1.num" does not exist.
Available fields: num

dfx.stat.approxQuantile("num", Array(0.1), 0.0)
// throws: org.apache.spark.sql.AnalysisException: Reference 'num' is ambiguous, could be: tt2.num, tt1.num.;{code}
Bug: Cached blockInfo in BlockMatrix.scala is never released
Status: Open | Priority: Minor | Resolution: 
Created: 17/Jan/20 08:51
Affects Version/s: 3.1.0

The private variable _blockInfo_ in mllib.linalg.distribtued.BlockMatrix is never unpersisted since a BlockMatrix instance is created.

{code:scala}
 private lazy val blockInfo = blocks.mapValues(block => (block.numRows, block.numCols)).cache()
{code}

I think we should add an API to unpersist this variable.
Bug: Writing data to parquet with dynamic partitionOverwriteMode should not do the folder rename in commitjob stage
Status: Open | Priority: Minor | Resolution: 
Created: 09/Jan/20 18:15
Affects Version/s: 3.1.0

In the current spark implementation if you set,
{code:java}
spark.sql.sources.partitionOverwriteMode=dynamic
{code}
even with 
{code:java}
mapreduce.fileoutputcommitter.algorithm.version=2
{code}
it would still rename the partition folder *sequentially* in commitJob stage as shown here: 

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L188]

[https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L184]

 

This is very slow in cloud storage. We should commit the data similar to FileOutputCommitter v2?
Bug: Add `escapeChar` parameter to the `like` function
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jan/20 19:13
Affects Version/s: 3.0.0

SPARK-28083 supported LIKE ... ESCAPE syntax
{code:sql}
spark-sql> SELECT '_Apache Spark_' like '__%Spark__' escape '_';
true
{code}
but the `like` function can accept only 2 parameters. If we pass the third one, it fails with:
{code:sql}
spark-sql> SELECT like('_Apache Spark_', '__%Spark__', '_');
Error in query: Invalid number of arguments for function like. Expected: 2; Found: 3; line 1 pos 7
{code}
The ticket aims to support the third parameter in `like` as `escapeChar`.
Bug: Show the descripton of metrics for WholeStageCodegen in DAG viz
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/20 16:28
Affects Version/s: 3.0.0

In the DAG viz for SQL, A WholeStageCodegen-node shows some metrics like `33 ms (1 ms, 1 ms, 26 ms (stage 16 (attempt 0): task 172))` but users can't understand what they mean because there are no description about those metrics.
Bug: Support ANSI INSERT syntax
Status: Open | Priority: Minor | Resolution: 
Created: 30/Jan/20 16:30
Affects Version/s: 3.1.0

Related to the [ANSI SQL specification for insert syntax](https://en.wikipedia.org/wiki/Insert_(SQL)), could the parsing and underlying engine support the syntax of:
{{INSERT INTO <table> (<column list>) select <columns list) from <otherTable>}}

I think I read somewhere that there's some underlying technical detail where the columns inserted into SPARK tables must have the selected columns match the order of the table definition.  But, if this is the case, isn't' there a place in the parser-layer and execution-layer where the parser can translate something like:


{{insert into someTable (col1,col2)
select someCol1, someCol2 from otherTable}}


Where someTable has 3 columns (col3,col2,col1) (note the order here), the query is rewritten and sent to the engine as:

{{insert into someTable
select null, someCol2, someCol1 from otherTable}}

Note, the reordering and adding of the null column was done based on some table metadata on someTable so it knew which columns from the INSERT() map over to the columns from the select.

Is this possible? The lack of specifying column values is preventing our project from SPARK being a supported platform.
Bug: Eliminate warnings from deprecated constructors of java.lang.Integer and java.lang.Double
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/20 08:15
Affects Version/s: 3.0.0

The constructors of java.lang.Integer and java.lang.Double has been deprecated already, see [https://docs.oracle.com/javase/9/docs/api/java/lang/Integer.html]. The following warnings are printed while compiling Spark:
{code}
1. RDD.scala:240: constructor Integer in class Integer is deprecated: see corresponding Javadoc for more information.
2. MutableProjectionSuite.scala:63: constructor Integer in class Integer is deprecated: see corresponding Javadoc for more information.
3. UDFSuite.scala:446: constructor Integer in class Integer is deprecated: see corresponding Javadoc for more information.
4. UDFSuite.scala:451: constructor Double in class Double is deprecated: see corresponding Javadoc for more information.
5. HiveUserDefinedTypeSuite.scala:71: constructor Double in class Double is deprecated: see corresponding Javadoc for more information.
{code}
 The ticket aims to replace the constructors by the valueOf methods, or maybe by other ways.
Bug: Test cases in HiveShowCreateTableSuite should create Hive table instead of Datasource table
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/20 06:08
Affects Version/s: 3.0.0

Because SparkSQL now creates data source table if no provider is specified in SQL command, some test cases in HiveShowCreateTableSuite don't create Hive table, but data source table.

It is confusing and not good for the purpose of this test suite.
Bug: Eliminate warnings from deprecated BisectingKMeansModel.computeCost
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/20 11:15
Affects Version/s: 3.0.0

The computeCost() method of the BisectingKMeansModel class has been deprecated. That causes warnings while compiling BisectingKMeansSuite:
{code}
[warn] /Users/maxim/proj/kmeans-computeCost-warning/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala:108: method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.
[warn]     assert(model.computeCost(dataset) < 0.1)
[warn]                  ^
[warn] /Users/maxim/proj/kmeans-computeCost-warning/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala:135: method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.
[warn]     assert(model.computeCost(dataset) == summary.trainingCost)
[warn]                  ^
[warn] /Users/maxim/proj/kmeans-computeCost-warning/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala:323: method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.
[warn]       model.computeCost(dataset)
[warn]             ^
[warn] three warnings found
{code}
Bug: Spark UI is not showing Aggregated Metrics by Executor in stage page
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/20 09:32
Affects Version/s: 3.0.0

There are scenarios where Spark History Server is located behind the VPC. So whenever api calls hit to get the executor Summary(allexecutors). There can be delay in getting the response of executor summary and in mean time "stage-page-template.html" is loaded and the response of executor Summary is not added to the stage-page-template.html.

As the result of which Aggregated Metrics by Executor in stage page is showing blank.

This scenario can be easily found in the cases when there is some proxy-server which is responsible for sending the request and response to spark History server.
 This can be reproduced in Knox/In-house proxy servers which are used to send and receive response to Spark History Server.

Alternative scenario to test this case, Open the spark UI in developer mode in browser add some breakpoint in stagepage.js, this will add some delay in getting the response and now if we check the spark UI for stage Aggregated Metrics by Executor in stage page is showing blank.
Bug: Deprecate numTrees in GBT at 2.4.5 and remove it at 3.0.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jan/20 05:52
Affects Version/s: 2.4.5

Currently, GBT has
{code:java}
/**
 * Number of trees in ensemble
 */
@Since("2.0.0")
val getNumTrees: Int = trees.length{code}
and
{code:java}
/** Number of trees in ensemble */
val numTrees: Int = trees.length{code}
I will deprecate numTrees in 2.4.5 and remove it in 3.0.0
Bug: Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 23/Jan/20 09:25
Affects Version/s: 3.0.0

# We have implemented a complex ExternalCatalog which is used for retrieving multi isomerism database's metadata(sush as elasticsearch、postgresql), so that we can make a mixture query between hive and our online data.
 # But as spark require that value of spark.sql.catalogImplementation must be one of in-memory/hive, we have to modify SparkSession and rebuild spark to make our project work.
 # Finally, we hope spark removing above restriction, so that it's will be much easier to let us keep pace with new spark version. Thanks!
Bug: Update scalafmt to 1.0.3 with onlyChangedFiles feature
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jan/20 18:03
Affects Version/s: 3.0.0

[https://github.com/SimonJPegg/mvn_scalafmt/releases/tag/v1.0.3]

added an option onlyChangedFiles which was one of the things holding back the upgrade in SPARK-29293
Bug: RandomForest add Param bootstrap to control sampling method
Status: Resolved | Priority: Minor | Resolution: Resolved
Created: 17/Jan/20 07:47
Affects Version/s: 3.0.0

Current RF with numTrees=1 will directly build a tree using the orignial dataset,

while with numTrees>1 it will use bootstrap samples to build trees.

This design is to train a DecisionTreeModel by the impl of RandomForest, however, it is somewhat strange.

In Scikit-Learn, there is a param bootstrap to control bootstrap samples are used.
Bug: Duplicate query plan on Spark UI SQL page
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jan/20 16:39
Affects Version/s: 2.4.4

When you save a Spark UI SQL query page to disk and then display the html file with your browser, the query plan will be rendered a second time. This change avoids rendering the plan visualization when it exists already.

 

!https://user-images.githubusercontent.com/44700269/72543429-fcb8d980-3885-11ea-82aa-c0b3638847e5.png!

The fix does not call {{renderPlanViz()}} when the plan exists already:

!https://user-images.githubusercontent.com/44700269/72543641-57523580-3886-11ea-8cdf-5fb0cdffa983.png!
Bug: Update testthat dependency
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 22/Jan/20 23:14
Affects Version/s: 3.0.0

Currently SparkR fix {{testhat}} version to 1.0.2.

 

This has been introduced by SPARK-22817 to address removal of {{testthat}}:::run_tests in {{testthat}} 2.0.0.

 

As of 2.0.0 the replacement of {{run_tests}} is {{test_package_dir}}.  Same as {{run_tests}} it is considered internal, but it will allow us to upgrade to the latest {{testhat}} release.
Bug: bytesToString util adapts to negative number
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 09/Jan/20 02:38
Affects Version/s: 3.0.0

when the input of bytesToString is negative number, the string returned  is unfriendly.

For Example:

-5368709120 will return -5368709120B, not -5.0 GiB
Bug: Add sub-class of AppenderSkeleton reusable in tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jan/20 08:49
Affects Version/s: 2.4.4

Some tests define similar sub-class of AppenderSkeleton. The code duplication can be eliminated by defining common class in [SparkFunSuite.scala|https://github.com/apache/spark/compare/master...MaxGekk:dedup-appender-skeleton?expand=1#diff-d521001af1af1a2aace870feb25ae0b0]
Bug: Regression in interval string parsing
Status: Open | Priority: Minor | Resolution: 
Created: 18/Jan/20 16:59
Affects Version/s: 3.0.0

Previously:
  11 units w/o interval - 1972.8 ns per row
Regenerated results in the PR https://github.com/apache/spark/pull/27078:
  11 units w/o interval - 3272.6 ns per row

Regression is by 66%, see https://github.com/apache/spark/pull/27078/files#diff-586487fac2b9b1303aaf80adf8fa37abR28
Bug: Add classes to represent Java Regressors and RegressionModels
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jan/20 19:49
Affects Version/s: 3.0.0

Right now PySpark provides classed representing Java {{Classifiers}} and {{ClassifierModels}}, but lacks their regression counterparts.

We should provide these for consistency, feature parity and as prerequisite for SPARK-29212.
Bug: Refactor SimplifyBinaryComparison to reduce the time complexity
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/20 06:25
Affects Version/s: 3.0.0

The improvement of the rule SimplifyBinaryComparison in PR https://github.com/apache/spark/pull/27008 could bring performance regression in the optimizer. 
We need to improve the implementation and reduce the time complexity.
Bug: OneVsRest and OneVsRestModel _from_java and _to_java should handle weightCol
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/20 12:35
Affects Version/s: 3.0.0

Current behaviour

{code:python}
from pyspark.ml.classification import LogisticRegression, OneVsRest, OneVsRestModel
from pyspark.ml.linalg import DenseVector

df = spark.createDataFrame([(0, 1, DenseVector([1.0, 0.0])), (0, 1, DenseVector([1.0, 0.0]))], ("label", "w", "features"))

ovr = OneVsRest(classifier=LogisticRegression()).setWeightCol("w")
ovrm = ovr.fit(df)
ovr.getWeightCol()
## 'w'
ovrm.getWeightCol()
## 'w'

ovr.write().overwrite().save("/tmp/ovr")
ovr_ = OneVsRest.load("/tmp/ovr")
ovr_.getWeightCol()
## KeyError   
## ...
## KeyError: Param(parent='OneVsRest_5145d56b6bd1', name='weightCol', doc='weight column name. ...)

ovrm.write().overwrite().save("/tmp/ovrm")
ovrm_ = OneVsRestModel.load("/tmp/ovrm")
ovrm_ .getWeightCol()
## KeyError   
## ...
## KeyError: Param(parent='OneVsRestModel_598c6d900fad', name='weightCol', doc='weight column name ...
{code}

Expected behaviour:

{{OneVsRest}} and {{OneVsRestModel}} loaded from disk should have {{weightCol}}.
Bug: Deprecation log warning is not printed in Avro schema inferring
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jan/20 10:15
Affects Version/s: 3.0.0

The bug can be reproduced by the test:
{code}
  test("log a warning of ignoreExtension deprecation") {
    val logAppender = new LogAppender
    withTempPath { dir =>
      Seq(("a", 1, 2), ("b", 1, 2), ("c", 2, 1), ("d", 2, 1))
        .toDF("value", "p1", "p2")
        .repartition(2)
        .write
        .format("avro")
        .option("header", true)
        .save(dir.getCanonicalPath)
      withLogAppender(logAppender) {
        spark
          .read
          .format("avro")
          .option(AvroOptions.ignoreExtensionKey, false)
          .option("header", true)
          .load(dir.getCanonicalPath)
          .count()
      }
      val deprecatedEvents = logAppender.loggingEvents
        .filter(_.getRenderedMessage.contains(
          s"Option ${AvroOptions.ignoreExtensionKey} is deprecated"))
      assert(deprecatedEvents.size === 1)
    }
  }
{code}