av9ash/GitBugs huggingface dataset:
Bug: flatten the result dataframe of FValueTest
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/20 03:45
Affects Version/s: 3.1.0

add a new method
{code:java}
@Since("3.1.0")
def test(
    dataset: DataFrame,
    featuresCol: String,
    labelCol: String,
    flatten: Boolean): DataFrame {code}
 

Similar to new {{test}} method in {{ChiSquareTest}}, it will:

1, support df operation on the returned df;

2, make driver no longer a bottleneck when dim is high
Bug: Add a readiness probe to spark pod definitions
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 22/Apr/20 11:50
Affects Version/s: 2.4.5

Add a readiness probe to allow so Kubernetes can communicate basic spark pod state to the end user via get/describe pod commands.

A basic TCP/SYN probe of the RPC port should be enough to indicate that a spark process is running.
Bug: gapply and dapply docs should be more aligned, possibly combined
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 27/Apr/20 05:51
Affects Version/s: 2.4.5

This is a follow-up to https://issues.apache.org/jira/browse/SPARK-31568

There, we combined gapply and gapplyCollect to make it easier to sync arguments between those Rds.

dapply and dapplyCollect are also sufficiently similar that they could be on the same Rd.
Bug: Replace floorDiv by / in localRebaseGregorianToJulianDays()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Apr/20 15:42
Affects Version/s: 3.1.0

Most likely utcCal.getTimeInMillis % MILLIS_PER_DAY == 0 but need to check that for all available time zones in the range of [0001, 2100] years with the step of 1 hour or maybe smaller. If this hypothesis is confirmed, floorDiv can be replaced by /, and this should improve performance of RebaseDateTime.localRebaseGregorianToJulianDays.
Bug: DATE_PART functions produces incorrect result
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 14/Apr/20 22:53
Affects Version/s: 3.0.0

Spark does not extract correct date part from calendar interval. Below is one example for extracting day from calendar interval
{code:java}
spark.sql("SELECT EXTRACT(DAY FROM (cast('2020-01-15 00:00:00' as timestamp) - cast('2020-01-01 00:00:00' as timestamp)))").show{code}
+------------------------------------------------------------------------------------------------------------------------+
|date_part('DAY', subtracttimestamps(CAST('2020-01-15 00:00:00' AS TIMESTAMP), CAST('2020-01-01 00:00:00' AS TIMESTAMP)))|
+------------------------------------------------------------------------------------------------------------------------+
| 0|
+------------------------------------------------------------------------------------------------------------------------+

Actual output 0 days
Correct output 14 days

This is because SubtractTimestamps expression calculates difference and populates only microseconds field. months and days field are set to zero
{code:java}
new CalendarInterval(months=0, days=0, microseconds=end.asInstanceOf[Long] - start.asInstanceOf[Long]){code}

https://github.com/apache/spark/blob/2c5d489679ba3814973680d65853877664bcd931/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L2211

But ExtractIntervalDays expression retrieves days information from days field in CalendarInterval and returns zero.
{code:java}
def getDays(interval: CalendarInterval): Int = {
 interval.days
 }{code}
https://github.com/apache/spark/blob/2c5d489679ba3814973680d65853877664bcd931/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala#L73
Bug: NullPointerException when parsing event log with InMemoryStore
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Apr/20 00:16
Affects Version/s: 3.0.1

I compiled with the current branch-3.0 source and tested it in mac os. A java.lang.NullPointerException will be thrown when below conditions are met: 
 # Using InMemoryStore as kvstore when parsing the event log file (e.g., when spark.history.store.path is unset). 
 # At least one stage in this event log has task number greater than spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to delete extra task records.
 # The job has more than one stage, so parentToChildrenMap in InMemoryStore.java will have more than one key.

The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the method deleteParentIndex().
{code:java}
    private void deleteParentIndex(Object key) {
      if (hasNaturalParentIndex) {
        for (NaturalKeys v : parentToChildrenMap.values()) {
          if (v.remove(asKey(key))) {
            // `v` can be empty after removing the natural key and we can remove it from
            // `parentToChildrenMap`. However, `parentToChildrenMap` is a ConcurrentMap and such
            // checking and deleting can be slow.
            // This method is to delete one object with certain key, let's make it simple here.
            break;
          }
        }
      }
    }{code}
In “if (v.remove(asKey(key)))”, if the key is not contained in v,  "v.remove(asKey(key))" will return null, and java will throw a NullPointerException when executing "if (null)".

An exception stack trace is attached.

This issue can be fixed by updating if statement to
{code:java}
if (v.remove(asKey(key)) != null){code}
Bug: spark.submit.waitForCompletion flag to control spark-submit exit in Standalone Cluster Mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Apr/20 15:19
Affects Version/s: 3.1.0

Implementation of spark.submit.waitForCompletion flag which will control if the spark-submit process will wait to exit until the application finishes/fails/errors out. This flag is set to false by default where the application will exit after submission.
Bug: Add JDK11 example in `bin/docker-image-tool.sh` usage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/20 19:44
Affects Version/s: 3.0.0

{code:java}
$ bin/docker-image-tool.sh -h
...
  - Build and push JDK11-based image with tag "v3.0.0" to docker.io/myrepo
    bin/docker-image-tool.sh -r docker.io/myrepo -t v3.0.0 -b java_image_tag=11-jre-slim build
    bin/docker-image-tool.sh -r docker.io/myrepo -t v3.0.0 push {code}
 
{code:java}
$ docker run -it --rm docker.io/myrepo/spark:v3.0.0 java --version | tail -n3
openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment 18.9 (build 11.0.6+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.6+10, mixed mode) {code}
Bug: Sprak 3.0 - Full List of Pod Template not available
Status: Open | Priority: Minor | Resolution: 
Created: 18/Apr/20 20:41
Affects Version/s: 3.0.0

A full list of Pod Template Values not available and link is not working

URL - [https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html#pod-template-properties]

Section - Pod Template

Hyper Link Not Working - full_list as shown below

Link - For details, see the [full list|https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html#pod-template-properties] of pod template values that will be overwritten by spark.
Bug: grouping_id calculation should be improved
Status: Open | Priority: Minor | Resolution: 
Created: 27/Apr/20 22:27
Affects Version/s: 3.1.0

Unrelated to SPARK-21858 which identifies that grouping_id is determined by exclusion from a grouping_set rather than inclusion, when performing complex grouping_sets that are not in the order of the base select statement, flipping the bit in the grouping_id seems to be happen when the grouping set is identified rather than when the columns are selected in the sql.   I will of course use the exclusion strategy identified in SPARK-21858 as the baseline for this.  

 
{code:scala}
import spark.implicits._
val df= Seq(
 ("a","b","c","d"),
 ("a","b","c","d"),
 ("a","b","c","d"),
 ("a","b","c","d")
).toDF("a","b","c","d").createOrReplaceTempView("abc")
{code}
expected to have these references in the grouping_id:
 d=1
 c=2
 b=4
 a=8
{code:scala}
spark.sql("""
 select a,b,c,d,count(*), grouping_id() as gid, bin(grouping_id()) as gid_bin
 from abc
 group by GROUPING SETS (
 (),
 (a,b,d),
 (a,c),
 (a,d)
 )
 """).show(false)
{code}
This returns:
{noformat}
+----+----+----+----+--------+---+-------+
|a   |b   |c   |d   |count(1)|gid|gid_bin|
+----+----+----+----+--------+---+-------+
|a   |null|c   |null|4       |6  |110    |
|null|null|null|null|4       |15 |1111   |
|a   |null|null|d   |4       |5  |101    |
|a   |b   |null|d   |4       |1  |1      |
+----+----+----+----+--------+---+-------+
{noformat}
 

 In other words, I would have expected the excluded values one way but I received them excluded in the order they were first seen in the specified grouping sets.

 a,b,d included = excldes c = 2; expected gid=2. received gid=1
 a,d included = excludes b=4, c=2 expected gid=6, received gid=5

The grouping_id that actually is expected is (a,b,d,c) 

{code:scala}
spark.sql("""
 select a,b,c,d,count(*), grouping_id(a,b,d,c) as gid, bin(grouping_id(a,b,d,c)) as gid_bin
 from abc
 group by GROUPING SETS (
 (),
 (a,b,d),
 (a,c),
 (a,d)
 )
 """).show(false)
{code}


 columns forming groupingid seem to be creatred as the grouping sets are identified rather than ordinal position in parent query.

I'd like to at least point out that grouping_id is documented in many other rdbms and I believe the spark project should use a policy of flipping the bits so 1=inclusion; 0=exclusion in the grouping set.

However many rdms that do have the feature of a grouping_id do implement it by the ordinal position recognized as fields in the select clause, rather than allocating them as they are observed in the grouping sets.
Bug: Pretty string output for sql method of RuntimeReplaceable expressions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Apr/20 07:24
Affects Version/s: 3.1.0

The RuntimeReplaceable ones are runtime replaceable, thus, their original parameters are not going to be resolved to PrettyAttribute and remain debug style string if we directly implement their `sql` methods with their parameters' `sql` methods.
e.g. `sql` method from `to_timestamp`
Bug: Executors Stop() method is not executed when they are killed
Status: In Progress | Priority: Minor | Resolution: 
Created: 18/Apr/20 17:23
Affects Version/s: 3.1.0

In dynamic Allocation when executors are killed, stop() method of executors is never called. So executors never goes down properly.

In SPARK-29152, shutdown hook was added to stop the executors properly.

Instead of forcing a shutdown hook we should ask executors to stop themselves before killing them.
Bug: Make SPARK_JARS_DIR configurable
Status: In Progress | Priority: Minor | Resolution: 
Created: 13/Apr/20 07:22
Affects Version/s: 3.1.0

In the script under bin/sbin, it is better that we can specify SPARK_JARS_DIR as same as SPARK_CONF_DIR.

Our usecase:
 We are trying to employ spark 2.4.5 with YARN in HDP2.6.4. Since there is an incompatible conflict on commons-lang3 between spark 2.4.5 and HDP2.6.4, we tweak the jars by Maven Shade Plugin.
 The jars slightly differ from jars in spark 2.4.5, and we locate it in a directory different from the default. So it is useful for us if we can set SPARK_JARS_DIR for bin/sbin scripts to point the direcotry.
 We can do that without the modification by deploying spark home as many as set of jars, but it is somehow redundant.

Common usecase:
 I believe there is a similer usecase. For example, deploying spark built for scala 2.11 and 2.12 in a machine and switch jars location by setting SPARK_JARS_DIR.
Bug: Replace expression TimeSub(l, r) with TimeAdd(l -r)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Apr/20 02:16
Affects Version/s: 3.1.0

The implementation of TimeSub for the operation of timestamp subtracting interval is almost repetitive with TimeAdd. We can replace it with TimeAdd(l, -r) since there are equivalent. 

Suggestion from https://github.com/apache/spark/pull/28310#discussion_r414259239
Bug: namePrefix could be null in Utils.createDirectory
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 28/Apr/20 06:07
Affects Version/s: 3.0.0

In our production, we find that many shuffle files could be located in
/hadoop/2/yarn/local/usercache/b_carmel/appcache/application_1586487864336_4602/*null*-107d4e9c-d3c7-419e-9743-a21dc4eaeb3f/3a

The Util.createDirectory() uses a default parameter "spark"
{code}
  def createDirectory(root: String, namePrefix: String = "spark"): File = {
{code}
But in some cases, the actual namePrefix is null. If the method is called with null, then the default value would not be applied.
Bug: Changed PipelineModel(...) to self.cls(...) in pyspark.ml.pipeline.PipelineModelReader.load()
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 03/Apr/20 10:52
Affects Version/s: 2.4.5

PR: [https://github.com/apache/spark/pull/28110]
 * What changes were proposed in this pull request?
 pypsark.ml.pipeline.py line 245: Change PipelineModel(...) to self.cls(...)
 * Why are the changes needed?
 This change fixes the loading of class (which inherits from PipelineModel class) from file.
 E.g. Current issue:

{code:java}
CustomPipelineModel(PipelineModel):
    def _transform(self, df):
        ...
 CustomPipelineModel.save('path/to/file') # works
 CustomPipelineModel.load('path/to/file') # wrong: results in PipelineModel() instead of CustomPipelineModel()
 CustomPipelineModel.transform() # wrong: results in calling PipelineModel.transform() instead of CustomPipelineModel.transform(){code}
 * Does this introduce any user-facing change?
 No.
Bug: Unify the font color of label among all DAG-viz.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/20 22:14
Affects Version/s: 3.0.0

There are three types of DAG-viz in the WebUI.
One is for stages, another one is for RDDs and the last one is for query plans.
But the font color of labels are slightly different among them.
Bug: The catalogString doesn't distinguish Vectors in ml and mllib
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/20 18:48
Affects Version/s: 2.4.5

h2. Bug Description

The `catalogString` is not detailed enough to distinguish the pyspark.ml.linalg.Vectors and pyspark.mllib.linalg.Vectors.
h2. How to reproduce the bug

[Here|https://spark.apache.org/docs/latest/ml-features#minmaxscaler] is an example from the official document (Python code). If I keep all other lines untouched, and only modify the Vectors import line, which means:
{code:java}
# from pyspark.ml.linalg import Vectors
from pyspark.mllib.linalg import Vectors
{code}
Or you can directly execute the following code snippet:
{code:java}
from pyspark.ml.feature import MinMaxScaler
# from pyspark.ml.linalg import Vectors
from pyspark.mllib.linalg import Vectors
dataFrame = spark.createDataFrame([
    (0, Vectors.dense([1.0, 0.1, -1.0]),),
    (1, Vectors.dense([2.0, 1.1, 1.0]),),
    (2, Vectors.dense([3.0, 10.1, 3.0]),)
], ["id", "features"])
scaler = MinMaxScaler(inputCol="features", outputCol="scaledFeatures")
scalerModel = scaler.fit(dataFrame)
{code}
It will raise an error:
{code:java}
IllegalArgumentException: 'requirement failed: Column features must be of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>> but was actually struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.'
{code}
However, the actually struct and the desired struct are exactly the same string, which cannot provide useful information to the programmer. I would suggest making the catalogString distinguish pyspark.ml.linalg.Vectors and pyspark.mllib.linalg.Vectors.

Thanks!
Bug: Non-existed metric hiveClientCalls.count of CodeGenerator in Monitoring Doc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Apr/20 08:28
Affects Version/s: 3.0.0

There is a duplicated `hiveClientCalls.count` metric in both `namespace=HiveExternalCatalog` and  `namespace=CodeGenerator` bullet lists of [Spark Monitoring doc|https://spark.apache.org/docs/3.0.0-preview2/monitoring.html#component-instance--executor], but there is only one inside object HiveCatalogMetrics in [source code|https://github.com/apache/spark/blob/6550d0d5283efdbbd838f3aeaf0476c7f52a0fb1/core/src/main/scala/org/apache/spark/metrics/source/StaticSources.scala#L85].
{quote} * namespace=HiveExternalCatalog
 ** *note:*: these metrics are conditional to a configuration parameter: {{spark.metrics.staticSources.enabled}} (default is true)
 ** fileCacheHits.count
 ** filesDiscovered.count
 ** +{color:#ff0000}*hiveClientCalls.count*{color}+
 ** parallelListingJobCount.count
 ** partitionsFetched.count
 * namespace=CodeGenerator
 ** *note:*: these metrics are conditional to a configuration parameter: {{spark.metrics.staticSources.enabled}} (default is true)
 ** compilationTime (histogram)
 ** generatedClassSize (histogram)
 ** generatedMethodSize (histogram)
 ** *{color:#ff0000}+hiveClientCalls.count+{color}*
 ** sourceCodeSize (histogram){quote}
Bug: Refactor pyspark code for common methods in JavaParams and Pipeline/OneVsRest
Status: Open | Priority: Minor | Resolution: 
Created: 24/Apr/20 02:50
Affects Version/s: 3.0.0

Background: See discussion here
https://github.com/apache/spark/pull/28273#discussion_r411462216
and
https://github.com/apache/spark/pull/28279#discussion_r412699397
Bug: make window function using order by optional
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 21/Apr/20 23:55
Affects Version/s: 2.4.5

Hi all

In other sql dialect ,order by is not the must when using window function,we may make it pararmteterized

the below is the case :

*select row_number()over() from test1*

Error: org.apache.spark.sql.AnalysisException: Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table; (state=,code=0)

 

So. i suggest make it as a choice,or we will meet the error when migrate sql from other dialect，such as hive
Bug: flatten the result dataframe of ANOVATest
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Apr/20 04:29
Affects Version/s: 3.1.0

add a new method
{code:java}
@Since("3.1.0")
def test(
    dataset: DataFrame,
    featuresCol: String,
    labelCol: String,
    flatten: Boolean): DataFrame {code}
 

Similar to new {{test}} method in {{ChiSquareTest}}, it will:

1, support df operation on the returned df;

2, make driver no longer a bottleneck when dim is high
Bug: Add codegen-on test coverage for some tests in SQLMetricsSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Apr/20 23:15
Affects Version/s: 3.1.0

Many tests in SQLMetricsSuite run only with codegen turned off. Some complex code paths (for example, generated code in "SortMergeJoin metrics") aren't exercised at all. The generated code should be tested as well.

*List of tests that run with codegen off*

Filter metrics, SortMergeJoin metrics, SortMergeJoin(outer) metrics, BroadcastHashJoin metrics,  ShuffledHashJoin metrics, BroadcastHashJoin(outer) metrics, BroadcastNestedLoopJoin metrics, BroadcastLeftSemiJoinHash metrics, CartesianProduct metrics,  SortMergeJoin(left-anti) metrics
Bug: Benchmark conversions to/from Java 8 date-time types
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Apr/20 19:24
Affects Version/s: 3.0.0

DATE and TIMESTAMP column values can be converted to java.sql.Date and java.sql.Timestamp (by default), or to Java 8 date-time types java.time.LocalDate and java.time.Instant when spark.sql.datetime.java8API.enabled is set to true. DateTimeBenchmarks misses benchmarks of Java 8 date/timestamps. The ticket aims to fix that.
Bug: Allow broadcast variables when using isin code
Status: Resolved | Priority: Minor | Resolution: Abandoned
Created: 10/Apr/20 20:15
Affects Version/s: 2.4.0

*I would like some help to explore if the following feature makes sense and what's the best way to implement it. Allow users to use "isin" (or similar) predicates with huge Lists (which might be broadcasted).*

*Abandoning, I think maybe the best option without changing all the architecture is to deal with it inside the RDD/RelationProvider, when it finds a big list to be pushed-down (as long as unhandled Filters makes it disappear), it can just broadcast it and then use it. Only problem is that this might leave non-cleaned broadcast variables, unless there's a way (or someone implements it) to cleanup such broadcasts after an action.*

Read last comment for more info :)

As of now (AFAIK), users can only provide a list/sequence of elements which will be sent as part of the "In" predicate, which is part of the task, to the executors. So when this list is huge, this causes tasks to be "huge" (specially when the task number is big).

I'm coming from https://stackoverflow.com/questions/61111172/scala-spark-isin-broadcast-list (I'm the creator of the post, and also the 2nd answer). 

I know this is not common, but can be useful for people reading from "servers" which are not in the same nodes than Spark (maybe S3/cloud?). In my concrete case, querying Kudu with 200.000 elements in the isin takes ~2minutes (+1 minute "calculating plan" to send the BIG tasks to the executors) versus 30 minutes doing a full-scan (even if semi-filtered, most of the data has the same nature and thus is hard to filter, unless I use this "200.000" list) on my table.

More or less I have a clear view (explained in stackoverflow) on what to modify If I wanted to create my own "BroadcastIn" in https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala, however with my limited "idea", anyone who implemented In pushdown would have to re-adapt it to the new "BroadcastIn"

I've looked at Spark 3.0 sources and nothing has changed there, but I'm not 100% sure if there's an alternative there (I'm stuck in 2.4 anyways, working for a corporation which uses that version).

Maybe I can go ahead with a PR (slowly, in my free time), however I'm not an expert in Scala syntax/syntactic sugar, *anyone has any idea on how to extend current case class "In" while keeping the compatibility with PushDown implementations (aka creating a new Predicate not allowed I think), but not having only Seq[Expression] in the case class, but also allow Broadcast[Seq[Expression]]?* This is what makes the tasks huge, I made a test-run with a predicate which receives and Broadcast variable and uses the value inside, and it works much better.
Bug: Make html elements for a paged table possible to have different id attribute.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Apr/20 18:17
Affects Version/s: 3.0.0

Some pages have a paged table and  page navigations above / below the table.
But corresponding HTML elements between the two page navigations for a table have the same id attribute. Every id element should be unique.
For example, there are two `form-completedJob-table-page` id in JobsPage.
Bug: The usage of getopts and case statement is wrong in do-release.sh
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Apr/20 17:59
Affects Version/s: 3.1.0

In the current master, do-release.sh contains the following code.

{code}
while getopts "bn" opt; do
  case $opt in
    b) GIT_BRANCH=$OPTARG ;;
    n) DRY_RUN=1 ;;
    ?) error "Invalid option: $OPTARG" ;;
  esac
done
{code}

There are 3 wrong usage in getopts and case statement.
1. To set  $OPTARG to an argument passed for the option "b", the parameter for getopts should be "b:".
2. To set $OPTARG to the invalid option name passed, the parameter for getopts starts with ":".
3. It's minor but to match the character "?", it's better to escape like "\?".