av9ash/GitBugs huggingface dataset:
Bug: Failed mount warning from kubernetes and support the "optional" mount.
Status: Open | Priority: Major | Resolution: 
Created: 10/Feb/20 09:52
Affects Version/s: 3.0.0

1)https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#configmapvolumesource-v1-core

Kubernetes allows an optional field indicating, that if the mount for this config map fails, then it is not reattempted nor the pod is declared to be failed.

In our current code base, we try to mount the volumes and create them later, it works because, kubernetes reattempts failed mounting attempt, because the `optional` field is `false` by default.

But, if this optional field is set to true, then that mount will not take place at all. Because, when the mount is performed the volume is not created - so mount fails. And this time the mount is not reattempted because the optional field is set as true.
Bug: Improve CaseWhen sub-expression equality
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Feb/20 09:41
Affects Version/s: 3.1.0

We only support the first condition expression. But we can improve this pattern:
{code:sql}
CASE WHEN testUdf(a) > 3 THEN 4
WHEN testUdf(a) = 3 THEN 3
WHEN testUdf(a) = 2 THEN 2
WHEN testUdf(a) = 1 THEN 1
ELSE 0 END
{code}
Bug: Remove meaningless attributeId when Explain SQL query
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Feb/20 15:57
Affects Version/s: 3.1.0

When EXPLAIN sql query, the generated alias shouldn't include expr/attribute id. This will provide better readability of Explain results. This is a follow-up to address [#27368 (comment)|https://github.com/apache/spark/pull/27368#discussion_r376927143].
Bug: Constraints inferred from inferred attributes
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Feb/20 06:05
Affects Version/s: 3.1.0

{code:scala}
scala> spark.range(20).selectExpr("id as a", "id as b", "id as c").write.saveAsTable("t1")

scala> spark.sql("select count(*) from t1 where a = b and b = c and (c = 3 or c = 13)").explain(false)
== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[count(1)])
+- Exchange SinglePartition, true, [id=#76]
   +- *(1) HashAggregate(keys=[], functions=[partial_count(1)])
      +- *(1) Project
         +- *(1) Filter (((((((isnotnull(c#36L) AND ((b#35L = 3) OR (b#35L = 13))) AND isnotnull(b#35L)) AND (a#34L = c#36L)) AND isnotnull(a#34L)) AND (a#34L = b#35L)) AND (b#35L = c#36L)) AND ((c#36L = 3) OR (c#36L = 13)))
            +- *(1) ColumnarToRow
               +- FileScan parquet default.t1[a#34L,b#35L,c#36L] Batched: true, DataFilters: [isnotnull(c#36L), ((b#35L = 3) OR (b#35L = 13)), isnotnull(b#35L), (a#34L = c#36L), isnotnull(a#..., Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/Downloads/spark-3.0.0-preview2-bin-hadoop2.7/spark-warehous..., PartitionFilters: [], PushedFilters: [IsNotNull(c), Or(EqualTo(b,3),EqualTo(b,13)), IsNotNull(b), IsNotNull(a), Or(EqualTo(c,3),EqualT..., ReadSchema: struct<a:bigint,b:bigint,c:bigint>
{code}

We can infer more constraints: {{(a#34L = 3) OR (a#34L = 13)}}.
Bug: Throw Exception if runHive(sql) failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/20 17:25
Affects Version/s: 3.0.0

At present, HiveClientImpl.runHive will not throw an exception when it runs incorrectly, which will cause it to fail to feedback error information normally.
Example
{code:scala}
spark.sql("add jar file:///tmp/test.jar")
spark.sql("show databases").show()
{code}
/tmp/test.jar doesn't exist, thus add jar is failed. However this code will run completely without causing application failure.
Bug: Improve the readability of EXPLAIN FORMATTED style
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/20 12:51
Affects Version/s: 3.1.0

The style of EXPLAIN FORMATTED output needs to be improved. We’ve already got some observations/ideas in
[https://github.com/apache/spark/pull/27368#discussion_r376694496]. 
 
TODOs:
1.Using comma as the separator is not clear, especially commas are used inside the expressions too.
2.Show the column counts first? For example, `Results [4]: …`
3.Currently the attribute names are automatically generated, this need to refined.
4.Add arguments field in common implications as EXPLAIN FORMATTED did in QueryPlan
...
Bug: InsertAdaptiveSparkPlan should apply AQE on child plan of write commands
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/20 04:12
Affects Version/s: 3.1.0

Apply AQE on write commands with child plan will expose {{LogicalQueryStage}} to {{Analyzer}} while it should hider under {{AdaptiveSparkPlanExec}} only to avoid unexpected broken.
Bug: Remove the max split config after changing the multi sub joins to multi sub partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/20 02:21
Affects Version/s: 3.0.0

After merged PR#27493, we not need the "spark.sql.adaptive.skewedJoinOptimization.skewedPartitionMaxSplits" config to resolve the ui issue when split more sub joins.
Bug: Subqueries should not be AQE-ed if main query is not
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 20:09
Affects Version/s: 3.0.0

Currently there are unsupported queries by AQE, e.g., queries that contain DPP filters. But if the main query is unsupported while the sub-query is, the subquery itself will be AQE-ed, which can lead to performance regressions due to missed opportunity of sub-query reuse.
Bug: Combine the skewed readers into one in AQE skew join optimizations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/20 16:43
Affects Version/s: 3.0.0

Assume we have N partitions based on the original join keys, and for a specific partition id {{Pi}} (i = 1 to N), we slice the left partition into {{Li}} sub-partitions (L = 1 if no skew; L > 1 if skewed), the right partition into {{Mi}} sub-partitions (M = 1 if no skew; M > 1 if skewed). With the current approach, we’ll end up with a sum of {{Li}} * {{Mi}} (i = 1 to N where Li > 1 or Mi > 1) plus one (for the rest of the partitions without skew) joins. *This can be a serious performance concern as the size of the query plan now depends on the number and size of skewed partitions.*

Now instead of generating so many joins we can create a “repeated” reader for either side of the join so that:
 # for the left side, with each partition id Pi and any given slice {{Sj}} in {{Pi}} (j = 1 to Li), it generates {{Mi}} repeated partitions with respective join keys as {{PiSjT1}}, {{PiSjT2}}, …, {{PiSjTm}}

 # for the right side, with each partition id Pi and any given slice {{Tk}} in {{Pi}} (k = 1 to Mi), it generates {{Li}} repeated partitions with respective join keys as {{PiS1Tk}}, {{PiS2Tk}}, …, {{PiSlTk}}

That way, we can have one SMJ for all the partitions and only one type of special reader.
Bug: ML, GraphX 3.0 QA: Programming guide update and migration guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 03:54
Affects Version/s: 3.0.0

Before the release, we need to update the MLlib and GraphX Programming Guides. Updates will include:
 * Add migration guide subsection.
 ** Use the results of the QA audit JIRAs.
 * Check phrasing, especially in main sections (for outdated items such as "In this release, ...")
Bug: Support NativeBlas for level-1 routines
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/20 14:39
Affects Version/s: 2.4.3

In current ML BLAS.scala, all level-1 routines are fixed to use java implementation. But NativeBLAS(intel MKL, OpenBLAS) can bring up to 11X performance improvement based on performance test which apply direct calls against these methods. We should provide a way to allow user take advantage of NativeBLAS for level-1 routines, like also apply nativeBLAS for these methods.
Bug: ML 3.0 QA: API: Java compatibility, docs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 03:52
Affects Version/s: 3.0.0

Check Java compatibility for this release:
 * APIs in {{spark.ml}}
 * New APIs in {{spark.mllib}} (There should be few, if any.)

Checking compatibility means:
 * Checking for differences in how Scala and Java handle types. Some items to look out for are:
 ** Check for generic "Object" types where Java cannot understand complex Scala types.
 *** *Note*: The Java docs do not always match the bytecode. If you find a problem, please verify it using {{javap}}.
 ** Check Scala objects (especially with nesting!) carefully. These may not be understood in Java, or they may be accessible only via the weirdly named Java types (with "$" or "#") which are generated by the Scala compiler.
 ** Check for uses of Scala and Java enumerations, which can show up oddly in the other language's doc. (In {{spark.ml}}, we have largely tried to avoid using enumerations, and have instead favored plain strings.)
 * Check for differences in generated Scala vs Java docs. E.g., one past issue was that Javadocs did not respect Scala's package private modifier.

If you find issues, please comment here, or for larger items, create separate JIRAs and link here as "requires".
 * Remember that we should not break APIs from previous releases. If you find a problem, check if it was introduced in this Spark release (in which case we can fix it) or in a previous one (in which case we can create a java-friendly version of the API).
 * If needed for complex issues, create small Java unit tests which execute each method. (Algorithmic correctness can be checked in Scala.)

Recommendations for how to complete this task:
 * There are not great tools. In the past, this task has been done by:
 ** Generating API docs
 ** Building JAR and outputting the Java class signatures for MLlib
 ** Manually inspecting and searching the docs and class signatures for issues
 * If you do have ideas for better tooling, please say so we can make this task easier in the future!
Bug: ML 3.0 QA: API: Python API coverage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 03:51
Affects Version/s: 3.0.0

For new public APIs added to MLlib ({{spark.ml}} only), we need to check the generated HTML doc and compare the Scala & Python versions.
 * *GOAL*: Audit and create JIRAs to fix in the next release.
 * *NON-GOAL*: This JIRA is _not_ for fixing the API parity issues.

We need to track:
 * Inconsistency: Do class/method/parameter names match?
 * Docs: Is the Python doc missing or just a stub? We want the Python doc to be as complete as the Scala doc.
 * API breaking changes: These should be very rare but are occasionally either necessary (intentional) or accidental. These must be recorded and added in the Migration Guide for this release.
 ** Note: If the API change is for an Alpha/Experimental/DeveloperApi component, please note that as well.
 * Missing classes/methods/parameters: We should create to-do JIRAs for functionality missing from Python, to be added in the next release cycle. *Please use a _separate_ JIRA (linked below as "requires") for this list of to-do items.*
Bug: ML, GraphX 3.0 QA: Update user guide for new features & APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 03:53
Affects Version/s: 3.0.0

Check the user guide vs. a list of new APIs (classes, methods, data members) to see what items require updates to the user guide.

For each feature missing user guide doc:
 * Create a JIRA for that feature, and assign it to the author of the feature
 * Link it to (a) the original JIRA which introduced that feature ("related to") and (b) to this JIRA ("requires").

For MLlib:
 * This task does not include major reorganizations for the programming guide.
 * We should now begin copying algorithm details from the spark.mllib guide to spark.ml as needed, rather than just linking back to the corresponding algorithms in the spark.mllib user guide.

If you would like to work on this task, please comment, and we can create & link JIRAs for parts of this work.
Bug: Different behavior between rolling and non-rolling event log
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/20 20:25
Affects Version/s: 3.0.0

When creating a rolling event log, the application directory is created with a call to FileSystem.mkdirs, with the file permission 770. The default behavior of HDFS is to set the permission of a file created with FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the permission in the API call and umask is a system value set by fs.permissions.umask-mode and defaults to 0022. This means, with default settings, any mkdirs call can have at most 755 permissions, which causes rolling event log directories to be created with 750 permissions. This causes the history server to be unable to prune old applications if they are not run by the same user running the history server.

This is not a problem for non-rolling logs, because it uses SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then calls FileSystem.setPermission with 770 after the file has been created. setPermission doesn't have the umask applied to it, so this works fine.

Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm not sure the reason that's set in the first place or if this would hurt anything else. The main issue is there is different behavior between rolling and non-rolling event logs that might want to be updated in this repo to be consistent across each.
Bug: Use latest URL for Google Cloud Storage mirror of Maven Central 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 00:31
Affects Version/s: 2.4.5

Per https://travis-ci.community/t/maven-builds-that-use-the-gcs-maven-central-mirror-should-update-their-paths/5926 and https://storage-download.googleapis.com/maven-central/index.html, the Google Cloud Storage of Maven Central is now offering newer URLS which are updated more frequently; as a followup to SPARK-30601 we should update our build to use those new URLs.
Bug: Document ALTER TABLE statement in SQL Reference [Phase 2]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/20 17:37
Affects Version/s: 3.0.0

https://issues.apache.org/jira/browse/SPARK-28791 only covers a subset of ALTER TABLE statements. See the doc in preview-2 [https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-alter-table.html] 

We should add all the supported ALTER TABLE syntax. See [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L157-L198]
Bug: Accelerate InMemoryStore with a new index
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/20 22:51
Affects Version/s: 3.1.0

Spark uses the class `InMemoryStore` as the KV storage for live UI and history server(by default if no LevelDB file path is provided).
In `InMemoryStore`, all the task data in one application is stored in a hashmap, which key is the task ID and the value is the task data. This fine for getting or deleting with a provided task ID.
However, Spark stage UI always shows all the task data in one stage and the current implementation is to look up all the values in the hashmap. The time complexity is O(numOfTasks). 
Also, when there are too many stages (>spark.ui.retainedStages), Spark will linearly try to look up all the task data of the stages to be deleted as well.

This can be very bad for a large application with many stages and tasks. We can improve it by allowing the natural key of an entity to have a real parent index. So that on each lookup with parent node provided, Spark can look up all the natural keys(in our case, the task IDs) first, and then find the data with the natural keys in the hashmap.
Bug: Arrow enabled: to_pandas with date column fails
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/Feb/20 16:19
Affects Version/s: 2.4.5

Hi,

there seems to be a bug in the arrow enabled to_pandas conversion from spark dataframe to pandas dataframe when the dataframe has a column of type DateType. Here is a minimal example to reproduce the issue:
{code:java}
spark = SparkSession.builder.getOrCreate()
is_arrow_enabled = spark.conf.get("spark.sql.execution.arrow.enabled")
print("Arrow optimization is enabled: " + is_arrow_enabled)
spark_df = spark.createDataFrame(
    [['2019-12-06']], 'created_at: string') \
    .withColumn('created_at', F.to_date('created_at'))

# works
spark_df.toPandas()

spark.conf.set("spark.sql.execution.arrow.enabled", 'true')
is_arrow_enabled = spark.conf.get("spark.sql.execution.arrow.enabled")
print("Arrow optimization is enabled: " + is_arrow_enabled)
# raises AttributeError: Can only use .dt accessor with datetimelike values
# series is still of type object, .dt does not exist
spark_df.toPandas(){code}
A fix would be to modify the _check_series_convert_date function in pyspark.sql.types to:
{code:java}
def _check_series_convert_date(series, data_type):
    """
    Cast the series to datetime.date if it's a date type, otherwise returns the original series.    :param series: pandas.Series
    :param data_type: a Spark data type for the series
    """
    from pyspark.sql.utils import require_minimum_pandas_version
    require_minimum_pandas_version()    from pandas import to_datetime
    if type(data_type) == DateType:
        return to_datetime(series).dt.date
    else:
        return series
{code}
Let me know if I should prepare a Pull Request for the 2.4.5 branch.

I have not tested the behavior on master branch.

 

Thanks,

Nicolas
Bug: V1 table name should be fully qualified if catalog name is provided
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/20 06:43
Affects Version/s: 3.0.0

For the following example,

{code:java}
sql("CREATE TABLE t USING json AS SELECT 1 AS i")
sql("SELECT * FROM spark_catalog.t")
{code}
`spark_catalog.t` is expanded to `spark_catalog.default.t` assuming that the current namespace is `default`. However, this is not consistent with V2 behavior where namespace should be provided if the catalog name is also provided.
Bug: Log better message when accelerate resource is empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 14:02
Affects Version/s: 3.0.0

It's weird to see cpu/memory resources after logging resource is empty:



{code:java}
20/02/25 21:47:55 INFO ResourceUtils: ==============================================================
20/02/25 21:47:55 INFO ResourceUtils: Resources for spark.driver:
20/02/25 21:47:55 INFO ResourceUtils: ==============================================================
20/02/25 21:47:55 INFO SparkContext: Submitted application: Spark shell
20/02/25 21:47:55 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
{code}
Bug: ResourceProfile and Builder should be private in spark 3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/20 18:49
Affects Version/s: 3.0.0

it looks like ResourceProfile and ResourceProfileBuilder accidentally got opened up - they should be private[spark] until the stage level scheduling feature is complete, which won't make the 3.0 release.  So make them private in 3.0 branch
Bug: ResourceDiscoveryPluginSuite sometimes fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/20 16:31
Affects Version/s: 3.0.0

{color:#1d1c1d}I noted that {color}{{ResourceDiscoveryPluginSuite}}{color:#1d1c1d} is quite flack recently. And I did some investigation locally and it looks it’s a pure problem of the time:{color}
[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119030/testReport/]
[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119005/testReport/]
[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119029/testReport/]
{color:#1d1c1d}and many other places...{color}
Bug: Adjust abstraction structure for join operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/20 16:04
Affects Version/s: 3.1.0

Currently the join operators are not well abstracted, since there are lot of common logic. A trait can be created for easier pattern matching and other future handiness. 
 
This is a follow up based on comment 
[https://github.com/apache/spark/pull/27509#discussion_r379613391]
Bug: PruneHiveTablePartitions should be executed as earlyScanPushDownRules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/20 12:17
Affects Version/s: 3.0.0

Similar to PruneFileSourcePartitions, PruneHiveTablePartitions should also be executed as earlyScanPushDownRules to eliminate the impact on statistic computation later.
Bug: Column resolution doesn't respect current catalog/namespace for v2 tables.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/20 22:04
Affects Version/s: 3.0.0

For v1 tables, you can perform the following:
{code:sql}
SELECT default.t.id FROM t;
{code}

For v2 tables, the following fails:
{code:sql}
USE testcat.ns1.ns2;
SELECT testcat.ns1.ns2.t.id FROM t;

org.apache.spark.sql.AnalysisException: cannot resolve '`testcat.ns1.ns2.t.id`' given input columns: [t.id, t.point]; line 1 pos 7;
{code}
Bug: Overflow/round errors in conversions of milliseconds to/from microseconds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/20 09:04
Affects Version/s: 3.0.0

Spark has special methods in DataTimeUtils for converting microseconds from/to milliseconds - `fromMillis` and `toMillis()`. The methods handle arithmetic overflow and round negative values. The ticket aims to review all places in Spark SQL where microseconds are converted from/to milliseconds, and replace them by util methods from DateTimeUtils.
Bug: Thrift RowBasedSet serialization throws NullPointerException on NULL BigDecimal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/20 18:22
Affects Version/s: 3.0.0

Adding the following test to {{SparkThriftServerProtocolVersionsSuite}} reproduces the issue:
{code:scala}
test(s"$version get null as decimal") {
  testExecuteStatementWithProtocolVersion(version,
    "SELECT cast(null as decimal)") { rs =>
    assert(rs.next())
    assert(rs.getBigDecimal(1) === null)
  }
}{code}

The bug was introduced in https://github.com/apache/spark/commit/163f4a4
Bug: Fix the warning for requiring cores to be limiting resource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 20:06
Affects Version/s: 3.0.0

[https://github.com/apache/spark/pull/27615]

Changed a warning to only show when dynamic allocation enabled. In reality other things rely on the "slots", so the warning should always be present if custom resources are specified.
Bug: Show "batch ID" in tool tip string for Structured Streaming UI graphs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 22:37
Affects Version/s: 3.0.0

While we introduced the new UI for Structured Streaming, it was basically the port of Streaming UI to Structured Streaming.

That missed the major difference between twos, ID of the "batch". In Spark Streaming (DStream), timestamp is the key of the batch and you will need to find any relevant information with such timestamp.

In Structured Streaming, batchId is the key of the batch. It's not a simple difference because there're some known latency issues on periodical batches (at least in FileStreamSource and FileStreamSink) and it's bothering to find the right batchId if you only know about timestamp where the most of things are logged with the batchId in Structured Streaming world.

This issue tracks the efforts on showing batchId as main aspect of x axis, but only in tooltip string as changing x axis breaks coupling between Spark Streaming and Structured Streaming and lots of code should be duplicated and rewritten.