av9ash/GitBugs huggingface dataset:
Bug: Avoid floating-point division in millisToDays
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 14/Apr/20 17:18
Affects Version/s: 2.4.5

As the benchmark https://github.com/MaxGekk/spark/pull/27, and comparison to Spark 3.0+an optimisation of fromJavaDate in https://github.com/apache/spark/pull/28205 show that floating-point ops in millisToDays badly impact on the performance of conversion java.sql.Date to Catalyst's date values. The ticket aims to replace double ops by int/long ops.
Bug: Clean up the SQL documents in docs/sql-ref*
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Apr/20 07:58
Affects Version/s: 3.0.0

This ticket intends to clean up the SQL documents in `doc/sql-ref*`.
Main changes are as follows;

- Fixes wrong syntaxes and capitalize sub-titles
 - Adds some DDL queries in `Examples` so that users can run examples there
 - Makes query output in `Examples` follows the `Dataset.showString` (right-aligned) format 
 - Adds/Removes spaces, Indents, or blank lines to follow the format below;

{code}
---
license...
---

### Description

Writes what's the syntax is.

### Syntax

{% highlight sql %}
SELECT...
 WHERE... // 4 indents after the second line
 ...
{% endhighlight %}

### Parameters

<dl>
 <!-- 2 indents in this section -->
 <dt><code><em>Param Name</em></code></dt>
 <dd>
 Param Description
 </dd>
 ...
</dl>

### Examples

{% highlight sql %}
-- It is better that users are able to execute example queries here.
-- So, we prepare test data in the first section if possible.
CREATE TABLE t (key STRING, value DOUBLE);
INSERT INTO t VALUES
 ('a', 1.0), ('a', 2.0), ('b', 3.0), ('c', 4.0);

-- query output has 2 indents and it follows the `Dataset.showString`
-- format (right-aligned).
SELECT * FROM t;
 +---+-----+
 |key|value|
 +---+-----+
 | a| 1.0|
 | a| 2.0|
 | b| 3.0|
 | c| 4.0|
 +---+-----+

-- Query statements after the second line have 4 indents.
SELECT key, SUM(value)
 FROM t
 GROUP BY key;
 +---+----------+ 
 |key|sum(value)|
 +---+----------+
 | c| 4.0|
 | b| 3.0|
 | a| 3.0|
 +---+----------+
...
{% endhighlight %}

### Related Statements

* [XXX](xxx.html)
 * ...
{code}
Bug: [SPARK][Core]preferred location causing single node be a hot spot
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 09/Apr/20 10:58
Affects Version/s: 2.3.4

my job is run as follows:
 # build model, model is saved in HDFS named prob1. prob2. probN
 # then load it to RDD from certain ProbInputformat
 # do some calculation

The driver node which builds the model is scheduled more frequently than other nodes, because the HDFS block is firstly written to itself. The scheduling hot spot is unnecessary, and should better be flattened.
Bug: Set time zone in DateTimeBenchmark and DateTimeRebaseBenchmark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Apr/20 13:23
Affects Version/s: 3.0.0

Performance of date-time function can depend on the system JVM time zone or SQL config spark.sql.session.timeZone. To avoid any fluctuations of benchmarks results, the ticket aims to set a time zone explicitly in date-time benchmarks DateTimeBenchmark and DateTimeRebaseBenchmark.
Bug: Metadata-only queries should not include subquery in partition filters
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 28/Apr/20 04:04
Affects Version/s: 2.4.0

When using SPARK-23877, some sql execution errors.

code:
{code:scala}
        sql("set spark.sql.optimizer.metadataOnly=true")
        sql("CREATE TABLE test_tbl (a INT,d STRING,h STRING) USING PARQUET PARTITIONED BY (d ,h)")
        sql("""
            |INSERT OVERWRITE TABLE test_tbl PARTITION(d,h)
            |SELECT 1,'2020-01-01','23'
            |UNION ALL
            |SELECT 2,'2020-01-02','01'
            |UNION ALL
            |SELECT 3,'2020-01-02','02'
            """.stripMargin)
        sql(
          s"""
             |SELECT d, MAX(h) AS h
             |FROM test_tbl
             |WHERE d= (
             |  SELECT MAX(d) AS d
             |  FROM test_tbl
             |)
             |GROUP BY d
        """.stripMargin).collect()
{code}
Exception:
{code:java}
java.lang.UnsupportedOperationException: Cannot evaluate expression: scalar-subquery#48 []

...
at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.prunePartitions(PartitioningAwareFileIndex.scala:180)
{code}
optimizedPlan:
{code:java}
Aggregate [d#245], [d#245, max(h#246) AS h#243]
+- Project [d#245, h#246]
   +- Filter (isnotnull(d#245) AND (d#245 = scalar-subquery#242 []))
      :  +- Aggregate [max(d#245) AS d#241]
      :     +- LocalRelation <empty>, [d#245]
      +- Relation[a#244,d#245,h#246] parquet
{code}
Bug: Set setwd in R documentation build
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Apr/20 15:34
Affects Version/s: 3.0.0

{code}
> library(devtools); devtools::document(pkg="./pkg", roclets=c("rd"))
Loading required package: usethis
Error: Could not find package root, is your working directory inside a package?
{code}

Seems like in some environments it fails as above. 

https://stackoverflow.com/questions/52670051/how-to-troubleshoot-error-could-not-find-package-root
https://groups.google.com/forum/#!topic/rdevtools/79jjjdc_wjg
Bug: StreamingQuery stop() timeout exception should include the stream ID
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Apr/20 20:37
Affects Version/s: 3.1.0

[https://github.com/apache/spark/pull/26771/|https://github.com/apache/spark/pull/26771/files] added a conf to set a timeout when stop()ing streams; if the stream does not terminate in this time, an exception is thrown. Having hit this issue once in production, it would be nice to have the stream ID in the error message.
Bug: Add .asf.yaml to control Github settings
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Apr/20 12:25
Affects Version/s: 3.0.0

We added .asf.yaml file to Apache Airflow project ([PR|https://github.com/apache/airflow/pull/6689]) and I think it would be good to have the Spark's website at the top of Github repo.

Also, this would allow Spark's PMC Members and committer to control common Github project settings by themselves without having to ask Apache INFRA team.

More info: https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories

We (Airflow PMC and Committers) used this file to enable Github issues and disable PR Merge button etc
Bug: Incorrect datetime filter when reading Parquet files written in legacy mode
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 07/Jul/21 18:45
Affects Version/s: 3.1.2

We're seeing incorrect date filters on Parquet files written by Spark 2 or by Spark 3 with legacy rebase mode.

This is the expected behavior that we see in _corrected_ mode (Spark 3.1.2):
{code:title=Good (Corrected Mode)}
>>> spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")

>>> spark.sql("SELECT DATE '0001-01-01' AS date").write.mode("overwrite").parquet("date_written_by_spark3_corrected")

>>> spark.read.parquet("date_written_by_spark3_corrected").selectExpr("date", "date = '0001-01-01'").show()
+----------+-------------------+
|      date|(date = 0001-01-01)|
+----------+-------------------+
|0001-01-01|               true|
+----------+-------------------+

>>> spark.read.parquet("date_written_by_spark3_corrected").where("date = '0001-01-01'").show()
+----------+
|      date|
+----------+
|0001-01-01|
+----------+
{code}

This is how we get incorrect results in _legacy_ mode, in this case the filter is dropping rows it shouldn't:
{code:title=Bad (Legacy Mode)}
In [27]: spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY")

>>> spark.sql("SELECT DATE '0001-01-01' AS date").write.mode("overwrite").parquet("date_written_by_spark3_legacy")

>>> spark.read.parquet("date_written_by_spark3_legacy").selectExpr("date", "date = '0001-01-01'").show()
+----------+-------------------+
|      date|(date = 0001-01-01)|
+----------+-------------------+
|0001-01-01|               true|
+----------+-------------------+

>>> spark.read.parquet("date_written_by_spark3_legacy").where("date = '0001-01-01'").show()
+----+
|date|
+----+
+----+

>>> spark.read.parquet("date_written_by_spark3_legacy").where("date = '0001-01-01'").explain()
== Physical Plan ==
*(1) Filter (isnotnull(date#154) AND (date#154 = -719162))
+- *(1) ColumnarToRow
   +- FileScan parquet [date#154] Batched: true, DataFilters: [isnotnull(date#154), (date#154 = -719162)], Format: Parquet, Location: InMemoryFileIndex[file:/Volumes/git/spark-installs/spark-3.1.2-bin-hadoop3.2/date_written_by_spar..., PartitionFilters: [], PushedFilters: [IsNotNull(date), EqualTo(date,0001-01-01)], ReadSchema: struct<date:date>
{code}
Bug: Write data loss caused by stage retry when enable v2 FileOutputCommitter
Status: Resolved | Priority: Critical | Resolution: Auto Closed
Created: 13/Jul/21 12:56
Affects Version/s: 2.2.1

All our ETL scenarios are configured: mapreduce.fileoutputcommitter.algorithm.version=2, when shuffle fetchFailed occurs, the stage retry is triggered, and then the zombie stage and the retry stage may write tasks of the same part at the same time, and their task directory and file name are exactly the same. This may cause data part loss due to conflicts between delete and rename operations.

For example, this is also a data loss case I encountered recently: Stage 5.0 is a zombie stage caused by shuffle FetchFailed, and stage 5.1 is a retry stage. They have two tasks concurrently writing the same part file: part-00298.
 # The task of stage 5.1 has preemptively created part file: part-00298 and written data.
 # At the same time as the task commit of stage 5.1, the task of sage 5.0 is going to create this part file to write data, because the file already exists, it throw an exception and delete the task's temporary directory.
 # Then stage 5.0 starts commitTask, it will traverse the sub-directories and execute rename. At this time, because the file has been deleted, it finally moves empty without any exception, which causes data loss.

 

I read this part of the code, and currently I think of two ideas:
 # Add stageAttemptNumber to taskAttemptPath to avoid conflicts.
 # Check the number of files after commitTask, and throw an exception directly when it is found to be missing.
Bug: Remove Python 3.6 support in codebase and CI
Status: Resolved | Priority: Critical | Resolution: Done
Created: 15/Jul/21 01:35
Affects Version/s: 3.3.0

Python 3.6 is deprecated via SPARK-35938 at Apache Spark 3.2. We should remove it in Spark 3.3.

This JIRA also target to all the changes in CI and development not only user-facing changes.
Bug: Handle the COUNT bug for correlated IN/EXISTS subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 04:36
Affects Version/s: 3.2.0

Correlated IN/EXISTS subqueries are also subject to the COUNT bug which is not handled.

{code:sql}
create view t1(c1, c2) as values (0, 1), (1, 2)
create view t2(c1, c2) as values (0, 2), (0, 3)

-- Example 1: IN subquery
select * from t1 where c1 in (select count(*) + 1 from t2 where t1.c1 = t2.c1)
-- Correct answer: (1, 2)
+---+---+
|c1 |c2 |
+---+---+
+---+---+

-- Example 2: EXISTS subquery
select * from t1 where exists (select count(*) from t2 where t1.c1 = t2.c1)
-- Correct answer: [(0, 1), (1, 2)]
+---+---+
|c1 |c2 |
+---+---+
|0  |1  |
+---+---+
{code}
Bug: Use uuid as app id in kubernetes client mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/21 07:38
Affects Version/s: 3.2.0

Currently, spark on kubernetes with client mode would use `"spark-application-" + System.currentTimeMillis` as app id by default. It would cause app id conflict if submit several spark applications to kubernetes cluster in a short time.

Unfortunately, the event log use app id as the file name. With the conflict event log file, the exception was thrown.
{code:java}
Caused by: java.io.FileNotFoundException: File does not exist: xxx/spark-application-1624766876324.lz4.inprogress (inode 5984170846) Holder does not have any open files.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2697)
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.analyzeFileState(FSDirWriteFileOp.java:521)
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:161)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2579)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:846)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817)
{code}
Bug: Support ORDER BY and LIMIT to be on the correlation path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 23:00
Affects Version/s: 3.2.0

A correlation path is defined as the sub-tree of all the operators that are on the path from the operator hosting the correlated expressions up to the operator producing the correlated values. 

We want to support ORDER BY (Sort) and LIMT operators to be on the correlation path to achieve better feature parity with Postgres. Here is an example query in `postgreSQL/join.sql`:

{code:SQL}
select * from
  text_tbl t1
  left join int8_tbl i8
  on i8.q2 = 123,
  lateral (select i8.q1, t2.f1 from text_tbl t2 limit 1) as ss
where t1.f1 = ss.f1;
{code}
Bug: Unify the logic to handle COUNT bug for scalar and lateral subqueries
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/21 04:30
Affects Version/s: 3.2.0

Currently, for scalar subqueries, the logic to handle the count bug is in `RewriteCorrelatedScalarSubquery` (constructLeftJoins) while lateral subqueries rely on `DecorrelateInnerQuery` to handle the COUNT bug. 

We should unify them without introducing additional left outer joins for scalar subqueries.
Bug: Enable DecorrelateInnerQuery for IN/EXISTS subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 04:20
Affects Version/s: 3.2.0

Currently, `DecorrelateInnerQuery` is only enabled for scalar and lateral subqueries. We should enable `DecorrelateInnerQuery` for IN/EXISTS subqueries. Note we need to add the logic to rewrite domain joins in `RewritePredicateSubquery`.
Bug: Do not fail application in kubernetes if name is too long
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 28/Jul/21 03:13
Affects Version/s: 3.3.0

If we have a long spark app name and start with k8s master, we will get the execption.

{code:java}
java.lang.IllegalArgumentException: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-89fe2f7ae71c3570' in spark.kubernetes.executor.podNamePrefix is invalid. must conform https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names and the value length <= 47
	at org.apache.spark.internal.config.TypedConfigBuilder.$anonfun$checkValue$1(ConfigBuilder.scala:108)
	at org.apache.spark.internal.config.TypedConfigBuilder.$anonfun$transform$1(ConfigBuilder.scala:101)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.internal.config.OptionalConfigEntry.readFrom(ConfigEntry.scala:239)
	at org.apache.spark.internal.config.OptionalConfigEntry.readFrom(ConfigEntry.scala:214)
	at org.apache.spark.SparkConf.get(SparkConf.scala:261)
	at org.apache.spark.deploy.k8s.KubernetesConf.get(KubernetesConf.scala:67)
	at org.apache.spark.deploy.k8s.KubernetesExecutorConf.<init>(KubernetesConf.scala:147)
	at org.apache.spark.deploy.k8s.KubernetesConf$.createExecutorConf(KubernetesConf.scala:231)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$2(ExecutorPodsAllocator.scala:367)
{code}

Use app name as the executor pod name is the Spark internal behavior and we should not make application failure.
Bug: Spark SQL can't use "group by" on the column of map type.
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 19/Jul/21 03:27
Affects Version/s: 2.4.5

I want to know why the 'group by' can't use in column of map tyep.

 

*sql:*
select distinct idselect distinct id , cols , extend_value from test.test_table 
-- extend_value's type is map<String, String>.
*error:*
{color:#FF0000}Sql执行错误:org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column extend_value is map<string,string>;{color}
Bug: Issue with record count of data frame while reading in DropMalformed mode
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/21 20:31
Affects Version/s: 2.4.3

I am writing the steps to reproduce the issue for "count" pyspark api while using mode as dropmalformed.

I have a csv sample file in s3 bucket . I am reading the file using pyspark api for csv . I am reading the csv "without schema" and "with schema using mode 'dropmalformed' options  in two different dataframes . While displaying the "with schema using mode 'dropmalformed'" dataframe , the display looks good ,it is not showing the malformed records .But when we apply count api on the dataframe it gives the record count of actual file. I am expecting it should give me valid record count .

here is the code used:-

{code}
without_schema_df=spark.read.csv("s3://noa-poc-lakeformation/data/test_files/sample.csv",header=True)
schema = StructType([ \
    StructField("firstname",StringType(),True), \
    StructField("middlename",StringType(),True), \
    StructField("lastname",StringType(),True), \
    StructField("id", StringType(), True), \
    StructField("gender", StringType(), True), \
    StructField("salary", IntegerType(), True) \
  ])
with_schema_df = spark.read.csv("s3://noa-poc-lakeformation/data/test_files/sample.csv",header=True,schema=schema,mode="DROPMALFORMED")
print("The dataframe with schema")
with_schema_df.show()
print("The dataframe without schema")
without_schema_df.show()
cnt_with_schema=with_schema_df.count()
print("The  records count from with schema df :"+str(cnt_with_schema))
cnt_without_schema=without_schema_df.count()
print("The  records count from without schema df: "+str(cnt_without_schema))
{code}

here is the outputs screen shot 111.PNG is the outputs of the code and inputfile.csv is the input to the code
Bug: TO_UTC_TIMESTAMP and FROM_UTC_TIMESTAMP should return TimestampNTZ
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 08/Jul/21 08:26
Affects Version/s: 3.3.0

Currently, the SQL function to_utc_timestamp is confusing: it just takes the timestamp value in the local timezone and then pretends it’s in the provided timezone and then returns the UTC value, but the result is still treated as local timezone!
The same issue happens in from_utc_timestamp as well.
We even tried to deprecated in the OSS community: https://github.com/apache/spark/commit/c5e83ab92c0cb514963209dc3e70ba0e24570082

We should make TO_UTC_TIMESTAMP and FROM_UTC_TIMESTAMP return TimestampNTZ, which makes a lot of sense. converting the current local time to/from UTC local time.
The functions should accept both Timestamp types: 
1. given TimestampLTZ, convert it to TimestampNTZ and continue step #2
2. given TimestampNTZ, convert it as to/from UTC local time.
Bug: SPIP: Support Customized Kubernetes Schedulers
Status: Resolved | Priority: Major | Resolution: Done
Created: 08/Jul/21 17:47
Affects Version/s: 3.3.0

This is an umbrella issue for tracking the work for supporting Volcano & Yunikorn on Kubernetes. These schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes.

 

Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch).

 

They've taken slightly different approaches to solving the same problem, but from Spark's point of view we should be able to share much of the code.

 

See the initial brainstorming discussion in SPARK-35623.

 

DISCUSSION: [https://lists.apache.org/thread/zv3o62xrob4dvgkbftbv5w5wy75hkbxg]

VOTE: [https://lists.apache.org/thread/cz3cpp8q4pgmh7h35h6lvkwf6g3lwhcd]

VOTE Result: [https://lists.apache.org/thread/nvwfo0yo0q8997vs86o7wkjyby4tbp0m]

Design DOC: [https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg]

Recap slide: [https://lists.apache.org/thread/mwswfwkycj71npwz8gmv1r5nrvpwj77s]
Bug: Support set operators to be on correlation paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 20:25
Affects Version/s: 3.2.0

A correlation path is defined as the sub-tree of all the operators that are on the path from the operator hosting the correlated expressions up to the operator producing the correlated values. 

We want to support set operators such as union and intercept to be on correlation paths by adding them in DecorrelateInnerQuery. Please see page 391 for more details: [https://dl.gi.de/bitstream/handle/20.500.12116/2418/383.pdf]
Bug: Add Dataset.observe(Observation, Column, Column*) to PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 18:58
Affects Version/s: 3.3.0

With SPARK-34806 we now have a way to use the `Dataset.observe` method without the need to interact with `org.apache.spark.sql.util.QueryExecutionListener`. This allows us to easily retrieve observations in PySpark.

Adding a `Dataset.observe(Observation, Column, Column*)` equivalent to PySpark's `DataFrame` is straightforward while it allows to utilise observations from Python.