av9ash/GitBugs huggingface dataset:
Bug: More transparent proxy handling for HTTP redirects
Status: Open | Priority: Major | Resolution: 
Created: 09/Oct/23 05:22
Affects Version/s: 3.5.0

Currently, proxies can be made transparent for hyperlinks in Spark web UIs with spark.ui.proxyRoot or X-Forwarded-Context header alone. However, HTTP redirects (such as job/stage kill) currently requires explicit spark.ui.proxyRedirectUri as well for handling proxy. This is not ideal as proxy hostname may not be known at the time configuring Spark apps.

This can be mitigated by 1) always prepending spark.ui.proxyRoot to redirect path for those proxies that intelligently rewrite Location headers and 2) by using path without hostname (/jobs/, not [https://example.com/jobs/]) for those proxies without Location header rewrites. Then redirects behavior would be basically the same way as other hyperlinks.
h2. Example

Let's say proxy URL is [https://example.org/sparkui/]... forwarding to [http://drv.svc/]...
and spark.ui.proxyRoot is configured to be /sparkui
h3. Existing behavior (without spark.ui.proxyRedirectUri)

job/stage kill links redirects to [http://drv.svc/jobs/] - likely 404
(other hyperlinks are to paths with prefix, e.g., /sparkui/executors - works fine)
h3. After the change 2)

links redirects to /sparkui/jobs/ - works fine
also consistent with other hyperlinks

NOTE: while hostname was originally required in RFC 2616 in 1999, since RFC 7231 in 2014 hostname can be formally omitted as most browsers already supported it (it is rather hard to find any browser that doesn't support it).
Bug: Allow ShuffleDriverComponent to support reliable store with specified executorId
Status: Open | Priority: Major | Resolution: 
Created: 08/Oct/23 14:08
Affects Version/s: 3.5.1

After SPARK-42689, ShuffleDriverComponent.supportsReliableStorage is determined globally.

Downstream projects may have different shuffle policy(cause by cluster loads or columnar support) for different stages, for example or Apache Uniffle with Gluten or Apache Celeborn.

In this situation, ShuffleDriverComponent should use the mapTrackerMaster to decide weather support reliable storage by the specified executorId
Bug: CSV/JSON schema inference when timestamps do not match specified timestampFormat with only one row on each partition report error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/23 04:34
Affects Version/s: 3.3.0

CSV/JSON schema inference when timestamps do not match specified timestampFormat with `only one row on each partition` report error.
{code:java}
//eg
val csv = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss")
  .option("inferSchema", true).csv(Seq("2884-06-24T02:45:51.138").toDS())
csv.show() {code}
{code:java}
//error
Caused by: java.time.format.DateTimeParseException: Text '2884-06-24T02:45:51.138' could not be parsed, unparsed text found at index 19 {code}
This bug affect 3.3/3.4/3.5. Unlike https://issues.apache.org/jira/browse/SPARK-45424 , this is a different bug but has the same error message
Bug: Document that lazy checkpoint may not be a consistent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/23 10:07
Affects Version/s: 4.0.0

Some people may want to use checkpoint to get a consistent snapshot of the Dataset / RDD. Warn that this is not the case with lazy checkpoint, because checkpoint is computed only at the end of the first action, and the data used during the first action may be different because of non-determinism and retries.
Bug: Make Python data source registration session level
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/23 00:25
Affects Version/s: 4.0.0

Currently, registered data sources are stored in `sharedState` and can be accessed across multiple sessions. This, however, will not work with Spark Connect. We should make this registration session level, and support static registration (e.g. using pip install) in the future.
Bug: Support spark.read.schema(...) for Python data source API
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Oct/23 21:31
Affects Version/s: 4.0.0

Support `spark.read.schema(...)` for Python data source read.

Add test cases where we send the schema as a string instead of StructType, and a positive case as well as a negative case where it doesn't parse successfully with fromDDL?
Bug: Unify the prepare framework for `OffsetWindowFunctionFrame`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/23 11:35
Affects Version/s: 4.0.0

Currently, the implementation the `prepare` of  all the `OffsetWindowFunctionFrame` have the same code logic show below.
```
  override def prepare(rows: ExternalAppendOnlyUnsafeRowArray): Unit = {
    if (offset > rows.length) {
      fillDefaultValue(EmptyRow)
    } else {
      resetStates(rows)
      if (ignoreNulls) {
        ...
      } else {
        ...
      }
    }
  }
```
Bug: Use enhanced `switch` expressions to replace the regular `switch` statement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 09:03
Affects Version/s: 4.0.0

refer to [JEP 361|https://openjdk.org/jeps/361] 

 

Example:
{code:java}
double getPrice(String fruit) {
  switch (fruit) {
    case "Apple":
      return 1.0;
    case "Orange":
      return 1.5;
    case "Mango":
      return 2.0;
    default:
      throw new IllegalArgumentException();
   }
 } {code}

Can be changed to 
{code:java}
double getPrice(String fruit) {
    return switch (fruit) {
      case "Apple" -> 1.0;
      case "Orange" -> 1.5;
      case "Mango" -> 2.0;
      default -> throw new IllegalArgumentException();
    };
} {code}
Bug: Fix `Passing an explicit array value to a Scala varargs method is deprecated`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:22
Affects Version/s: 4.0.0

Passing an explicit array value to a Scala varargs method is deprecated (since 2.13.0) and will result in a defensive copy; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call

 
{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala:945:21: Passing an explicit array value to a Scala varargs method is deprecated (since 2.13.0) and will result in a defensive copy; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.hive.execution.AggregationQuerySuite, version=2.13.0
[warn]         df.agg(udaf(allColumns: _*)),
[warn]                     ^
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:156:48: Passing an explicit array value to a Scala varargs method is deprecated (since 2.13.0) and will result in a defensive copy; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite, version=2.13.0
[warn]         df.agg(aggFunctions.head, aggFunctions.tail: _*),
[warn]                                                ^
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:161:76: Passing an explicit array value to a Scala varargs method is deprecated (since 2.13.0) and will result in a defensive copy; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite, version=2.13.0
[warn]         df.groupBy($"id" % 4 as "mod").agg(aggFunctions.head, aggFunctions.tail: _*),
[warn]                                                                            ^
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala:171:50: Passing an explicit array value to a Scala varargs method is deprecated (since 2.13.0) and will result in a defensive copy; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.hive.execution.ObjectHashAggregateSuite, version=2.13.0
[warn]           df.agg(aggFunctions.head, aggFunctions.tail: _*),
[warn]  {code}
Bug: Clean up the deprecated API usage related to `SeqOps`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:15
Affects Version/s: 4.0.0

* method transform in trait SeqOps is deprecated (since 2.13.0)
 * method reverseMap in trait SeqOps is deprecated (since 2.13.0)
 * method union in trait SeqOps is deprecated (since 2.13.0)

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:675:15: method transform in trait SeqOps is deprecated (since 2.13.0): Use `mapInPlace` on an `IndexedSeq` instead
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.ml.classification.LogisticRegression.train.$anonfun, origin=scala.collection.mutable.SeqOps.transform, version=2.13.0
[warn]       centers.transform(_ / numCoefficientSets)
[warn]               ^ {code}
Bug: Clean up the deprecated API usage related to `SetOps`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 03:32
Affects Version/s: 4.0.0

* method - in trait SetOps is deprecated (since 2.13.0)
 * method -- in trait SetOps is deprecated (since 2.13.0)
 * method + in trait SetOps is deprecated (since 2.13.0)
 * method retain in trait SetOps is deprecated (since 2.13.0)

 
{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/core/src/main/scala/org/apache/spark/storage/BlockReplicationPolicy.scala:70:32: method + in trait SetOps is deprecated (since 2.13.0): Consider requiring an immutable Set or fall back to Set.union
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.storage.BlockReplicationUtils.getSampleIds.indices.$anonfun, origin=scala.collection.SetOps.+, version=2.13.0
[warn]       if (set.contains(t)) set + i else set + t
[warn]                                ^ {code}
Bug: Replace `s.c.MapOps.filterKeys` with `s.c.MapOps.view.filterKeys`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/23 03:47
Affects Version/s: 4.0.0

{code:java}
/** Filters this map by retaining only keys satisfying a predicate.
  *  @param  p   the predicate used to test keys
  *  @return an immutable map consisting only of those key value pairs of this map where the key satisfies
  *          the predicate `p`. The resulting map wraps the original map without copying any elements.
  */
@deprecated("Use .view.filterKeys(f). A future version will include a strict version of this method (for now, .view.filterKeys(p).toMap).", "2.13.0")
def filterKeys(p: K => Boolean): MapView[K, V] = new MapView.FilterKeys(this, p) {code}
Bug: Fix `Implicit definition should have explicit type`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/23 05:13
Affects Version/s: 4.0.0

{code:java}
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:343:16: Implicit definition should have explicit type (inferred org.json4s.DefaultFormats.type) [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-implicit-type, site=org.apache.spark.deploy.TestMasterInfo.formats
[error]   implicit val formats = org.json4s.DefaultFormats
[error]   {code}
Bug: Correctness bug in correlated scalar subqueries with COUNT aggregates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 00:41
Affects Version/s: 4.0.0

{code:java}
 
create view if not exists t1(a1, a2) as values (0, 1), (1, 2);
create view if not exists t2(b1, b2) as values (0, 2), (0, 3);
create view if not exists t3(c1, c2) as values (0, 2), (0, 3);

-- Example 1
select (
  select SUM(l.cnt + r.cnt)
  from (select count(*) cnt from t2 where t1.a1 = t2.b1 having cnt = 0) l
  join (select count(*) cnt from t3 where t1.a1 = t3.c1 having cnt = 0) r
  on l.cnt = r.cnt
) from t1

-- Correct answer: (null, 0)
+----------------------+
|scalarsubquery(c1, c1)|
+----------------------+
|null                  |
|null                  |
+----------------------+

-- Example 2
select ( select sum(cnt) from (select count(*) cnt from t2 where t1.c1 = t2.c1) ) from t1

-- Correct answer: (2, 0)
+------------------+
|scalarsubquery(c1)|
+------------------+
|2                 |
|null              |
+------------------+

-- Example 3
select ( select count(*) from (select count(*) cnt from t2 where t1.c1 = t2.c1) ) from t1

-- Correct answer: (1, 1)
+------------------+
|scalarsubquery(c1)|
+------------------+
|1                 |
|0                 |
+------------------+ {code}
 

 

DB fiddle for correctness check:[https://www.db-fiddle.com/f/4jyoMCicNSZpjMt4jFYoz5/10403#]
Bug: Fix `Unicode escapes in triple quoted strings are deprecated`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:59
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala:1686:44: Unicode escapes in triple quoted strings are deprecated; use the literal character instead
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, version=2.13.2
[warn]         |  COLLECTION ITEMS TERMINATED BY '\u0002'
[warn]                                            ^ {code}
Bug: Fix `method tryCompleteWith in trait Promise is deprecated`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:36
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/core/src/main/scala/org/apache/spark/FutureAction.scala:190:32: method tryCompleteWith in trait Promise is deprecated (since 2.13.0): Since this method is semantically equivalent to `completeWith`, use that instead.
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.ComplexFutureAction.p, origin=scala.concurrent.Promise.tryCompleteWith, version=2.13.0
[warn]   private val p = Promise[T]().tryCompleteWith(run(jobSubmitter))
[warn]                                ^ {code}
Bug: SPIP: State Data Source - Reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 06:48
Affects Version/s: 4.0.0

State Store has been a black box from the introduction of the stateful operator. It has been the “internal” data to the streaming query, and Spark does not expose the data outside of the streaming query. There is no feature/tool for users to read and modify the content of state stores.

Specific to the ability to read the state, the lack of feature brings up various limitations like following:
 * Users are unable to see the content in the state store, leading to inability to debug.
 * Users have to perform some indirect approach on verifying the content of the state store in unit tests. The only option they can take is relying on the output of the query.

Given that, we propose to introduce a feature which enables users to read the state from the outside of the streaming query.

SPIP: [https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing]
Bug: Fix   "method + in class Byte/Short/Char/Long/Double/Int is deprecated"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:08
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala:127:42: method + in class Int is deprecated (since 2.13.0): Adding a number and a String is deprecated. Use the string interpolation `s"$num$str"`
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.rdd.PipedRDDSuite, origin=scala.Int.+, version=2.13.0
[warn]       (i: Int, f: String => Unit) => f(i + "_")) {code}
Bug: Support session window aggregation in state reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/23 23:44
Affects Version/s: 3.5.0

We are introducing state reader in SPARK-45511, but currently session window operator is not supported because the numColPrefixKey is unknown. We can read the operator state metadata introduced in SPARK-45558 to determine the number of prefix columns and load the state of session window correctly.
Bug: XML: Make 'rowTag' a required option
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/23 22:17
Affects Version/s: 4.0.0

User can specify `rowTag` option that is the name of the XML element that maps to a `DataFrame Row`.  A non-existent `rowTag` will not infer any schema or generate any `DataFrame` rows. Currently, not specifying `rowTag` option results in picking up its default value of  `ROW`. This results in an empty dataframe and confuse new users. Creating a Jira task to make this a required option.
Bug: Upgrade AWS SDK to v2 for Kubernetes integration tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/23 09:25
Affects Version/s: 4.0.0

Sub-task of [SPARK-44124|https://issues.apache.org/jira/browse/SPARK-44124]. In this issue, we will upgrade AWS SDK in Credentials providers, AWS clients and related Kubernetes integration tests.
Bug: Update partition statistics with ANALYZE TABLE command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/23 23:28
Affects Version/s: 4.0.0

Currently {{ANALYZE TABLE}} command only updates table-level stats but not partition stats, even though it can be applied to both non-partitioned and partitioned tables. It seems make sense for it to update partition stats as well.

Note users can use {{ANALYZE TABLE PARTITION(..)}} to get the same effect, but the syntax is more verbose as they need to specify all the partition columns.
Bug: Enhance PySpark testing utils
Status: Reopened | Priority: Major | Resolution: 
Created: 16/Oct/23 11:26
Affects Version/s: 4.0.0

We have added several testing utils for PySpark testing: [https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.testing.assertDataFrameEqual.html.]

We might need to enhance the functionality of those functions to make testing easier for users. The existing testing utils could be enhanced by:
 * Adding new parameters to the existing utils
 * Returning a debuggable object for failed assertions

and any other suggestions are welcome.
Bug: Support loading Python data sources in DataFrameReader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/23 20:53
Affects Version/s: 4.0.0

Allow users to read from a Python data source using `spark.read.format(...).load()` in PySpark. For example

Users can extend the DataSource and the DataSourceReader classes to create their own Python data source reader and use them in PySpark:
{code:java}
class MyReader(DataSourceReader):
    def read(self, partition):
        yield (0, 1)

class MyDataSource(DataSource):
    def schema(self):
        return "id INT, value INT"
    
    def reader(self, schema):
        return MyReader()

df = spark.read.format("MyDataSource").load()
df.show()
+---+-----+
| id|value|
+---+-----+
|  0|    1|
+---+-----+
{code}
Bug: current_date() not supported in Streaming Query Observed metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/23 20:48
Affects Version/s: 3.4.1

Streaming queries do not support current_date() inside CollectMetrics. The primary reason is that current_date() (resolves to CurrentBatchTimestamp) is marked as non-deterministic. However, {{current_date}} and {{current_timestamp}} are both deterministic today, and {{current_batch_timestamp}} should be the same.

 

As an example, the query below fails due to observe call on the DataFrame.

 
{quote}val inputData = MemoryStream[Timestamp]

inputData.toDF()
      .filter("value < current_date()")
      .observe("metrics", count(expr("value >= current_date()")).alias("dropped"))
      .writeStream
      .queryName("ts_metrics_test")
      .format("memory")
      .outputMode("append")
      .start()
{quote}
Bug: Implement an option similar to corrupt record column in State Data Source Reader
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 05:43
Affects Version/s: 4.0.0

Querying against the state would be most likely failing if the underlying state file is corrupted. There may be another case that the binary data (raw) state store read from state file does not fit with state schema and ends up with exception/fatal error in runtime.

(We can't catch the case where the data is loaded with incorrect schema if it does not throw an exception. We cannot add the schema for every data.)

To handle above cases without failure, we want to provide state rows for valid rows, with also providing binary data for corrupted rows (like we do for CSV/JSON) if users specify an option.
Bug: Provide a unified user-facing schema for state format versions in state data source - reader
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 05:56
Affects Version/s: 4.0.0

As of now, except stream-stream join with joinSide option being specified, state data source would provide the state "as it is" in the state store. This means state data source will provide the different schema for operators having multiple state format versions.

From users' perspective, they do not care about the state format version, hence may be confused if the state data source produces different schema.

That said, we could probably consider defining and providing the same user facing schema for each operator.

*Note that this would need further discussion* before coming up with code, because there is a clear trade-off. It makes a strong coupling between state data source and the implementation of stateful operators. Also, for the argument of non-predictable output schema, users can call printSchema() to see the output schema in prior to query.
Bug: Investigate the behavior difference in self-join
Status: Open | Priority: Major | Resolution: 
Created: 12/Oct/23 01:05
Affects Version/s: 3.5.0

SPARK-45220 discovers a behavior difference for a self-join scenario between classic Spark and Spark Connect.

For instance, here is the query that works without Spark Connect: 
{code:java}
df = spark.createDataFrame([Row(name="Alice", age=2), Row(name="Bob", age=5)])
df2 = spark.createDataFrame([Row(name="Tom", height=80), Row(name="Bob", height=85)]){code}
{code:java}
joined = df.join(df2, df.name == df2.name, "outer").sort(sf.desc(df.name)) 
joined.show(){code}
But in Spark Connect, it throws this exception:
{code:java}
pyspark.errors.exceptions.connect.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `name` cannot be resolved. Did you mean one of the following? [`name`, `name`, `age`, `height`].;
'Sort ['name DESC NULLS LAST], true
+- Join FullOuter, (name#64 = name#78)
   :- LocalRelation [name#64, age#65L]
   +- LocalRelation [name#78, height#79L]
 {code}
 

On the other hand, this query failed in classic Spark Connect:
{code:java}
df.join(df, df.name == df.name, "outer").select(df.name).show() {code}
{code:java}
pyspark.errors.exceptions.captured.AnalysisException: Column name#0 are ambiguous... {code}
 

but this query works with Spark Connect.

We need to investigate the behavior difference and fix it.