av9ash/GitBugs huggingface dataset:
Bug: Document local-cluster mode option in developper tools doc
Status: In Progress | Priority: Minor | Resolution: 
Created: 29/Jul/21 02:05
Affects Version/s: 3.1.2

Local-cluster mode is often used by developers but is not documented on any testing related docs. I think It should be documented on developer tools doc (https://spark.apache.org/developer-tools.html).

Also this mode is slightly mentioned on a user reference doc though it is only for unit tests, so we should remove it to avoid confusing people.

 

Related links:

https://issues.apache.org/jira/browse/SPARK-595

[https://github.com/apache/spark/pull/33537]
Bug: Replace revertPartialWritesAndClose with close in ExternalSorter.spill and ExternalAppendOnlyMap.spill
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Jul/21 07:26
Affects Version/s: 3.3.0

ExternalAppendOnlyMap.spill method call  `revertPartialWritesAndClose` method when `objectsWritten == 0` as follows:
{code:java}
try {
  while (inMemoryIterator.hasNext) {
    ...
    if (objectsWritten == serializerBatchSize) {
      flush()
    }
  }
  if (objectsWritten > 0) {
    flush()
    writer.close()
  } else {
    writer.revertPartialWritesAndClose()
  }
  success = true
} finally {
  ...
}{code}
 

writer.revertPartialWritesAndClose() can replace with writer.close to reduce a set of file operations includes open, truncate and close.

 

A similar situation exists for ExternalSorter
Bug: Ensure spill file closed before set success to true in ExternalSorter.spillMemoryIteratorToDisk method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jul/21 09:38
Affects Version/s: 3.0.3

The processes of ExternalSorter.spillMemoryIteratorToDisk and ExternalAppendOnlyMap.spillMemoryIteratorToDisk are similar, but there are some differences in setting `success = true`

 

Code of ExternalSorter.spillMemoryIteratorToDisk as follows:

 
{code:java}
      if (objectsWritten > 0) {
        flush()
      } else {
        writer.revertPartialWritesAndClose()
      }
      success = true
    } finally {
      if (success) {
        writer.close()
      } else {
        ...
      }
    }{code}
Code of ExternalSorter.spillMemoryIteratorToDisk as follows:
{code:java}
  if (objectsWritten > 0) {
    flush()
    writer.close()
  } else {
    writer.revertPartialWritesAndClose()
  }
  success = true
} finally {
  if (!success) {
    ...
  }
}{code}
It seems that the processing of `ExternalSorter.spillMemoryIteratorToDisk` mehod is more reasonable, We should make sure setting `success = true` after the spill file is closed
Bug: type check fails for `F.udf(...).asNonDeterministic()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jul/21 14:48
Affects Version/s: 3.1.2

The following code should type-check, but doesn't:
----

{{import uuid}}

{{pyspark.sql.functions as F}}

{{my_udf = F.udf(lambda: str(uuid.uuid4())).asNondeterministic()}}

----
In {{python/pyspark/sql/functions.pyi}} the {{udf}} signature is wrong
Bug: rewrite distinct count case when expressions without Expand node
Status: In Progress | Priority: Minor | Resolution: 
Created: 26/Jul/21 12:51
Affects Version/s: 3.2.0

Currently, RewriteDistinctAggregates rule will rewrite distinct aggregates with extra Expand node. This causes unnecessary memory waste and performance fallback for some specific aggregates(e.g. count distinct case when).

We can rewrite count distinct case when aggregates without Expand:

for query:

* {{{
 * SELECT
 * cat1,
 * COUNT(DISTINCT CASE WHEN cond1 THEN cat2 ELSE null end) as cat2_cnt1,
 * COUNT(DISTINCT CASE WHEN cond2 THEN cat2 ELSE null end) as cat2_cnt2,
 * FROM
 * data
 * GROUP BY
 * key
 * }}}

 

we should rewrite to :

* {{{
 * Aggregate(
 * key = ['key]
 * functions = [count(if (01 & 'bit_vector != 0) 0 else null),
 * count(if (10 & 'bit_vector != 0) 0 else null)]
 * output = ['key, 'cat2_cnt1, 'cat2_cnt2])
 * Aggregate(
 * key = ['key, 'cat1]
 * functions = [bit_or(if (cond1) 01 else 00, if (cond2) 10 else 00)]
 * output = ['key, 'cat1, 'bit_vector])
 * LocalTableScan [...]
 * }}}

 

This method will improve performance and reduce memory waste
Bug: Fix only set data columns to Hive column names config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jul/21 04:12
Affects Version/s: 3.3.0

When reading Hive table, we set the Hive column id and column name configs (`hive.io.file.readcolumn.ids` and `hive.io.file.readcolumn.names`). We should set non-partition columns (data columns) for both configs, as Spark always appends partition columns in its own reader - [https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L240] . The column id config has only non-partition columns, but column name config has both partition and non-partition columns. We should keep them to be consistent with only non-partition columns. This does not cause issue for public OSS Hive file format, but for customized internal Hive file format, it causes the issue as we are expecting these two configs to be same.
Bug: https://spark.apache.org/docs/latest/sql-programming-guide.html contains invalid link to Python doc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jul/21 13:09
Affects Version/s: 3.1.2

On https://spark.apache.org/docs/latest/sql-programming-guide.html , the link to the python doc points to https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame which returns a "Not found"
Bug: Upgrade zstd-jni to 1.5.0-3 to avoid a bug about buffer size calculation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jul/21 14:19
Affects Version/s: 3.2.0

zstd-jni 1.5.0-3 was released few days ago.
This release resolves an issue about buffer size calculation, which can affect usage in Spark.
https://github.com/luben/zstd-jni/releases/tag/v1.5.0-3
Bug: Consider mapper location and shuffle block size in OptimizeLocalShuffleReader
Status: Open | Priority: Minor | Resolution: 
Created: 20/Jul/21 22:30
Affects Version/s: 3.1.2

This is a follow-up to SPARK-36105 (OptimizeLocalShuffleReader support reading data of multiple mappers in one task). We should consider using the mapper locations along with shuffle block size when coalescing mappers (specifically in events where there are more mappers than there is parallelism.
Bug: Preserve column insertion order in Dataset.withColumns
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jul/21 14:17
Affects Version/s: 3.0.3

Dataset.withColumns uses a Map (columnMap) to store the mapping of column name to column. however this loses the order of the columns. also none of the operations used on the Map (find and filter) benefits from the map's lookup features. so it seems simpler to use a Seq instead, which also preserves the insertion order.
Bug: Incorrect pyspark.sql.types.Row __new__ and __init__ type annotations
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Jul/21 03:26
Affects Version/s: 3.1.2

This bug involves incorrect type annotations for {{pyspark.sql.types.Row}}'s {{\_\_new\_\_}} and {{\_\_init\_\_}} methods when invoked without keyword arguments (_i.e._, {{\*args}} rather than {{\*\*kwargs}}).

When creating a [Row|https://hyukjin-spark.readthedocs.io/en/latest/reference/api/pyspark.sql.types.Row.html] with unnamed fields which are not of type {{str}} (_e.g._, {{row1 = Row("Alice", 11)}} appears in the {{Row}} documentation) type checkers produce an error.

The implementation doesn't assume the arguments are of type {{str}}, and in fact the documentation includes an example where non-{{str}} types are provided in this way (see [the final example here|https://hyukjin-spark.readthedocs.io/en/latest/reference/api/pyspark.sql.types.Row.html]).

An example of the type error produced by [pyright|https://github.com/microsoft/pyright] is

{code}
error: No overloads for "__init__" match the provided arguments
    Argument types: (Literal['Alice'], Literal[11]) (reportGeneralTypeIssues)
{code}
Bug: Add exception for Kafka readstream when decryption fails
Status: Open | Priority: Minor | Resolution: 
Created: 19/Jul/21 16:28
Affects Version/s: 3.0.0

A silent failure is possible when reading from a Kafka broker under the following circumstances:

SDF.isStreaming = True

SDF.printSchema() => returns expected schema

Query results are empty.

Issue: Tsl Decryption has failed, but there is no exception or warning.

Request: Add a warning throw an exception when decryption fails, such that developers can efficiently diagnose the readstream problem.

 

This is my first ticket submitted. Please notify me if I should change anything in this ticket to make it more conformant to community standards. I'm still a beginner with Spark.
Bug: Add reference to kubernetes-client's version
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/21 05:56
Affects Version/s: 3.1.2

Running Spark on Kubernetes potentially has upper limitation of Kubernetes version.

[Compatibility Matrix|https://github.com/fabric8io/kubernetes-client#kubernetes-compatibility-matrix]

e.g. Spark 3.1.2 uses kubernetes-client 4.12.0 and that does not support Kubernetes >= 1.19.1

I think it is better for users to notice it because Kubernetes update speed is so fast that users tends to run Spark Jobs on unsupported version.
Bug: Spark does not passon needClientAuth to Jetty SSLContextFactory. Does not allow to configure mTLS authentication.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/21 14:27
Affects Version/s: 2.4.7

Spark does not pass on the needClientAuth flag to Jetty engine. This prevents the UI from honouring mutual TLS authentication using x509 certificates.
Bug: TimeWindow expression: apply filter before project
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jul/21 11:31
Affects Version/s: 3.3.0

https://github.com/apache/spark/blob/4dfd266b27fea6954593c6b9e3a2819b290f0aec/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3901-L3906

In the case of tumbling window, we apply project and then filter, while filter is not dependent to project. We can just swap two operator to ensure less rows would be projected if there're some rows being filtered out.
Bug: Upgrade commons-compress to 1.21 to deal with CVEs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/21 03:14
Affects Version/s: 2.4.8

Some CVEs which affect commons-compress 1.20 are reported and fixed in 1.21.
https://commons.apache.org/proper/commons-compress/security-reports.html
Bug: Regression: Remote blocks stored on disk by BlockManager are not deleted
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/21 02:11
Affects Version/s: 3.3.0

There was a regression since Spark started storing large remote files on disk (https://issues.apache.org/jira/browse/SPARK-22062). A refactoring introduced a hidden reference preventing the auto-deletion of the files (https://github.com/apache/spark/commit/a97001d21757ae214c86371141bd78a376200f66#diff-42a673b8fa5f2b999371dc97a5de7ebd2c2ec19447353d39efb7e8ebc012fe32L1677). Since then all underlying files of DownloadFile instances are kept on disk for the duration of the Spark application which sometimes results in "no space left" errors.
Bug: Add time cost info for writing rows out and committing the task.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jul/21 08:30
Affects Version/s: 3.2.0

We have a job that has a stage that contains about 8k tasks.  Most tasks take about 1~10min to finish but 3 of them tasks run extremely slow. They take about 1 hour each to finish and also do their speculations. The root cause is most likely the delay of the storage system. On the spark side, we can record the time cost in logs for better bug hunting or performance tuning.
Bug: Unify write exception and delete abnormal disk block object file process
Status: In Progress | Priority: Minor | Resolution: 
Created: 08/Jul/21 12:03
Affects Version/s: 3.3.0

There are some duplicate codes related to cleaning up failed files after DiskBlockObjectWriter writes data abnormally in `BypassMergeSortShuffleWriter`, `ExternalAppendOnlyMap` and `ExternalSorter`, the duplicate codes as follows:
{code:java}
        writer.revertPartialWritesAndClose()
        if (file.exists()) {
          if (!file.delete()) {
            logWarning(s"Error deleting ${file}")
          }
        }
{code}
Bug: Missing GraphX classes in registerKryoClasses util method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jul/21 21:20
Affects Version/s: 3.1.2

{{VertexRDDImpl}} uses an {{RDD[ShippableVertexPartition[VD]]}} however, {{GraphXUtils.registerKryoClasses}} does not register {{ShippableVertexPartition}}. This means when running with {{spark.kryo.registrationRequired}} set to {{true}}, we get a "Class is not registered" exception. This is an issue as it prevents other unregistered classes from being discovered using {{spark.kryo.registrationRequired}} as the first unregistered class found halts the whole job. It also potentially decreases the serialised size of the RDD when using Kryo.
Bug: POM errors
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 03/Jul/21 15:27
Affects Version/s: 3.1.1

Usage is ZincServer within scala-maven-plugin is not allowed.

<useZincServer>true</useZincServer> -> results in error

 

Similar issue within mvn_scalafmt_2.12

<parameters>${scalafmt.parameters}</parameters> <!-- (Optional) Additional command line arguments -->
<skip>${scalafmt.skip}</skip> <!-- (Optional) skip formatting -->

 

Should these be removed?
Bug: Switch the datasource example due to the depreciation of the dataset
Status: Open | Priority: Trivial | Resolution: 
Created: 06/Jul/21 05:01
Affects Version/s: 3.1.2

The S3 bucket that used for an example in "Integration with Cloud Infrastructures" document will be deleted on Jul 1, 2021 [https://registry.opendata.aws/landsat-8/ |https://registry.opendata.aws/landsat-8/]

The dataset will move to another bucket but it requires `--request-payer requester` option so users have to pay S3 cost. [https://registry.opendata.aws/usgs-landsat/]

 

So I think it's better to change the datasource like this.

[https://github.com/yoda-mon/spark/commit/cdb24acdbb57a429e5bf1729502653b91a600022]

 

I chose [NYC Taxi data| [https://registry.opendata.aws/nyc-tlc-trip-records-pds/|https://registry.opendata.aws/nyc-tlc-trip-records-pds/),]] here for an example. 
 Unlike landat data it's not compressed, but it is just an example and there are several tutorials using Spark  (e.g. [https://github.com/aws-samples/amazon-eks-apache-spark-etl-sample)]

 

Reed test result
{code:java}
scala> sc.textFile("s3a://nyc-tlc/misc/taxi _zone_lookup.csv").take(10).foreach(println) "LocationID","Borough","Zone","service_zone" 1,"EWR","Newark Airport","EWR" 2,"Queens","Jamaica Bay","Boro Zone" 3,"Bronx","Allerton/Pelham Gardens","Boro Zone" 4,"Manhattan","Alphabet City","Yellow Zone" 5,"Staten Island","Arden Heights","Boro Zone" 6,"Staten Island","Arrochar/Fort Wadsworth","Boro Zone" 7,"Queens","Astoria","Boro Zone" 8,"Queens","Astoria Park","Boro Zone" 9,"Queens","Auburndale","Boro Zone"
{code}
Bug: Remove automatic update of documentation build in the guides
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Jul/21 10:09
Affects Version/s: 3.2.0

At https://github.com/apache/spark/tree/master/docs#automatically-rebuilding-api-docs, 

{code}
cd "$SPARK_HOME/python/docs"
find .. -type f -name '*.py' \
| entr -s 'make html && cp -r _build/html/. ../../docs/api/python'
{code}

is documented. However, this doesn't work very well:

1. It doesn't detect the changes in .rst files. But PySpark internally generates rst files so we can't just simply include it in the detection. Otherwise, it goes to an infinite loop
2. During PySpark documentation generation, it launches some jobs to generate plot images now. This is broken with {{entr}} command, and the job fails. Seems like it's related to how {{entr}} creates the process internally.
Bug: Add documentation for ORC nested column vectorized reader
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Jul/21 07:22
Affects Version/s: 3.2.0

In https://issues.apache.org/jira/browse/SPARK-34862, we added support for ORC nested column vectorized reader, and it is disabled by default for now. So we would like to add the user-facing documentation for it, and user can opt-in to use it if they want.
Bug: Use Map.computeIfAbsent to simplify the process of HeapMemoryAllocator.bufferPoolsBySize init new item
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 28/Jul/21 08:45
Affects Version/s: 3.3.0

HeapMemoryAllocator initializes the new element ofbufferPoolsBySizein the following way：
{code:java}
synchronized (this) {
  LinkedList<WeakReference<long[]>> pool = bufferPoolsBySize.get(alignedSize);
  if (pool == null) {
    pool = new LinkedList<>();
    bufferPoolsBySize.put(alignedSize, pool);
  }
  pool.add(new WeakReference<>(array));
}
{code}
The codes can be simplified by Map.computeIfAbsent
Bug: Add support for running make-distribution without a "clean"
Status: Open | Priority: Trivial | Resolution: 
Created: 22/Jul/21 00:04
Affects Version/s: 3.2.0

Running the K8s integration tests requires* building a distribution, but clean builds are really really slow. We could make the param BUILD_COMMAND set if unset or add a --skip-clean to our shell script to allow for folks to more quickly test there k8s related changes.
Bug: Add logging for slow fetches to diagnose external shuffle service issues
Status: In Progress | Priority: Trivial | Resolution: 
Created: 20/Jul/21 01:43
Affects Version/s: 3.2.0

Currently we can see from the metrics that a task or stage has slow fetches, and the logs indicate _all_ of the shuffle servers those tasks were fetching from, but often this is a big set (dozens or even hundreds) and narrowing down which one caused issues can be very difficult. We should add some logging when a fetch is "slow" as determined by some preconfigured thresholds.
Bug: Replace the handwriting compare methods with static compare methods in Java code
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Jul/21 09:10
Affects Version/s: 3.3.0

There are some handwriting compare methods like `ShuffleInMemorySorter.SortComparator`
{code:java}
private static final class SortComparator implements Comparator<PackedRecordPointer> {
  @Override
  public int compare(PackedRecordPointer left, PackedRecordPointer right) {
    int leftId = left.getPartitionId();
    int rightId = right.getPartitionId();
    return Integer.compare(leftId, rightId);
  }
}
{code}
the handwriting compare methods can replace with `Integer.compare()` method and similar methods after Java 1.7
Bug: Reduce the run time of DateExpressionsSuite
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 06/Jul/21 09:22
Affects Version/s: 3.2.0

Some of the test cases in `DateExpressionsSuite` are quite slow:
* Hour: 24s
* Minute: 26s
* Day / DayOfMonth: 8s
* SPARK-34761,SPARK-35889: add a day-time interval to a timestamp: 8s
* Year: 4s

Each test case has a large loop. We should improve them.
Bug: Different Behaviour join vs joinWith in self joining
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 04/Jun/21 14:19
Affects Version/s: 3.1.2

It seems like spark inner join is performing a cartesian join in self joining using `joinWith` and an inner join using `join` 

Snippet:

 
{code:java}
scala> val df = spark.range(0,5) 
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> df.show 
+---+ 
| id| 
+---+ 
| 0|
| 1| 
| 2| 
| 3| 
| 4| 
+---+ 

scala> df.join(df, df("id") === df("id")).count 
21/06/04 16:01:39 WARN Column: Constructing trivially true equals predicate, 'id#1649L = id#1649L'. Perhaps you need to use aliases. 
res21: Long = 5

scala> df.joinWith(df, df("id") === df("id")).count
21/06/04 16:01:47 WARN Column: Constructing trivially true equals predicate, 'id#1649L = id#1649L'. Perhaps you need to use aliases. 
res22: Long = 25

{code}
According to the comment in code source, joinWith is expected to manage this case, right?
{code:java}
def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)] = {
    // Creates a Join node and resolve it first, to get join condition resolved, self-join resolved,
    // etc.
{code}
I find it weird that join and joinWith haven't the same behaviour.
Bug: Relax constraint for Spark bucket join and remove HashClusteredDistribution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 19:23
Affects Version/s: 3.2.0

Currently Spark has {{HashClusteredDistribution}} and {{ClusteredDistribution}}. The only difference between the two is that the former is more strict when deciding whether bucket join is allowed to avoid shuffle: comparing to the latter, it requires *exact* match between the clustering keys from the output partitioning (i.e., {{HashPartitioning}}) and the join keys. However, this is unnecessary, as we should be able to avoid shuffle when the set of clustering keys is a subset of join keys, just like {{ClusteredDistribution}}.
Bug: The Statistics size of PARQUET table is not estimated correctly
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/21 12:34
Affects Version/s: 3.1.2

{code:java}
// Table 'test_txt' and 'test_parquet' have the same data.
scala> val sql="select * from tmp_db.test_txt"
sql: String = select * from tmp_db.test_txt
scala> spark.sql(sql).queryExecution.optimizedPlan.stats.sizeInBytes
res5: BigInt = 92990
scala> val sql = "select * from tmp_db.test_parquet"
sql: String = select * from tmp_db.test_parquet
scala> spark.sql(sql).queryExecution.optimizedPlan.stats.sizeInBytes
res6: BigInt = 37556
{code}
PARQUET file is compressed by default, this could lead to choose the wrong type of JOIN, like BROADCASTJOIN. Driver may be OOM in this case, because the actual size may be much greater than the AUTO_BROADCASTJOIN_THRESHOLD.  Can we improve this?
Bug: add fs.s3a.endpoint if unset and fs.s3a.endpoint.region is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 12:45
Affects Version/s: 3.2.0

People working with S3A and hadoop 3.3.1 outside of EC2, and without the AWS CLI setup, are likely to hit: HADOOP-17771

It should be straightforward to fix up the config similar to SPARK-35868...this will be backwards (harmless) and forwards compatible (it's the recommended workaround)
Bug: Respect Py4J server if InheritableThread API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 03:46
Affects Version/s: 3.2.0

Currently we sets the enviornment variables at the client side of Py4J (python/pyspark/util.py ). If the Py4J gateway is created somewhere else (e.g., Zeppelin, etc), it could introduce a breakage at:

{code}
        from pyspark import SparkContext
        jvm = SparkContext._jvm
        thread_connection = jvm._gateway_client.get_thread_connection()
        # ^ the MLlibMLflowIntegrationSuite test suite failed at this line
        # `AttributeError: 'GatewayClient' object has no attribute 'get_thread_connection'`
{code}