av9ash/GitBugs huggingface dataset:
Bug: Duplicates after task failure in dropDuplicates and repartition
Status: Open | Priority: Major | Resolution: 
Created: 09/Feb/22 12:36
Affects Version/s: 3.0.2

We're seeing duplicates after running the following
{code:java}
def compute_shipments(shipments):
    shipments = shipments.dropDuplicates(["ship_trck_num"])
    shipments = shipments.repartition(4)
    return shipments
{code}
and observing lost executors (OOMs) and task retries in the repartition stage.

We're seeing this reliably in one of our pipelines. But I haven't managed to reproduce outside of that pipeline. I'll attach driver logs - maybe you have ideas.
Bug: Remove option newlines.topLevelStatements in scalafmt.conf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 04:33
Affects Version/s: 3.3.0

The configuration

newlines.topLevelStatements = [before,after]

is to add blank line before the first member or after the last member of the class.

This is neither encouraged nor discouraged as per https://github.com/databricks/scala-style-guide#blanklines
Bug: Support CREATE EXTERNAL TABLE LIKE syntax
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Feb/22 05:41
Affects Version/s: 3.2.1

Spark already has the syntax of `CREATE TABLE LIKE`. It's intuitive for users to say `CREATE EXTERNAL TABLE a LIKE b LOCATION 'path'`. However this syntax is not supported in Spark right now and we should make these CREATE TABLE DDLs consistent.
Bug: Python UDFs freeze when libraries with non-standard imports are imported in the namespace
Status: Open | Priority: Major | Resolution: 
Created: 08/Feb/22 21:05
Affects Version/s: 3.1.2

When importing libraries with nontraditional imports (e.g. an `__init__.py` that doesn't just explicitly import its submodules, but instead uses a function or a custom loader) PySpark UDFs will freeze.

Two libraries that demonstrate this behavior:
 * [snscrape|https://github.com/JustAnotherArchivist/snscrape/blob/master/snscrape/modules/__init__.py]
 * [transformers|https://github.com/huggingface/transformers/blob/master/src/transformers/__init__.py]

The attached `example.zip` archive contains a module that demonstrates this behavior. To reproduce:

```bash
$ pip install transformers snscrape
$ unzip example.zip
$ cd example
$ pip install --editable .
$ example
```

This example module uses relative imports everywhere, with no circular imports.

To demonstrate a fix to this issue, simply erase `import_stuff.py`:

```bash
$ echo "print('erased')" > example/import_stuff.py
```

Then rerun the example:

```bash
$ example
```

My hypothesis is that some libraries take a long time to import. transformers can take up to 4 seconds to import, and snscrape up to 1 second. I think this import performance is causing Spark execution of our UDFs to just slow down to an unusable level.

*Note:* For some reason, changing the import behavior fixes the Spark performance issue, even though the import performance itself doesn't change. 

So, it's some combination of "library that takes a long time to import" and "library with weird imports" that breaks Spark UDFs. I have no clue what Spark internals cause this to be true, but I can demonstrate it with the `snscrape` library with a small modification to their import behavior [demonstrated here|https://github.com/michaelcredera/snscrape/blob/feature/update-imports/snscrape/modules/__init__.py]. This fork of `snscrape` has no associated UDF performance issues, despite the time to import the library being constant.

This is something we can avoid in our code by deferring the import of these "weird" libraries, but seems like odd behavior that I'd like to understand better. If anyone can help me get some insight into why this might be the case in Spark/PySpark, I would greatly appreciate it!
Bug: Repartition+Shuffle+ non deterministic function leads to bad results
Status: Open | Priority: Major | Resolution: 
Created: 08/Feb/22 09:17
Affects Version/s: 3.1.1

Hi,

The results when using a non deterministic function in repartition (like rand) leads into incorrect results.

Reproduce: (correct)

 
{code:java}
// code placeholder
import scala.sys.process._
import org.apache.spark.TaskContext
import org.apache.spark.sql.functions.rand

val res = spark.range(0, 100 * 100, 1).repartition(200).map { x =>
  x
}.repartition(200).map { x =>
  if (TaskContext.get.attemptNumber == 0 && TaskContext.get.partitionId < 2) {
    throw new Exception("pkill -f java".!!)
  }
  x
}
res.distinct().count() {code}
The correct result 10000 

Reproduce: (bad)

 
{code:java}
// code placeholder
import scala.sys.process._
import org.apache.spark.TaskContext
import org.apache.spark.sql.functions.rand

val res = spark.range(0, 100 * 100, 1).repartition(200).map { x =>
  x
}.repartition(10, Array(rand):_*).map { x =>
  if (TaskContext.get.attemptNumber == 0 && TaskContext.get.partitionId < 2) {
    throw new Exception("pkill -f java".!!)
  }
  x
}
res.distinct().count() {code}
The bad result 9396
Bug: Remove NotPropagation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/22 20:11
Affects Version/s: 3.3.0

To mitigate the bug introduced by SPARK-36665. Remove {{NotPropagation}} optimization for now until we find a better approach.

{{NotPropagation}} optimization previously broke {{RewritePredicateSubquery}} so that it does not properly rewrite the predicate to a NULL-aware left anti join anymore.
Bug: Adaptively enable timeout for BroadcastQueryStageExec
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Feb/22 10:50
Affects Version/s: 3.2.0

We should disable timeout for BroadcastQueryStageExec when it comes from shuffle query stages which runtime statistics are usually correct in AQE, but should enable timeout for it when it comes from others which statistics may be incorrect, and keep it the same as non-AQE.
Bug: Query using an aggregation on a literal value with an empty underlying dataframe returns null
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Feb/22 15:29
Affects Version/s: 3.2.0

Running a query with an aggregation functions such as average on literal value input with an empty dataframe in the FROM clause causes Spark to return null.

Minimal reproducible example using Spark 3.2.0 with Java 11:

 
{code:java}
sparkSession.emptyDataFrame().createOrReplaceTempView("empty");
StructType someSchema = new StructType(new StructField[]{DataTypes.createStructField("a", DataTypes.StringType, false)});
final Row aRow = Row.fromSeq(asScalaBuffer(List.of("a")));
sparkSession.createDataFrame(List.of(aRow), someSchema).createOrReplaceTempView("non_empty");
sparkSession.sql("SELECT avg(1)").show(); // standalone query works
sparkSession.sql("SELECT avg(1) FROM empty").show(); // empty DF gives null
sparkSession.sql("SELECT avg(1) FROM non_empty").show(); // It does work with any non-empty DF{code}
Output is as follows:
{noformat}
+------+
|avg(1)|
+------+
|   1.0|
+------+
+------+
|avg(1)|
+------+
|  null|
+------+
+------+
|avg(1)|
+------+
|   1.0|
+------+
{noformat}
I would expect that the second query also returns 1.0. It seems that any non-empty DataFrame returns 1.0. 

 

Out of curiosity: is this Spark Catalyst doing some empty DataFrame optimizations that affect the result?
Bug: Use error classes in the parsing errors of joins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 15:56
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* joinCriteriaUnimplementedError
* naturalCrossJoinUnsupportedError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Executor metrics are missing on prometheus sink
Status: Open | Priority: Major | Resolution: 
Created: 05/Feb/22 14:22
Affects Version/s: 3.1.2

Setting up prometheus sink in this way:
{code:java}
-c spark.ui.prometheus.enabled=true
-c spark.executor.processTreeMetrics.enabled=true
-c spark.metrics.conf=/spark/conf/metric.properties{code}
{_}*metric.properties:*{_}{{{{}}{}}}
{code:java}
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus{code}

Result:

Both of these endpoints have some metrics
{code:java}
<driver-ip>:4040/metrics/prometheus 
<driver-ip>:4040/metrics/executors/prometheus{code}

{{But the executor one}} misses metrics under the executor namespace described here: [https://spark.apache.org/docs/3.1.2/monitoring.html#component-instance--executor]
So everything is missing from {{bytesRead.count}} to {{threadpool.startedTasks}}

There are neither error nor warn level entries in the driver/executor logs.

By changing to ConsoleSink  I can see all the necessary metrics:
{code:java}
*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink
*.sink.console.period=10
*.sink.console.unit=seconds{code}

{{ }}
{{Something is wrong with the spark-prometheus integration}}
Bug: Show the root cause exception in TestUtils.assertExceptionMsg
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/22 18:44
Affects Version/s: 3.3.0

{{TestUtils.assertExceptionMsg}} is great, but when the assertion _doesn't_ match, it can be challenging to tell why, because the exception tree that was searched isn't printed. Only way I could find to fix it up was to run things in a debugger and check the exception tree.
It would be very helpful if {{assertExceptionMsg}} printed out the exception tree in which it was searching (upon failure).
Bug: Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/22 19:21
Affects Version/s: 3.2.1

Currently we have handled this in the handleShuffleMergeFinalized during finalization ensuring the finalize request is indeed for the current stage's shuffle dependency shuffleMergeId. The same check has to be done before registering merge statuses as well.
Bug: Make links for stderr/stdout on Spark on Kube configurable
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/22 18:45
Affects Version/s: 3.2.2

Unlike YARN different clusters store pod logs in different locations. We should allow people to configure the links so that they can go a web UI for their clusters stderr/stdout or print out the kubectl commands for users who don't have a link configured.
Bug: set the amount of explained variance as parameter of pyspark.ml.feature.PCA
Status: Open | Priority: Major | Resolution: 
Created: 01/Feb/22 17:38
Affects Version/s: 3.2.2

As in [sklearn.decomposition.PCA|https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html], where:

if {{0 < n_components < 1}} select the number of components such that the amount of variance that needs to be explained is greater than the percentage specified by n_components

it would be useful to have a similar behavior with the k parameter in pyspark.ml.feature.PCA.
Bug: Add support for ArrayType of nested StructType to arrow-based conversion
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Feb/22 15:21
Affects Version/s: 3.2.1

This proposes to add support for ArrayType of nested StructType to arrow-based conversion.
This allows Pandas UDFs, mapInArrow UDFs, and toPandas to operate on columns of type Array of Struct, via arrow serialization.
Bug: Add SQL migration guide on restoring loose upcast from string
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Feb/22 02:01
Affects Version/s: 3.2.1

After SPARK-24586, loose upcasting from string to other types are not allowed by default. User can still set {{spark.sql.legacy.looseUpcast=true}} to preserve old behavior but it's not documented in the SQL migration guide.
Bug: [TESTS] Replace 'abc symbols with Symbol("abc") in tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/22 15:10
Affects Version/s: 3.2.1

When building with Scala 2.13 many test classes produce warnings like:
{code:java}
[warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @  | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead
[warn]           'd.cast("string"),
[warn]           ^
[warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @  | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead
[warn]           'e.cast("string")).collect())
 {code}
For easier migration to Scala 3.x later it would be good to fix this warnings!
Bug: Sync missing R packages with CI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Feb/22 16:37
Affects Version/s: 3.2.1

Current GitHub workflow job *Linters, licenses, dependencies and documentation generation* is missing R packages to complete Documentation and API build.

*Build and test* -  is not failing as these packages are installed in the base image.

We need to keep them in-sync IMO with the base image for easy switch back to ubuntu runner when ready.

These R packages are missing: *markdown* and  *e1071*

Reference:

Base image - https://hub.docker.com/layers/dongjoon/apache-spark-github-action-image/20220207/images/sha256-af09d172ff8e2cbd71df9a1bc5384a47578c4a4cc293786c539333cafaf4a7ce?context=explore
Bug: Upgarde `org.scalatestplus:mockito` to 3.2.11.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Feb/22 03:08
Affects Version/s: 3.3.0

Upgarde `org.scalatestplus:mockito` from 3.2.110.0 to 3.2.11.0 and the actual mockito is upgraded from 3.12 to 4.2.0
Bug: Enable a callable in pyspark.pandas.DataFrame.loc
Status: Open | Priority: Minor | Resolution: 
Created: 18/Feb/22 20:28
Affects Version/s: 3.2.1

Hi,

I was hoping that you would enable a callable to be used in the pyspark.pandas.DataFrame.loc method.

I use a lambda function in loc all the time in my pandas code, and I was hoping to be able to use most of my pandas code with your new pandas API.

 

Thank you!
Bug: PersistentVolumeClaim does not work in clusters with multiple nodes
Status: Closed | Priority: Minor | Resolution: Invalid
Created: 15/Feb/22 23:07
Affects Version/s: 3.2.1

We are using {{spark-submit}} to establish a ThriftServer warehouse on Google Kubernetes Engine. The Spark documentation on running on Kubernetes suggests that we can use [persistentVolumeClaim|https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim] for Spark applications.
{code:bash}
spark-submit \
  --master k8s://$KUBERNETES_SERVICE_HOST \
  --deploy-mode cluster \
  --class $THRIFTSERVER \
  --conf spark.sql.catalogImplementation=hive \
  --conf spark.sql.hive.metastore.sharedPrefixes=org.postgresql \
  --conf spark.hadoop.hive.metastore.schema.verification=false \
  --conf spark.hadoop.datanucleus.schema.autoCreateTables=true \
  --conf spark.hadoop.datanucleus.autoCreateSchema=false \
  --conf spark.sql.parquet.int96RebaseModeInWrite=CORRECTED \
  --conf spark.hadoop.javax.jdo.option.ConnectionDriverName=org.postgresql.Driver \
  --conf spark.hadoop.javax.jdo.option.ConnectionUserName=spark \
  --conf spark.hadoop.javax.jdo.option.ConnectionPassword=Password1! \
  --conf spark.sql.warehouse.dir=$MOUNT_PATH \
  --conf spark.kubernetes.driver.pod.name=spark-hive-thriftserver-driver \
  --conf spark.kubernetes.driver.label.app.kubernetes.io/name=thriftserver \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$CLAIM_NAME \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.readOnly=false \
  --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$CLAIM_NAME \
  --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.readOnly=false \
  --conf spark.kubernetes.executor.deleteOnTermination=true \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-kube \
  --conf spark.kubernetes.container.image=$IMAGE \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.executor.memory=2g \
  --conf spark.driver.memory=2g \
  local:///$JAR {code}
When it ran, it created one driver and two executors. Each of these wanted to use the same pvc. Unfortunately, at least one of these pods was scheduled on a different node from the rest. As GKE mounts pvs to nodes in order to honor pvcs for pods, that odd pod out was unable to attach the pv:
{code:java}
FailedMount
Unable to attach or mount volumes: unmounted volumes=[spark-warehouse], unattached volumes=[kube-api-access-grfld spark-conf-volume-exec spark-warehouse spark-local-dir-1]: timed out waiting for the condition {code}
This is because GKE like many cloud providers does not support {{ReadWriteMany}} for pvcs/pvs.
----
I suggest changing the documentation not to suggest using pvcs for ThriftServers.
Bug: Show full stacktrace in tests by default in PySpark tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Feb/22 10:13
Affects Version/s: 3.3.0

SPARK-33407 and SPARK-31849 hide Java stacktrace and internal Python worker side traceback by default but that makes a bit harder to debug the test failures. We should probably show the full stacktrace by default in tests.
Bug: Retrieve a Spark dataframe as Arrow batches
Status: Open | Priority: Minor | Resolution: 
Created: 04/Feb/22 17:15
Affects Version/s: 3.2.0

Using the Java API, is there a way to efficiently retrieve a dataframe as Arrow batches ?

I have a pretty large dataset on my cluster so I cannot collect it using [collectAsList|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html#collectAsList--] which download every thing at once and saturate my JVM memory

Seeing that Arrow is becoming a standard to transfer large datasets and that Spark uses a lot Arrow, is there a way to transfer my Spark dataframe with Arrow batches ?

This would be ideal to process the data batch per batch and avoid saturating the memory.
 

I am looking for an API like this (in Java)

 
{code:java}
var stream = dataframe.collectAsArrowStream()
while (stream.hasNextBatch()) {
    var batch = stream.getNextBatch()
    // do some stuff with the arrow batch
}
{code}

It would be even better if I can split the dataframe into several streams so I can download and process it in parallel
Bug: Instrument __enter__ and __exit__ magic methods for pandas API on Spark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/22 16:19
Affects Version/s: 3.2.1

Create the ticket since instrumenting _{_}enter{_}_ and _{_}exit{_}_ magic methods for pandas API on Spark can help improve accuracy of the usage data. Besides, we are interested in extending the pandas-on-Spark usage logger to other PySpark modules in the future so it will help improve accuracy of usage data of other PySpark modules.

For example, for the following code:

 
{code:java}
pdf = pd.DataFrame(
    [(0.2, 0.3), (0.0, 0.6), (0.6, 0.0), (0.2, 0.1)], columns=["dogs", "cats"]
)
psdf = ps.from_pandas(pdf)

with psdf.spark.cache() as cached_df:
    self.assert_eq(isinstance(cached_df, CachedDataFrame), True)
    self.assert_eq(
        repr(cached_df.spark.storage_level), repr(StorageLevel(True, True, False, True))
    ){code}
 

pandas-on-Spark usage logger records the internal call [self.spark.unpersist()|https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L12518] since __enter__ and __exit__ methods of [CachedDataFrame|https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L12492] are not instrumented.
Bug: Replace `toIterator` with `iterator` for `IterableLike`/`IterableOnce` to cleanup deprecated api usage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Feb/22 03:44
Affects Version/s: 3.3.0

In Scala 2.12, {{IterableLike.toIterator}} identified as {{{}@deprecatedOverriding{}}}:

 
{code:java}
@deprecatedOverriding("toIterator should stay consistent with iterator for all Iterables: override iterator instead.", "2.11.0") 
override def toIterator: Iterator[A] = iterator {code}
In Scala 2.13, {{IterableOnce.toIterator}} identified as {{{}@deprecated{}}}:
{code:java}
@deprecated("Use .iterator instead of .toIterator", "2.13.0") @`inline` final def toIterator: Iterator[A] = iterator {code}
Bug: JDBC Source with MariaDB connection returns column names as values
Status: Open | Priority: Minor | Resolution: 
Created: 25/Feb/22 08:03
Affects Version/s: 3.2.1

Using a JDBC source with the official MariaDB JDBC driver and a JDBC connection URL like the following does not work as expected:
{noformat}
jdbc:mariadb://db.example.com:3306/schema {noformat}
Assume we have a table "values" like the following in MariaDB:
||id (binary)||name (varchar)||
|0xAB|Name 1|
|0xBC|Name 2|

We intend to create and display a data frame from it like this:
{code:scala}
spark.read
  .format("jdbc")
  .option("url", "jdbc:mariadb://db.example.com:3306/schema")
  .option("dbtable", "values")
  .load()
  .show{code}
*Expected Behavior*

Using such a connection URL on an arbitrary MariaDB table or query results in a data frame that reflects the table structure and content from MariaDB correctly, with columns having the correct type and values.

The output of the above should be
{noformat}
+----+------+
|  id|  name|
+----+------+
|[AB]|Name 1|
|[BC]|Name 2|
+----+------+{noformat}
*Observed Behavior*

Result rows contain column names as values, making them effectively useless to work with.

The actual output is
{noformat}
+-------+----+
|     id|name|
+-------+----+
|[69 64]|name|
|[69 64]|name|
+-------+----+{noformat}
*Further information*

An easy workaround appears to be specifying "mysql" instead of "mariadb" in the connection URL while explicitly specifying the MariaDB driver. I'd expect the mariadb URL to work out of the box, however.

It looks like this has been an issue since at least 2016 according to a [StackOverflow post|https://stackoverflow.com/questions/38808463/incorrect-data-while-loading-jdbc-table-in-spark-sql].