av9ash/GitBugs huggingface dataset:
Bug: Cached partitions are traversed multiple times (invalidating Accumulator consistency)
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/22 08:56
Affects Version/s: 2.4.4

We are trying to use Accumulators to count RDDs without having to force `.count()` on them for efficiency reasons.  We are aware tasks can fail and re-run, which will invalidate the value of the accumulator, so we count the number of times a partition has been traversed, so we can detect this.

The problem is that partitions are being traversed multiple times even though

 - We cache the RDD in memory _after we have applied the logic below_
 - No tasks are failing, no executors are dying.
 - There is plenty of memory (no RDD eviction)

The code we use:

{code:java}
val count: LongAccumulator
val partitionTraverseCounts: List[LongAccumulator]

def increment(): Unit = count.add(1)

def incrementTimesCalled(partitionIndex: Int): Unit =
      partitionTraverseCounts(partitionIndex).add(1)

def incrementForPartition[T](index: Int, it: Iterator[T]): Iterator[T] = {
    incrementTimesCalled(index)

    it.map { x =>
      increment()
      x
    }
  }
{code}

How we use the above:

{code:java}
rdd.mapPartitionsWithIndex(safeCounter.incrementForPartition)
{code}


We have a 50 partition RDD, and we frequently see odd traverse counts:

{code:java}
traverseCounts: List(2, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2)
{code}

As you can see, some partitions are traversed twice, while others are traversed only once.

To confirm no task failures:

{code:bash}
cat job.log | grep -i task | grep -i fail
{code}

To confirm no memory issues:

{code:bash}
cat job.log | grep -i memory
{code}

We see every log line has multiple GB memory free.

We also don't see any errors or exceptions.

Question:

1. Why is spark traversing a cached RDD multiple times?
2. Is there any way to disable this?

Many thanks,

Sam
Bug: Pandas API on Spark requires Pandas
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 09/Aug/22 03:36
Affects Version/s: 3.3.0

Pandas API on Spark includes code like:

> import pandas as pd
> from pandas.api.types import is_hashable, is_list_like  # type: ignore[attr-defined]

This breaks if you don't have pandas installed on your Spark cluster.

Pandas API was supposed to be an API not pandas integration, why does it require pandas to be installed?

In many places Spark jobs may be run on various Spark clusters with no assurance of particular Python packages installed at a root level. 

Can this dependency be removed? Or the required version of Pandas be bundled with the Spark distribution? Similar for numpy and other deps.

If not the docs should clearly state it is not merely a Spark API that mirror the Pandas API, but something quite different.
Bug: Issue with Spark Core  version 3.3.0
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Aug/22 17:48
Affects Version/s: 3.3.0

Hi,

While creating object in java of SparkConf myConf = new SparkConf();   shpwing noClassDefintion Found error

 

Not able set conf properties. Spark Version 3.3.0.

 

I have imported the  import org.apache.spark.SparkConf;   also but still not getting initialized
Bug: How to write (save) PySpark dataframe containing vector column?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Aug/22 22:51
Affects Version/s: 3.3.0

I'm trying to same the PySpark dataframe after transforming it using ML Pipeline. But when I save it the weird error is triggered every time. Here are the columns of this dataframe:

|-- label: integer (nullable = true)

|-- dest_index: double (nullable = false)

|-- dest_fact: vector (nullable = true)

|-- carrier_index: double (nullable = false)

|-- carrier_fact: vector (nullable = true)

|-- features: vector (nullable = true)

And the following error occurs when trying to save this dataframe that contains vector data:
{code:java}
// training.write.parquet("training_files.parquet", mode = "overwrite") {code}
{noformat}
Py4JJavaError: An error occurred while calling o440.parquet. : org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:638) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:278) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
...
{noformat}
 

I tried to use differently available {{winutils}} for Hadoop from [this GitHub repository|https://github.com/cdarlint/winutils] but with not much luck. Please help me in this regard. How can I save this dataframe so that I can read it in any other jupyter notebook file? Feel free to ask any questions. Thanks
Bug: Limit improperly pushed down through window using ntile function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/22 01:29
Affects Version/s: 3.2.2

Limit is pushed down through a window using the ntile function, which causes results that differ from Hive 2.3.9, and Prestodb 0.268, and older versions of Spark (e.g., 3.1.3).

Assume this data:
{noformat}
create table t1 stored as parquet as
select *
from range(101);
{noformat}
Also assume this query:
{noformat}
select id, ntile(10) over (order by id) as nt
from t1
limit 10;
{noformat}
Spark 3.2.2, Spark 3.3.0, and master produce the following:
{noformat}
+---+---+
|id |nt |
+---+---+
|0  |1  |
|1  |2  |
|2  |3  |
|3  |4  |
|4  |5  |
|5  |6  |
|6  |7  |
|7  |8  |
|8  |9  |
|9  |10 |
+---+---+
{noformat}
However, Spark 3.1.3, Hive 2.3.9, and Prestodb 0.268 produce the following:
{noformat}
+---+---+
|id |nt |
+---+---+
|0  |1  |
|1  |1  |
|2  |1  |
|3  |1  |
|4  |1  |
|5  |1  |
|6  |1  |
|7  |1  |
|8  |1  |
|9  |1  |
+---+---+
{noformat}
Bug: Upgrade slf4j to 1.7.36
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/22 19:21
Affects Version/s: 3.4.0

Snyk have [open up a PR at my branch|https://github.com/bjornjorgensen/spark/pull/19], where they want to upgrade [slf4j|https://www.slf4j.org]

The recommended version is 4 versions ahead of your current version. 
The recommended version was released 6 months ago, on 2022-02-08.

[Release log|https://www.slf4j.org/news.html]
Bug: Global aggregation against pandas aggregate UDF does not take the column order into account
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/22 03:08
Affects Version/s: 3.1.3

{code}
import pandas as pd 
from pyspark.sql import functions as f 

@f.pandas_udf("double") 
def AVG(x: pd.Series) -> float: 
    return x.mean() 


abc = spark.createDataFrame([(1.0, 5.0, 17.0)], schema=["a", "b", "c"]) 
abc.agg(AVG("a"), AVG("c")).show()
abc.select("c", "a").agg(AVG("a"), AVG("c")).show()
{code}

{code}
+------+------+
|AVG(a)|AVG(c)|
+------+------+
|   1.0|  17.0|
+------+------+

+------+------+
|AVG(a)|AVG(c)|
+------+------+
|  17.0|   1.0|
+------+------+
{code}

Both have to be the same.
Bug: Consolidate `lit` function with NumPy scalar in sql and pandas module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 22:55
Affects Version/s: 3.4.0

Per [https://github.com/apache/spark/pull/37560#discussion_r952882996,]

function `lit` with NumPy scalar in sql and pandas module have different implementations, thus, sql has a less precise result than pandas.

We shall make their result consistent, the more precise, the better.
Bug: Re-enable excel I/O test for pandas API on Spark.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 07:50
Affects Version/s: 3.4.0

Currently we're skipping the `read_excel` and `to_excel` tests for pandas API on Spark, since the `openpyxl` is not installed in our test environments.
We should enable this test for better test coverage.
Bug: Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/22 00:39
Affects Version/s: 3.4.0

Currently on S3 the checkpoint file manager (called FileContextBasedCheckpointFileManager) is based on rename. So when a file is opened for an atomic stream a temporary file used instead and when the stream is committed the file is renamed.

But on S3 a rename will be a file copy. So it has some serious performance implication.


But on Hadoop 3 there is new interface introduce called *Abortable* and *S3AFileSystem* has this capability which is implemented by on top S3's multipart upload. So when the file is committed a POST is sent ([https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html]) and when aborted a DELETE will be send 
([https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html])
Bug: Not able to format using scalafmt
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/22 18:58
Affects Version/s: 3.4.0

I'm following the guidance in [https://spark.apache.org/developer-tools.html] using 
{code:java}
./dev/scalafmt{code}

to format the code, but getting this error:
{code:java}
[ERROR] Failed to execute goal org.antipathy:mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) on project spark-parent_2.12: Error formatting Scala files: missing setting 'version'. To fix this problem, add the following line to .scalafmt.conf: 'version=3.2.1'. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException {code}
Bug: Incorrect ASCII value for Latin-1 Supplement characters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/22 22:30
Affects Version/s: 3.2.2

the `ascii()` built-in function in spark doesn't support Latin-1 Supplement characters which value between [128, 256). Instead, it produces a wrong value, -62 or -61 for all the chars. But the `chr()` built-in function supports value in [0, 256) and normally `ascii` should be the inverse of `chr()`
Bug: DataType argument in StructType.add is incorrectly throwing scala.MatchError
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 30/Aug/22 23:27
Affects Version/s: 3.3.0

*Problem Description*

as part of contract mentioned here, Spark should be able to support {{IntegerType}} as an argument in StructType.add method. However, it complaints with {{scala.MatchError}} today.

 

If we call the override version which access String value as Type e.g. "Integer" - it works.

*How to Reproduce*
 # Create a Kotlin Project - I have used Kotlin but Java will also work (needs minor adjustment)
 # Place the attached CSV file in {{src/main/resources}}   
 # Compile the project with Java 11
 # Run - it will give you error.

{code:java}
Exception in thread "main" scala.MatchError: org.apache.spark.sql.types.IntegerType@363fe35a (of class org.apache.spark.sql.types.IntegerType)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.externalDataTypeFor(RowEncoder.scala:240)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.externalDataTypeForInput(RowEncoder.scala:236)
    at org.apache.spark.sql.catalyst.expressions.objects.ValidateExternalType.<init>(objects.scala:1890)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:197)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:192)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:73)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:81)
    at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:92)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89)
    at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:444)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:185) {code}



 # Now change line (commented as HERE) - to have a String value i.e. "Integer"
 # It works

*Ask*
 # Why does it not accept IntegerType, StringType as DataType as part of the parameters supplied through {{add}} function in {{StructType}} ?
 # If this is a bug, do we know when the fix can come?
Bug: Initialize projection used for Python UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/22 10:01
Affects Version/s: 3.1.3

{code}
>>> from pyspark.sql.functions import udf, rand
>>> spark.range(10).select(udf(lambda x: x)(rand())).show()
{code}

{code}
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$10(EvalPythonExec.scala:126)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:1161)
	at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:1176)
	at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1213)
{code}
Bug: Remove Arrow in AppVeyor for now
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/22 05:20
Affects Version/s: 3.4.0

SparkR does not support Arrow 9.0.0 (SPARK-40114) so the tests currently fail, https://ci.appveyor.com/project/HyukjinKwon/spark/builds/44490387.
Bug: Convert condition to java in DataFrameWriterV2.overwrite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/22 05:28
Affects Version/s: 3.1.3

DataFrameWriterV2.overwrite() fails to convert the condition parameter to java. This prevents the function from being called.

It is caused by the following commit that deleted the `_to_java_column` call instead of fixing it: [https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9]
Bug: is kubernetes jar required if not using that executor?
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 16/Aug/22 02:00
Affects Version/s: 3.3.0

my docker file is very big with pyspark

can i remove dis files below if i don't use 'kubernetes executor'?

11M     total
4.0M    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-core-5.12.2.jar
840K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-client-5.12.2.jar
760K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-admissionregistration-5.12.2.jar
704K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-apiextensions-5.12.2.jar
640K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-autoscaling-5.12.2.jar
528K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-extensions-5.12.2.jar
516K    /usr/local/lib/python3.10/site-packages/pyspark/jars/spark-kubernetes_2.12-3.3.0.jar
456K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-networking-5.12.2.jar
436K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-apps-5.12.2.jar
364K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-storageclass-5.12.2.jar
336K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-policy-5.12.2.jar
264K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-flowcontrol-5.12.2.jar
244K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-batch-5.12.2.jar
192K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-discovery-5.12.2.jar
176K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-rbac-5.12.2.jar
160K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-node-5.12.2.jar
144K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-certificates-5.12.2.jar
104K    /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-events-5.12.2.jar
80K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-metrics-5.12.2.jar
68K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-scheduling-5.12.2.jar
48K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-coordination-5.12.2.jar
20K     /usr/local/lib/python3.10/site-packages/pyspark/jars/kubernetes-model-common-5.12.2.jar
Bug: Support read/write.csv() in SparkR
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Aug/22 09:13
Affects Version/s: 3.3.0

Today, almost languages support the DataFrameReader.csv API, only R is missing. we need to use df.read() to read the csv file. We need a more high-level api for it.

Java:
[DataFrameReader.csv()|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameReader.html]

Scala:
[DataFrameReader.csv()|https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameReader.html#csv(paths:String*):org.apache.spark.sql.DataFrame]

Python:
[DataFrameReader.csv()|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameReader.csv.html#pyspark.sql.DataFrameReader.csv]
Bug: Pin Arrow version to 8.0.0 in AppVeyor
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Aug/22 05:19
Affects Version/s: 3.4.0

Currently SparkR tests fail https://ci.appveyor.com/project/HyukjinKwon/spark/builds/44490387 because SparkR does not support Arrow 9.0.0+, see also SPARK-40114

We should pin the version to 8.0.0 for now.
Bug: Fix flaky tests in ImageFileFormatSuite
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/22 03:35
Affects Version/s: 3.2.2

There are 3 test cases that become flaky in the GitHub action tests:

[https://github.com/apache/spark/runs/7941765326?check_suite_focus=true]

We should fix them.
Bug: spark  concurrent overwrite mode writes data to files in HDFS format, all request data write success
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 31/Aug/22 04:59
Affects Version/s: 3.0.1

We use Spark as a service. The same Spark service needs to handle multiple requests, but I have a problem with this

When multiple requests are overwritten to a directory at the same time, the results of two overwrite requests may be written successfully. I think this does not meet the definition of overwrite write

First I ran Write SQL1, then I ran Write SQL2, and I found that both data were written in the end, which I thought was unreasonable
{code:java}
sparkSession.udf.register("sleep",  (time: Long) => Thread.sleep(time))



-- write sql1
sparkSession.sql("select 1 as id, sleep(40000) as time").write.mode(SaveMode.Overwrite).parquet("path")

-- write sql2
 sparkSession.sql("select 2 as id, 1 as time").write.mode(SaveMode.Overwrite).parquet("path") {code}
When the spark source, and I saw that all these logic in InsertIntoHadoopFsRelationCommand this class.

 

When the target directory already exists, Spark directly deletes the target directory and writes to the _temporary directory that it requests. However, when multiple requests are written, the data will all append in; For example, in Write SQL above, this procedure occurs

1. excute write sql1, spark  create the _temporary directory for SQL1, and continue

2. excute write sql2 , spark will  delete the entire target directory and create its own 
_temporary
3. sql2 writes  its data

4. sql1 complete the calculation, The corresponding _temporary /0/attemp_id directory does not exist and so the request fail. However, the task is retried, but the _temporary  directory is not deleted when the task is retried. Therefore, the execution result of sql1  result is append to the target directory 

 

Based on the above process, the write process, can  spark do a directory check before the write task or some other way to avoid this kind of problem?
Bug: Load Data from S3 deletes data source file
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 31/Aug/22 06:03
Affects Version/s: 3.2.1

Hello, 

I'm using spark to [load data|https://spark.apache.org/docs/latest/sql-ref-syntax-dml-load.html] into a hive table through Pyspark, and when I load data from a path in Amazon S3, the original file is getting wiped from the Directory. The file is found, and is populating the table with data. I also tried to add the `Local` clause but that throws an error when looking for the file. When looking through the documentation it doesn't explicitly state that this is the intended behavior.

Thanks in advance!
{code:java}
spark.sql("CREATE TABLE src (key INT, value STRING) STORED AS textfile")
spark.sql("LOAD DATA INPATH 's3://bucket/kv1.txt' OVERWRITE INTO TABLE src"){code}