av9ash/GitBugs huggingface dataset:
Bug: Provide latest Spark 2.4.7 runnable distribution
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 01/Nov/20 07:44
Affects Version/s: 2.4.7

Not sure if this is the right approach, however it would be great if latest Spark 2.4.7 runnable distribution can be provided here - [https://spark.apache.org/downloads.html]

Currently it seems the last build was done on Sept 12th' 2020. 

I'm working on running Spark workloads on EKS using EKS IRSA. I'm able to run Spark workloads on EKS using IRSA with Spark 3.0/ Hadoop 3.2, however I want to do the same with Spark 2.4.7/ Hadoop 2.7. 

Recently this PR was merged with 2.4.x - [https://github.com/apache/spark/pull/29877] and therefore I'm in need of latest Spark distribution 

 

PS: I tried building latest Spark 2.4.7 myself as well using Maven, however there are too many errors every-time when it reaches R, therefore it would be great if Spark community itself can provide the latest build.
Bug: R/run-tests.sh is not compatible with testthat >= 3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/20 21:07
Affects Version/s: 3.0.0

Currently we use {{testthat:::test_package_dir}} to run full SparkR tests with {{testthat > 1.0}}. However, it has been dropped in {{testthat}} 3.0 (https://github.com/r-lib/testthat/pull/1054).

Because of that AppVeyor tests fail with

{code:r}
Spark package found in SPARK_HOME: C:\projects\spark\bin\..
Error in get(name, envir = asNamespace(pkg), inherits = FALSE) : 
  object 'test_package_dir' not found
Calls: ::: -> get
Execution halted
{code}

It seems like we can use {{testthat::test_dir}} which, since {{testthat}} 3.0, supports {{package}} parameter.
Bug: Run PySpark coverage only in the master branch
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Nov/20 06:10
Affects Version/s: 3.0.1

Currently PySpark test coverage runs in branch-3.0 (https://github.com/apache/spark/pull/23117#issuecomment-735557536). We should only run this in the master branch.
Bug: PythonException: An exception was thrown from a UDF: 'OSError: Invalid IPC message: negative bodyLength'.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 27/Nov/20 04:09
Affects Version/s: 3.0.1

Hello,

We are using Databricks on Azure to process large amount of ecommerce data. Databricks runtime is 7.3 which includes Apache spark 3.0.1 and Scala 2.12.

During processing, there is a groupby operation on the DataFrame that consistently gets an exception of this type:

 

{color:#ff0000}PythonException: An exception was thrown from a UDF: 'OSError: Invalid IPC message: negative bodyLength'. Full traceback below: Traceback (most recent call last): File "/databricks/spark/python/pyspark/worker.py", line 654, in main process() File "/databricks/spark/python/pyspark/worker.py", line 646, in process serializer.dump_stream(out_iter, outfile) File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 281, in dump_stream timely_flush_timeout_ms=self.timely_flush_timeout_ms) File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 97, in dump_stream for batch in iterator: File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 271, in init_stream_yield_batches for series in iterator: File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 287, in load_stream for batch in batches: File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 228, in load_stream for batch in batches: File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 118, in load_stream for batch in reader: File "pyarrow/ipc.pxi", line 412, in __iter__ File "pyarrow/ipc.pxi", line 432, in pyarrow.lib._CRecordBatchReader.read_next_batch File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status OSError: Invalid IPC message: negative bodyLength{color}

 

Code that causes this:

{color:#ff0000}x = df.groupby('providerid').apply(domain_features){color}

{color:#ff0000}display(x.info()){color}

Dataframe size - 22 million rows, 31 columns
 One of the columns is a string ('providerid') on which we do a groupby followed by an apply  operation. There are 3 distinct provider ids in this set. While trying to enumerate/count the results, we get this exception.

We've put all possible checks in the code for null values, or corrupt data and we are not able to track this to application level code. I hope we can get some help troubleshooting this as this is a blocker for rolling out at scale.

The cluster has 8 nodes + driver, all 28GB RAM. I can provide any other settings that could be useful. 
 Hope to get some insights into the problem. 

Thanks,

Darshat Shah
Bug: Partition predicate pushdown into Hive metastore support not-equals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/20 07:41
Affects Version/s: 3.1.0

https://github.com/apache/hive/blob/b8bd4594bef718b1eeac9fceb437d7df7b480ed1/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L2194-L2207

https://issues.apache.org/jira/browse/HIVE-2702
Bug:  Hive partition pruning support date type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 08:34
Affects Version/s: 3.1.0

Hive partition pruning can support date type:
https://issues.apache.org/jira/browse/HIVE-5679
https://github.com/apache/hive/commit/5106bf1c8671740099fca8e1a7d4b37afe97137f
Bug: Improve the style of the checkbox in executor page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 16:42
Affects Version/s: 3.1.0

The width of class `container-fluid-div` is set as 200px after https://github.com/apache/spark/pull/21688 . This makes the checkbox in the executor page messy.
We should remove the width style.
Bug: Adjust mypy configuration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/20 01:57
Affects Version/s: 3.1.0

At the moment, with exception to type ignores, we use default MyPy configuration. These already provide decent coverage, but are somewhat less restrictive than the ones used in {{typeshed}} and {{pyspark-stubs}}.

We should consider at least the following:

- {{strict_optional}}
- {{no_implicit_optional}}

It might be also a good idea to add {{disallow_untyped_defs}}, which will allow us to catch any instances of user-facing code, that are missing annotations.
Bug: Re SPARK-21820.  Creating Spark dataframe with carriage return/line feed leaves cr in multiline
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 19/Nov/20 12:03
Affects Version/s: 2.4.5

In SPARK-21820 I see what seems to be the same issue reported, but marked as resolved there.  Over the past few days I have battled a dataset that occasionally has \r\n at the end of lines and I claim I do see this errant behavior of not removing \r\n.

In my code, I do 
{code:java}
// code placeholder# CSV options
infer_schema = "false"
first_row_is_header = "true"
multi_line = "true"
delimiter = ","
# The applied options are for CSV files. For other file types, these will be ignored.
df_train = spark.read.format(train_file_type) \
  .option("inferSchema", infer_schema) \
  .option("header", first_row_is_header) \
  .option("sep", delimiter) \
  .option("multiLine", multi_line) \
  .option("escape", '"') \
  .load(train_file_location)
{code}
So I am reading in a csv file and setting multiLine to true.  However, all cases where there are \r\n in the training_file, \r is left behind.  This includes the header which has a column ending in \r.  The only way I have been able to workaround this is to manually edit the data file to remove the \r, but I do not want to do this on a case to case basis.

Therefore, I am claiming this behavior is still present in 2.4.5 and is a bug.

I am using version 2.4.5 because I am using Spark-NLP which to my knowledge has not been built to use 3 yet, so the version is key for me.
Bug: Add/remove (un)necessary cache and restructure GitHub Actions yaml
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/20 04:08
Affects Version/s: 2.4.8

Currently, GitHub Actions build has some unnecessary cache/commands. For example, if you run SBT only .m2 cache is not needed. We should clean up and re-organize.
Also, we should add {{~/.sbt}} into cache. See https://github.com/sbt/sbt/issues/3681
Bug: Add gcs-connector to hadoop-cloud module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 10:30
Affects Version/s: 3.4.0

Spark comes with some S3 batteries included, which makes it easier to use with S3, for GCS to work users are required to manually configure the jars. This is especially problematic for python users who may not be accustomed to java dependencies etc. This is an example of workaround for pyspark: [pyspark_gcs|https://github.com/ravwojdyla/pyspark_gcs]. If we include the [GCS connector|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage], it would make things easier for GCS users.

Please let me know what you think.
Bug: Partition Parameters are not updated even after ANALYZE TABLE command
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Nov/20 12:03
Affects Version/s: 3.0.1

Here are the reproduction steps:
{code:java}
scala> spark.sql("CREATE TABLE t (a string,b string) PARTITIONED BY (p string) STORED AS PARQUET")
Hive Session ID = d44e21ee-2d5c-48ab-91bf-26cb25775486
res0: org.apache.spark.sql.DataFrame = []
scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('aaa', 'bbb')")
res1: org.apache.spark.sql.DataFrame = []

scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('ccc', 'ddd')")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("ANALYZE TABLE t PARTITION(p='p1') COMPUTE STATISTICS")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.sql("DESCRIBE FORMATTED t PARTITION (p='p1')").show(50, false)
...
|Partition Parameters |{rawDataSize=0, numFiles=1, numFilesErasureCoded=0, transient_lastDdlTime=1604404640, totalSize=532, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"a":"true","b":"true"}}, numRows=0}| |
...
|Partition Statistics |1064 bytes, 2 rows | |
...
{code}
My expectation would be that the Partition Parameters should be updated after ANALYZE TABLE.
Bug: Standardize exception messages in Spark
Status: Open | Priority: Major | Resolution: 
Created: 24/Nov/20 18:13
Affects Version/s: 3.1.0

In the SPIP: Standardize Exception Messages in Spark, there are three major improvements proposed:
 # Group error messages in dedicated files.
 # Establish an error message guideline for developers.
 # Improve error message quality.

The first step is to centralize error messages for each component into its own dedicated file(s). This can help with auditing error messages and subsequent tasks to establish a guideline and improve message quality in the future. 

A general rule of thumb for grouping exceptions: 
 * AnalysisException => QueryCompilationErrors
 * SparkException, RuntimeException(UnsupportedOperationException, IllegalArgumentException...) => QueryExecutionErrors
 * ParseException => QueryParsingErrors

(A special case for {{Command}}: since commands are executed eagerly, users can immediately see the errors even if the exceptions are thrown in {{SparkPlan.execute}}. In this case, we should group the exceptions in commands as QueryCompilationErrors.)

Here is an example RP to group all `AnalysisExcpetion` in Analyzer into QueryCompilationErrors: [SPARK-32670|https://github.com/apache/spark/pull/29497]

Please see the [SPIP|https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing] for more details.

Exceptions per component:
||component||exception||
|sql|714|
|core|334|
|mllib|161|
|streaming|43|
|resource-managers|42|
|external|35|
|examples|20|
|mllib-local|10|
|graphx|6|
|repl|5|
|hadoop-cloud|1|
Bug: Errors in running spark job on K8 with RPC Authentication Secret File properties
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/20 23:20
Affects Version/s: 3.0.0

I am running a spark job on Kubernetes with RPC authentication feature. First i am mounting a K8 secret to both driver and executor to a path and providing the same path for Auth configuration. Below is the spark-conf i am passing related to RPC Auth.
{code:java}
--conf spark.authenticate=true
--conf spark.kubernetes.driver.secrets.spark-secret-sa=/tmp/secrets
--conf spark.kubernetes.executor.secrets.spark-secret-sa=/tmp/secrets
--conf spark.authenticate.secret.driver.file=/tmp/secrets/token
--conf spark.authenticate.secret.executor.file=/tmp/secrets/token
{code}
 

Mounting of secret is successfully done on both driver and executor pods and i can see the token file written to the containers. Driver is able to load the secret as well from the file but executor pods are failing to load the secret from file, and fails with the below exception:
{code:java}
java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config
 at org.apache.spark.SecurityManager.$anonfun$getSecretKey$6(SecurityManager.scala:298)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:297){code}
 

The value of the spark.authenticate.secret.executor.file property is being passed in SPARK_JAVA_OPS

SPARK_JAVA_OPT_8: -Dspark.authenticate.secret.driver.file=/tmp/secrets/token
 SPARK_JAVA_OPT_9: -Dspark.authenticate.secret.executor.file=/tmp/secrets/token

I am suspecting that values from java options are not being read into the spark conf and ultimately leading to failure.

 

I am not sure if its a bug or a mis-configuration on my end. Amy help is greatly appreciated.
Bug: ExecutorMetrics are not written to CSV and StatsD sinks
Status: Open | Priority: Major | Resolution: 
Created: 02/Nov/20 15:14
Affects Version/s: 2.4.4

Metrics from the {{ExecutorMetrics}} namespace are not written to the CSV and StatsD sinks, even though some of them is available through the REST API (e.g.: {{memoryMetrics.usedOnHeapStorageMemory}}).

I couldn't find the {{ExecutorMetrics}} either on the driver or the workers.
Bug: Collapse Partial and Final Aggregation into Complete Aggregation mode
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Nov/20 08:44
Affects Version/s: 2.3.4

We should merge the Partial and Final Aggregation into one if there is no exchange between them.

 

Example: select col1, max(col2) from t1 join t2 on col1 group by col1

In this case, after the SortMergeJoin, Spark will do PartialAggregation and then FinalAggregation. So it will create HashTables two times which is not required. If there is lot of data after Join with many distinct col1's then there is a possibility of Spill also in HashAggregateExec. So Spill will also happen twice which can be avoided.
Bug: Group exception messages in catalyst/parser
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 08:10
Affects Version/s: 3.2.0

'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser'
|| Filename                     ||   Count ||
| AstBuilder.scala             |      36 |
| LegacyTypeStringParser.scala |       1 |
| ParseDriver.scala            |       3 |
| ParserUtils.scala            |       4 |
Bug: Group exception messages in sql/execution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 08:31
Affects Version/s: 3.2.0

'/core/src/main/scala/org/apache/spark/sql/execution'
||Filename||Count||
|ExplainUtils.scala|1|
|SQLExecution.scala|1|
|SparkPlan.scala|4|
|SparkSqlParser.scala|4|
|SparkStrategies.scala|2|
|WholeStageCodegenExec.scala|1|

 '/core/src/main/scala/org/apache/spark/sql/execution/adaptive'
||Filename||Count||
|CustomShuffleReaderExec.scala|3|
|QueryStageExec.scala|2|
|simpleCosting.scala|1|

'/core/src/main/scala/org/apache/spark/sql/execution/aggregate'
||Filename||Count||
|HashAggregateExec.scala|2|
|RowBasedHashMapGenerator.scala|1|
|TungstenAggregationIterator.scala|1|

'/core/src/main/scala/org/apache/spark/sql/execution/analysis'
||Filename||Count||
|DetectAmbiguousSelfJoin.scala|1|

'/core/src/main/scala/org/apache/spark/sql/execution/arrow'
||Filename||Count||
|ArrowWriter.scala|1|

'/core/src/main/scala/org/apache/spark/sql/execution/columnar'
||Filename||Count||
|ColumnAccessor.scala|2|
|ColumnBuilder.scala|1|
|ColumnType.scala|1|
|InMemoryRelation.scala|1|

'/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression'
||Filename||Count||
|compressionSchemes.scala|2|

'/core/src/main/scala/org/apache/spark/sql/execution/joins'
|| Filename             ||   Count ||
| HashJoin.scala       |       1 |
| HashedRelation.scala |       6 |


'/core/src/main/scala/org/apache/spark/sql/execution/metric'
|| Filename         ||   Count ||
| SQLMetrics.scala |       1 |


'/core/src/main/scala/org/apache/spark/sql/execution/python'
|| Filename                 ||   Count ||
| ExtractPythonUDFs.scala  |       1 |
| RowQueue.scala           |       1 |
| WindowInPandasExec.scala |       1 |


'/core/src/main/scala/org/apache/spark/sql/execution/stat'
|| Filename            ||   Count ||
| StatFunctions.scala |       2 |

'/core/src/main/scala/org/apache/spark/sql/execution/ui'
|| Filename                   ||   Count ||
| AllExecutionsPage.scala    |       1 |
| SQLAppStatusListener.scala |       1 |
Bug: python/run-tests.py calling python3.8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 21:44
Affects Version/s: 3.0.1

this line in run-tests.py on master:
|python_execs = [x for x in ["python3.6", "python3.8", "pypy3"] if which(x)]|

 

and this line in branch-3.0:

python_execs = [x for x in ["python3.8", "python2.7", "pypy3", "pypy"] if which(x)]

...are currently breaking builds on the new ubuntu 20.04LTS workers.

the default  system python is /usr/bin/python3.8 and we do NOT have a working python3.8 anaconda deployment yet.  this is causing python test breakages.

PRs incoming
Bug: spark run on kubernetes has Could not load KUBERNETES classes issue
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/20 10:01
Affects Version/s: 3.0.1

Hi, I am trying to run spark on my kubernetes cluster (it's not a minikube cluster). And I follow this doc: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] to create spark docker image and then submit the application step by step. However, it failed and I check the log of spark driver, it showed below error:

```+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.30.140.13 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner file:/root/Work/spark/examples/src/main/python/wordcount.py+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.30.140.13 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner file:/root/Work/spark/examples/src/main/python/wordcount.pyException in thread "main" org.apache.spark.SparkException: Could not load KUBERNETES classes. This copy of Spark may not have been compiled with KUBERNETES support. at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:942) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:265) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:877) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1013) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1022) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

```

I am not sure if I am missing which step. I have been blocked here several days. Cloud you please help me about this? Thanks in advance!

 

By the way, below is the step I did:
 # Prepare a kubernetes cluster and check I have appropriate permissions to list, create, edit and delete pods;
About this, I am sure, I have all necessary permissions.
 # Build distribution
```
./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
```
 # Build spark docker image
```

./bin/docker-image-tool.sh spark -t latest build
```

 # submit application 
```

./bin/spark-submit --master k8s://https://c7.us-south.containers.cloud.ibm.com:31937 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=docker.io/bluebosh/spark:python3 examples/src/main/python/wordcount.py
```
BTW, I am sure the master is correct and also my docker image has contained python.
Bug: Improve locality for push-based shuffle especially for join like operations
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Nov/20 18:07
Affects Version/s: 3.1.0

Currently, we only set locality for ShuffledRDD and ShuffledRowRDD with push-based shuffle.

In simple stage DAGs where a ShuffledRDD or ShuffledRowRDD is the only input RDD, Spark can handle locality fine. However, if we have a join operation where a stage can consume multiple shuffle inputs or other non-shuffle inputs, the locality will take a hit with how DAGScheduler currently determines the preferred location.

With push-based shuffle, we could potentially reuse the same set of merger locations across sibling ShuffleMapStages. This would enable a much better locality on the reducer stage side, where corresponding merged shuffle partitions for the multiple shuffle inputs are already colocated.
Bug: Limit the number of pending blocks in memory and store blocks that collide
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/20 22:24
Affects Version/s: 3.1.0

This jira addresses the below two points:
 1. In {{RemoteBlockPushResolver}}, bytes that cannot be merged immediately are stored in memory. The stream callback maintains a list of {{deferredBufs}}. When a block cannot be merged it is added to this list. Currently, there isn't a limit on the number of pending blocks. We can limit the number of pending blocks in memory. There has been a discussion around this here:
[https://github.com/apache/spark/pull/30062#discussion_r514026014]

2. When a stream doesn't get an opportunity to merge, then {{RemoteBlockPushResolver}} ignores the data from that stream. Another approach is to store the data of the stream in {{AppShufflePartitionInfo}} when it reaches the worst-case scenario. This may increase the memory usage of the shuffle service though. However, given a limit introduced with 1 we can try this out.
 More information can be found in this discussion:
 [https://github.com/apache/spark/pull/30062#discussion_r517524546]
Bug: Group exception messages in execution/command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 08:27
Affects Version/s: 3.2.0

'/core/src/main/scala/org/apache/spark/sql/execution/command'
|| Filename                      ||   Count ||
| AnalyzeColumnCommand.scala    |       3 |
| AnalyzePartitionCommand.scala |       2 |
| AnalyzeTableCommand.scala     |       1 |
| SetCommand.scala              |       2 |
| createDataSourceTables.scala  |       2 |
| ddl.scala                     |       1 |
| functions.scala               |       4 |
| tables.scala                  |       7 |
| views.scala                   |       3 |