av9ash/GitBugs huggingface dataset:
Bug: Not able to run Pandas UDF 
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/20 03:39
Affects Version/s: 3.0.0

Hi,

I am facing issues in running Pandas UDF on a yarn cluster with multiple nodes, I am trying to perform a simple DBSCAN algorithm to multiple groups in my dataframe, to start with, I am just using a simple example to test things out - 
{code:python}
import pandas as pd
from pyspark.sql.types import StructType, StructField, DoubleType, StringType, IntegerType
from sklearn.cluster import DBSCAN
from pyspark.sql.functions import pandas_udf, PandasUDFTypedata 

data = [(1, 11.6133, 48.1075),
         (1, 11.6142, 48.1066),
         (1, 11.6108, 48.1061),
         (1, 11.6207, 48.1192),
         (1, 11.6221, 48.1223),
         (1, 11.5969, 48.1276),
         (2, 11.5995, 48.1258),
         (2, 11.6127, 48.1066),
         (2, 11.6430, 48.1275),
         (2, 11.6368, 48.1278),
         (2, 11.5930, 48.1156)]

df = spark.createDataFrame(data, ["id", "X", "Y"])

output_schema = StructType(
            [
                StructField('id', IntegerType()),
                StructField('X', DoubleType()),
                StructField('Y', DoubleType()),
                StructField('cluster', IntegerType())
             ]
    )

@pandas_udf(output_schema, PandasUDFType.GROUPED_MAP)
def dbscan(data):
    data["cluster"] = DBSCAN(eps=5, min_samples=3).fit_predict(data[["X", "Y"]])
    result = pd.DataFrame(data, columns=["id", "X", "Y", "cluster"])
    return result

res = df.groupby("id").apply(dbscan)


res.show()

{code}
 

The code keeps running forever on the yarn cluster, I expect it to be finished within seconds(this works fine on standalone mode and finishes in 2-4 seconds), on checking the Spark UI, I can see that the Spark job is stuck(99/580) and doesn't make any progress forever.

 

Also it doesn't run in parallel, am I missing something?  !Screenshot 2020-08-31 at 9.04.07 AM.png!

 

 

I am new to Spark, and still trying to understand a lot of things.
Bug: Add column name in _infer_schema error message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 22:54
Affects Version/s: 3.0.1

The current error message from _infer_type in _infer_schema only includes the unsupported column type but not the column name. This ticket suggests adding the column name in the error message to make it easier for users to identify which column should they drop or convert.
Bug: Bucketing and Partitioning information are not passed on to non FileFormat datasource writes
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/20 12:30
Affects Version/s: 3.0.0

When writing to a FileFormat datasource, bucket spec and partition columns are passed into InsertIntoHadoopFsRelationCommand: [https://github.com/apache/spark/blob/v3.0.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L474-L475].

 

However, from what I can tell, the RelationProvider API does not have a way to pass in these: [https://github.com/apache/spark/blob/v3.0.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L511-L513].
Bug: Ambiguous self join error in non self join with window
Status: Closed | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 23:35
Affects Version/s: 3.0.0

Following code fails ambiguous self join analysis, even when it doesn't have self join 

val v1 = spark.range(3).toDF("m")
 val v2 = spark.range(3).toDF("d")
 val v3 = v1.join(v2, v1("m").===(v2("d")))
 val v4 = v3("d");
 val w1 = Window.partitionBy(v4)
 val out = v3.select(v4.as("a"), sum(v4).over(w1).as("b"))

org.apache.spark.sql.AnalysisException: Column a#45L are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.;
Bug: Cant able to process Timestamp 0001-01-01T00:00:00.000+0000 with TimestampType
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Aug/20 16:32
Affects Version/s: 3.0.0

Spark Version : 3.0.0

Below is the sample code to reproduce the problem with TimestampType.
{code:java}
from pyspark.sql.functions import lit
from pyspark.sql.types import TimestampType
df=spark.createDataFrame([(1, 'foo'),(2, 'bar'),],['id', 'txt'])
new_df=df.withColumn("test_timestamp",lit("0001-01-01T00:00:00.000+0000").cast(TimestampType()))
new_df.printSchema()
root
 |-- id: long (nullable = true)
 |-- txt: string (nullable = true)
 |-- test_timestamp: timestamp (nullable = true)

new_df.show()
+---+---+-------------------+
| id|txt|     test_timestamp|
+---+---+-------------------+
|  1|foo|0001-01-01 00:00:00|
|  2|bar|0001-01-01 00:00:00|
+---+---+-------------------+
{code}
 

new_df.rdd.isEmpty() operation is failing with *year 0 is out of range*

 
{code:java}
new_df.rdd.isEmpty()

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure:  Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 177, in _read_with_length
 return self.loads(obj)
 File "/databricks/spark/python/pyspark/serializers.py", line 466, in loads
 return pickle.loads(obj, encoding=encoding)
 File "/databricks/spark/python/pyspark/sql/types.py", line 1415, in <lambda>
 return lambda *a: dataType.fromInternal(a)
 File "/databricks/spark/python/pyspark/sql/types.py", line 635, in fromInternal
 for f, v, c in zip(self.fields, obj, self._needConversion)]
 File "/databricks/spark/python/pyspark/sql/types.py", line 635, in <listcomp>
 for f, v, c in zip(self.fields, obj, self._needConversion)]
 File "/databricks/spark/python/pyspark/sql/types.py", line 447, in fromInternal
 return self.dataType.fromInternal(obj)
 File "/databricks/spark/python/pyspark/sql/types.py", line 201, in fromInternal
 return datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)
ValueError: year 0 is out of range{code}
Bug: FileOutputCommitter warns "No Output found for attempt"
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/20 14:11
Affects Version/s: 2.4.0

Hi team,

This is my first time to report an issue here.

We submitted and ran the spark job on the cluster. 

We found that one of the parquet output partition is missing in the output directory. We checked the spark job log, all the tasks status are showing success. The output record size matches expected number.

However, we checked the container log, found that there was a warning says *No Output found for attempt_20200819094307_0003_m_000002_11*, which stopped moving the output from taskAttemptPath to output directory. As a result, we are missing some of the output rows.

Re-run the job helped to solve the issue, however the report is critical for us. It is appreciated if you can advise the cause for the issue.

 

Below are the container logs:

 
{code:java}
20/08/19 09:44:57 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/08/19 09:44:57 INFO datasources.SQLHadoopMapReduceCommitProtocol: Using user defined output committer class parquet.hadoop.ParquetOutputCommitter
20/08/19 09:44:57 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
20/08/19 09:44:57 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/08/19 09:44:57 INFO datasources.SQLHadoopMapReduceCommitProtocol: Using output committer class parquet.hadoop.ParquetOutputCommitter
20/08/19 09:44:57 INFO codegen.CodeGenerator: Code generated in 12.370642 ms
20/08/19 09:44:57 INFO codegen.CodeGenerator: Code generated in 6.927118 ms
20/08/19 09:44:57 INFO codegen.CodeGenerator: Code generated in 12.004204 ms
20/08/19 09:44:57 INFO parquet.ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
..... (skipped)
20/08/19 09:44:57 WARN output.FileOutputCommitter: No Output found for attempt_20200819094307_0003_m_000002_11
20/08/19 09:44:57 INFO mapred.SparkHadoopMapRedUtil: attempt_20200819094307_0003_m_000002_11: Committed
{code}
Bug: Add Flake8 check for missing imports
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 20:11
Affects Version/s: 3.0.0

Add Flake8 check to detect missing imports. While working on SPARK-17333 I've noticed that we're missing some imports. This PR will enable a check using Flake8. One of the side effects is that we can't use wildcard imports, since Flake8 is unable to figure them out. However, having wildcard imports isn't the best practice since it can be unclear from which wildcard import a specific class is coming from.
Bug: Add extended information - arguments/examples/since/notes of expressions to the remarks field of GetFunctionsOperation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/20 08:30
Affects Version/s: 3.1.0

the remarks field current only provide the usage info but without arguments/examples/since/notes, we should add them to the field. It will make the users easier to learn  how to use those functions
Bug: Distinct Function Weird Bug
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Aug/20 01:20
Affects Version/s: 2.4.6

A weird spark display and counting error. When I was loading in my CSV file into spark and trying to do check all distinct value from a column inside of a dataframe. Everything I try in spark resulted in a wrong answer. But if I convert my spark dataframe into pandas dataframe, it works. Please help. This bug only happens in this one CSV file, all my other CSV files work properly. Here are the pictures.

 

!image-2020-08-01-21-19-06-402.png!!image-2020-08-01-21-19-03-289.png!!image-2020-08-01-21-18-58-625.png!
Bug: Use bloom filter to improve dynamicPartitionPruning
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Aug/20 14:02
Affects Version/s: 3.1.0

It will throw exception when {{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}} is disabled:
{code:sql}
select catalog_sales.* from  catalog_sales join catalog_returns  where cr_order_number = cs_sold_date_sk and cr_returned_time_sk < 40000;
{code}
{noformat}
20/08/16 06:44:42 ERROR TaskSetManager: Total size of serialized results of 494 tasks (1225.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
{noformat}

We can improve it with minimum, maximum and Bloom filter to reduce serialized results.
Bug: spark-sql doesn't support insert into mixed static & dynamic partition 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Aug/20 08:05
Affects Version/s: 2.3.2

spark-sql doesn't support insert into mixed static & dynamic partition, for example:

source table :
{code:java}
CREATE TABLE `id_name`(`id` int, `name` string)
PARTITIONED BY (`dt` string)
{code}
dest table:
{code:java}
CREATE TABLE `id_name_dt1_dt2`(`id` int, `name` string)
PARTITIONED BY (`dt1` string, `dt2` string)
{code}
insert sql:
{code:java}
insert into table tmp.id_name_dt1_dt2 partition(dt1='beijing',dt2) select * from tmp.id_name;
{code}
result:

data not inserted, dest table partition not added.

and there are two warns:
{code:java}
20/08/07 14:32:28 WARN warehouse: Cannot create partition spec from hdfs://nameservice/; missing keys [dt1]
20/08/07 14:32:28 WARN FileOperations: Ignoring invalid DP directory hdfs://nameservice/user/hive/warehouse/tmp.db/id_name_dt1_dt2/.hive-staging_hive_2020-08-07_14-32-02_538_7897451753303149223-1/-ext-10000/dt2=2002
{code}
source code maybe in there :

[Warehouse.java|https://github.com/hortonworks/hive-release/blob/HDP-3.1.4.0-315-tag/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java]

[Utilities.java|https://github.com/hortonworks/hive-release/blob/HDP-3.1.4.0-315-tag/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java]
Bug: thread safe endpoints may hang due to fatal error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/20 04:20
Affects Version/s: 2.3.4

Processing for `ThreadSafeRpcEndpoint` is controlled by 'numActiveThreads' in `Inbox`. Now if any fatal error happens during `Inbox.process`, 'numActiveThreads' is not reduced. Then other threads can not process messages in that inbox, which causes the endpoint to "hang".

This problem is more serious in previous Spark 2.x versions since the driver, executor and block manager endpoints are all thread safe endpoints.
Bug: Show Avro related API in documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/20 10:15
Affects Version/s: 3.0.1

Currently, the Avro related APIs are missing in the documentation https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html . This PR is to:
1. Mark internal Avro related classes as private
2. Show Avro related API in Spark official API documentation
Bug: Eliminate filter clause in aggregate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 07:51
Affects Version/s: 3.1.0

SQL like:

{code:java}
SELECT COUNT(DISTINCT 1) FILTER (WHERE true) FROM testData;
{code}
Could converted to
{code:java}
SELECT COUNT(DISTINCT 1) FROM testData;
{code}
Bug: Applications with Maven dependencies fail when submitted in cluster mode on K8s
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/20 13:18
Affects Version/s: 3.0.0



A Spark application with Maven dependencies (specified with {{--packages}}) submitted in cluster mode on Kubernetes fails with a {{ClassNotFoundException}} because the executors do not have the dependencies installed.

I think this has been introduced by SPARK-23153 (in SparkSubmit.scala). Now the executors are not aware that they should request the resolved Maven dependencies from the driver. 

Sample {{spark-submit}}:
{code:bash}
spark-submit \
    --deploy-mode cluster \
    --master k8s://https://kubernetes.docker.internal:6443 \
    --conf spark.kubernetes.container.image=some_spark3_image \
    --conf spark.kubernetes.driver.container.image=some_spark3_image \
    --conf spark.driver.cores=1 \
    --conf spark.driver.memory=512m \
    --conf spark.executor.cores=1 \
    --conf spark.executor.memory=512m \
    --conf spark.executor.instances=1 \
    --conf spark.jars.packages=org:artifact_2.12:1.0.0 \
    http://hostname/app_2.12-0.1.0.jar
{code}

If {{app}} imports any code from {{org.artifact}} the executors will fail with a {{ClassNotFoundException}}
Bug: Spark application failed due to stage fatch failed without retry
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 06/Aug/20 01:38
Affects Version/s: 2.3.4

We got a exception when running a spark application under spark 2.3.4 and spark 3.0 using conf : *spark.shuffle.useOldFetchProtocol=true*, the application failed due to stage fatch failed and the stage not retry.
code like following:
{code:java}
val Array(input) = args

val sparkConf = new SparkConf().setAppName("Spark Fatch Failed Test")
// for running directly in IDE
sparkConf.setIfMissing("spark.master", "local[2]")
val sc = new SparkContext(sparkConf)

val lines = sc.textFile(input)
  .repartition(1)
  .map(data => data.trim)
  .repartition(1)

val doc = lines.map(data => (data, 1)).reduceByKey(_ + _).collect(){code}
The application DAG like following:
 
!https://i.stack.imgur.com/0TfZW.png!
If stage 3 failed due to fatch failed,  the application will not retry stage 2 and stage 3 and fail the job. Because spark think stage 2 and stage 3 are non-retryable, rdds in stage 2 and stage 3 is *INDETERMINATE.*
 
Actually, if shuffle result belongs to stage 1 exist completely, stage 2 and stage 3 are retryable, because rdds in them is not order-sensitive. If allow stage 2 and stage 3 to retry, we have trouble in handling *DAGScheduler.getMissingParentStages.* And i am not sure if *DAGScheduler.getMissingParentStages* breaks the rule that *INDETERMINATE* rdd non-retryable.
 
I would appreciate it if someone would reply.
Bug: DecommissionWorkerSuite has started failing sporadically again
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 22:58
Affects Version/s: 3.1.0

Test "decommission workers ensure that fetch failures lead to rerun" is failing: 

 

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127357/testReport/org.apache.spark.deploy/DecommissionWorkerSuite/decommission_workers_ensure_that_fetch_failures_lead_to_rerun/]

https://github.com/apache/spark/pull/29367/checks?check_run_id=972990200#step:14:13579
Bug: HiveSerDeReadWriteSuite and ScriptTransformationSuite are currently failed under hive1.2 profile in branch-3.0 and master
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 23/Aug/20 02:06
Affects Version/s: 3.0.1

There are three tests which are currently failed under hive1.2 profiles in branch-3.0 and master branches:

org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite.Read/Write Hive PARQUET serde table

org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite.Read/Write Hive TEXTFILE serde table

org.apache.spark.sql.hive.execution.ScriptTransformationSuite.SPARK-32608: Script Transform ROW FORMAT DELIMIT value should format value

Please see [https://github.com/apache/spark/pull/29517].


This test is failed under hive1.2 profiles in master branch:
org.apache.spark.sql.hive.orc.HiveOrcHadoopFsRelationSuite.save()/load() - partitioned table - simple queries - partition columns in data
Bug: Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/20 15:10
Affects Version/s: 3.1.0

We can save the host directly in the ExecutorDecommissionState. Therefore, when the executor lost, we could unregister the shuffle map status on the host. Thus, we don't need to hold the cache to wait for FetchFailureException to do the unregister.
Bug: Update commons-crypto to v1.1.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/20 12:04
Affects Version/s: 2.4.7

Tests of org.apache.spark.DistributedSuite are failed on arm64 jenkins: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/ 

    - caching in memory and disk, replicated (encryption = on) (with replication as stream) *** FAILED ***
  3 did not equal 2; got 3 replicas instead of 2 (DistributedSuite.scala:191)
    - caching in memory and disk, serialized, replicated (encryption = on) (with replication as stream) *** FAILED ***
  3 did not equal 2; got 3 replicas instead of 2 (DistributedSuite.scala:191)
    - caching in memory, serialized, replicated (encryption = on) (with replication as stream) *** FAILED ***
  3 did not equal 2; got 3 replicas instead of 2 (DistributedSuite.scala:191)
    .......
Bug: Script transform hive serde default field.delimit is '\t'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 15:56
Affects Version/s: 3.0.0

 
{code:java}
select split(value, "\t") from (
SELECT TRANSFORM(a, b, c, null)
USING 'cat' 
FROM (select 1 as a, 2 as b, 3  as c) t
) temp;

result is :
_c0
["2","3","\\N"]{code}
 
{code:java}
select split(value, "\t") from (
SELECT TRANSFORM(a, b, c, null)
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
USING 'cat' 
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
  WITH SERDEPROPERTIES (
   'serialization.last.column.takes.rest' = 'true'
  )
FROM (select 1 as a, 2 as b, 3  as c) t
) temp;


result is :
_c0
["2","3","\\N"]{code}
 

 

 
{code:java}
select split(value, "\t") from (
SELECT TRANSFORM(a, b, c, null)
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
USING 'cat' 
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
FROM (select 1 as a, 2 as b, 3  as c) t
) temp;

result is :
_c0 
["2"]
{code}
Bug: Simplify if clauses with null and boolean
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/20 01:46
Affects Version/s: 3.0.0

The following if clause:
{code:sql}
if(p, null, false) 
{code}
can be simplified to:
{code:sql}
and(p, null)
{code}

And similarly, the following clause:
{code:sql}
if(p, null, true)
{code}
can be simplified to:
{code:sql}
or(not(p), null)
{code}

iff predicate {{p}} is deterministic, i.e., can be evaluated to either true or false, but not null.

{{and}} and {{or}} clauses are more optimization friendly. For instance, by converting {{if(col > 42, null, false)}} to {{and(col > 42, null)}}, we can potentially push the filter {{col > 42}} down to data sources to avoid unnecessary IO.
Bug: CTAS with V2 catalog wrongly accessed unresolved query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 08:47
Affects Version/s: 3.0.0

Case:

{{CREATE TABLE t USING delta AS SELECT * from nonexist }}

 

Expected:

throw AnalysisException with "Table or view not found"

 

Actual:

{{throw UnresolvedException with "org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolved object, tree: *"}}
Bug: Pushdown cast to data sources
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Aug/20 20:48
Affects Version/s: 3.0.0

Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet:

{code}
scala> case class Person(name: String, age: Short)
scala> Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
scala> val personDS = spark.read.parquet("/tmp/person.parquet")
scala> personDS.createOrReplaceTempView("person")
scala> spark.sql("SELECT * FROM person where age < 30")
{code}

The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well.
Bug: CREATE/REPLACE TABLE AS SELECT not support multi-part identifiers
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Aug/20 16:56
Affects Version/s: 3.1.0

{code:java}
== SQL ==
CREATE TABLE h2.test.abc AS SELECT * FROM h2.test.people
^^^

org.apache.spark.sql.catalyst.parser.ParseException: 
Operation not allowed: CREATE TABLE ... STORED AS ... does not support multi-part identifiers
{code}
{code:java}
== SQL ==
CREATE OR REPLACE TABLE h2.test.abc AS SELECT 1 as col
------------------------------------^^^

org.apache.spark.sql.catalyst.parser.ParseException: 
mismatched input 'AS' expecting {'(', 'USING'}
{code}