av9ash/GitBugs huggingface dataset:
Bug: percentile_approx function doesn't work with pivot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/21 04:17
Affects Version/s: 3.1.1

The percentile_approx PySpark function does not appear to treat the "accuracy" parameter correctly when pivoting on a column, causing the query below to fail (this also fails if the accuracy parameter is left unspecified):
----
{{import pyspark.sql.functions as F}}

{{df = sc.parallelize([}}
 {{    ["a", -1.0],}}
 {{    ["a", 5.5],}}
 {{    ["a", 2.5],}}
 {{    ["b", 3.0],}}
 {{    ["b", 5.2]}}
 {{]).toDF(["type", "value"])}}
 {{    .groupBy()}}
 {{    .pivot("type", ["a", "b"])}}
 {{    .agg(F.percentile_approx("value", [0.5], 10000).alias("percentiles"))}}
----
Error message: 

{{AnalysisException: cannot resolve 'percentile_approx((IF((`type` <=> CAST('a' AS STRING)), `value`, CAST(NULL AS DOUBLE))), (IF((`type` <=> CAST('a' AS STRING)), array(0.5D), NULL)), (IF((`type` <=> CAST('a' AS STRING)), 10000, CAST(NULL AS INT))))' due to data type mismatch: The accuracy or percentage provided must be a constant literal; 'Aggregate [percentile_approx(if ((type#242 <=> cast(a as string))) value#243 else cast(null as double), if ((type#242 <=> cast(a as string))) array(0.5) else cast(null as array<double>), if ((type#242 <=> cast(a as string))) 10000 else cast(null as int), 0, 0) AS a#251, percentile_approx(if ((type#242 <=> cast(b as string))) value#243 else cast(null as double), if ((type#242 <=> cast(b as string))) array(0.5) else cast(null as array<double>), if ((type#242 <=> cast(b as string))) 10000 else cast(null as int), 0, 0) AS b#253|#242 <=> cast(a as string))) value#243 else cast(null as double), if ((type#242 <=> cast(a as string))) array(0.5) else cast(null as array<double>), if ((type#242 <=> cast(a as string))) 10000 else cast(null as int), 0, 0) AS a#251, percentile_approx(if ((type#242 <=> cast(b as string))) value#243 else cast(null as double), if ((type#242 <=> cast(b as string))) array(0.5) else cast(null as array<double>), if ((type#242 <=> cast(b as string))) 10000 else cast(null as int), 0, 0) AS b#253] +- LogicalRDD [type#242, value#243|#242, value#243], false}}
Bug: Not able to capture driver and executor logs in console
Status: Open | Priority: Major | Resolution: 
Created: 18/May/21 14:06
Affects Version/s: 2.2.0

Running the spark job using the below command, but not able to capture the driver and executor logs in console
  # {{# Run on a Spark standalone cluster in cluster deploy mode with supervise}}
 # {{./bin/spark-submit \  --class org.apache.spark.examples.SparkPi \  --master spark://207.184.161.138:7077 \  --deploy-mode cluster \  --supervise \  --executor-memory 20G \  --total-executor-cores 100 \  /path/to/examples.jar \  1000}}
Bug: Enable spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 04:28
Affects Version/s: 3.2.0

SPARK-33407 added the simplification of the stacktraces in Python UDFs which hides unnecessary internal traceback which makes users focus on the exceptions from their own codes.

This JIRA aims to enable it by default.
Bug: parquet read with schema should fail on non-existing columns
Status: Open | Priority: Major | Resolution: 
Created: 12/May/21 13:55
Affects Version/s: 3.0.1

When read schema is specified as I user I would prefer/like if spark failed on missing columns.

{code:python}
from pyspark.sql.dataframe import DoubleType, StructType

spark: SparkSession = ...

spark.read.parquet("/tmp/data.snappy.parquet")
# inferred schema, includes 3 columns: col1, col2, new_col
# DataFrame[col1: bigint, col2: bigint, new_col: bigint]

# let's specify a custom read_schema, with **non nullable** col3 (which is not present):
read_schema = StructType(fields=[StructField("col3",DoubleType(),False)])

df = spark.read.schema(read_schema).parquet("/tmp/data.snappy.parquet")

df.schema
# we get a DataFrame with **nullable** col3:
# StructType(List(StructField(col3,DoubleType,true)))

df.count()
# 0
{code}

Is this a feature or a bug? In this case there's just a single parquet file, I have also tried {{option("mergeSchema", "true")}}, which doesn't help.

Similar read pattern would fail on pandas (and likely dask).
Bug: Improve parameter validation in DataFrame.show
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/21 05:50
Affects Version/s: 3.1.1

Being more used to Scala API the user may  be tempted to call  
{code:python}
df.show(False)
{code}
in PySparkl and she will receive an error message that does not easily map to the user code 
{noformat}
py4j.Py4JException: Method showString([class java.lang.Boolean, class java.lang.Integer, class java.lang.Boolean]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
	at py4j.Gateway.invoke(Gateway.java:274)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748
{noformat}
Bug: Flaky Test: GaussianMixture at ml/clustering.py and Word2Vec at ml/feature.py
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/21 03:27
Affects Version/s: 3.2.0

Some python tests should be updated/removed, because of [SPARK-35295|https://issues.apache.org/jira/browse/SPARK-35295]


[https://github.com/apache/spark/runs/2568540411]
{code:java}
**********************************************************************
File "/__w/spark/spark/python/pyspark/ml/clustering.py", line 276, in __main__.GaussianMixture
Failed example:
    summary.logLikelihood
Expected:
    65.02945...
Got:
    93.36008975083433
**********************************************************************
{code}

and

{code}
**********************************************************************
File "/__w/spark/spark/python/pyspark/ml/feature.py", line 4681, in __main__.Word2Vec
Failed example:
    model.getVectors().show()
Expected:
    +----+--------------------+
    |word|              vector|
    +----+--------------------+
    |   a|[0.09511678665876...|
    |   b|[-1.2028766870498...|
    |   c|[0.30153277516365...|
    +----+--------------------+
    ...
Got:
    +----+--------------------+
    |word|              vector|
    +----+--------------------+
    |   a|[0.09511695802211...|
    |   b|[-1.2028766870498...|
    |   c|[0.30153274536132...|
    +----+--------------------+
    <BLANKLINE>
**********************************************************************
{code}
Bug: Unable to Connect to SSL enabled Oracle RDS Instance using pyspark with spark-submit cluster mode
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/May/21 21:47
Affects Version/s: 3.1.1

We are not able to connect to SSL enabled Oracle RDS instance from pyspark in EMR.

we have created the keystore file and placed in the EMR Master Node and using jdbc connection.

The connectivity is not working when the executing spark-submit in master yarn and cluster deploy mode.

The same keystore file is working fine with master local and client deploy mode.

 

Getting the below error in yarn cluster mode:

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
 at sun.security.ssl.Handshaker$1.run(Handshaker.java:970)
 at sun.security.ssl.Handshaker$1.run(Handshaker.java:967)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459)
 at oracle.net.nt.SSLSocketChannel.runTasks(SSLSocketChannel.java:644)
 at oracle.net.nt.SSLSocketChannel.doSSLHandshake(SSLSocketChannel.java:436)
 ... 31 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
 at sun.security.validator.Validator.validate(Validator.java:262)
 at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
 ... 39 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
 ... 45 more
Bug: Dataframe overwrite on S3A does not delete old files with S3 object-put to table path/
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 04/May/21 00:21
Affects Version/s: 2.2.0

To reproduce:

test_table path: s3a://test_bucket/test_table/

 

df = spark_session.sql("SELECT * FROM test_table")

df.count()  # produce row number 1000

#####S3 operation######

s3 = boto3.client("s3")
 s3.put_object(
     Bucket="test_bucket", Body="", Key=f"test_table/"
 )

#####S3 operation######

df.write.insertInto(test_table, overwrite=True)

#Same goes to df.write.save(mode="overwrite", format="parquet", path="s3a://test_bucket/test_table")

df = spark_session.sql("SELECT * FROM test_table")

df.count()  # produce row number 2000

 

Overwrite is not functioning correctly. Old files will not be deleted on S3.
Bug: Upgrade ZooKeeper to 3.7.0
Status: Resolved | Priority: Major | Resolution: Later
Created: 04/May/21 07:03
Affects Version/s: 3.2.0

Upgrade ZooKeeper to 3.7.0 to fix the vulnerabilities.

*List of CVE's:*
 * CVE-2021-21295
 * CVE-2021-21290
 * CVE-2021-21409
Bug: More clause needed for combining groupby and cube
Status: Open | Priority: Major | Resolution: 
Created: 08/May/21 03:39
Affects Version/s: 3.0.0

As we all know, aggregation clause must follow after groupby, rollup or cube clause in pyspark. I think we should have more features in this part. Because in sql, we can write it like this "group by xxx, xxx, cube(xxx,xxx)". While in pyspark, if you just need cube for one field and group for the others, it's not gonna happen. Using cube for all fields brings much more cost for useless data. So I think we need to improve it. Thank you!
Bug: Override getJDBCType in MySQLDialect to map FloatType to FLOAT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/21 16:09
Affects Version/s: 3.1.1

In [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L165,] FloatType is mapped to REAL. However, MySQL treats REAL as a synonym to DOUBLE by default (see [https://dev.mysql.com/doc/refman/8.0/en/numeric-types.html] for more information). 

`MySQLDialect` could override `getJDBCType` so that it maps FloatType to FLOAT instead.
Bug: Shuffle file leak with external shuffle service enable
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 31/May/21 03:04
Affects Version/s: 3.1.2

Unlike rdd block, external shuffle service doesn't offer a cleaning up of shuffle file. The cleaning up of shuffle file mainly rely on alive executors to response the request from context cleaner. As long as the executor exit, the shuffle file left will not be cleaned until application exits. For streaming application or long running application, disk may run out. 

I'm confused that shuffle file was left like above while the lifecycle of rdd block was properly handled. Is there any difference between them?
Bug: tag v3.0.0 org.apache.spark.sql.catalyst.parser.AstBuilder import error
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 31/May/21 03:53
Affects Version/s: 3.0.0

org.apache.spark.sql.catalyst.parser.AstBuilder:
https://github.com/apache/spark/blob/v3.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

line 36:
`import org.apache.spark.sql.catalyst.parser.SqlBaseParser._`

SqlBaseParser do not exists in pkg `org.apache.spark.sql.catalyst.parser`
https://github.com/apache/spark/tree/v3.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser

also line 54 :
SqlBaseBaseVisitor does not import and could not compile
Bug: Fix a bug in SPARK-35266 that creates benchmark files in the invalid path with the wrong name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/21 09:25
Affects Version/s: 3.2.0

Change in SPARK-35266 has a bug that benchmark files are created in the invalid path with the wrong name.
 e.g. For {{BLASBenchmark}},
 * Desired: Creates {{BLASBenchmark-results.txt}} in \{SPARK_HOME}/mllib-local/benchmarks/
 * Current: Creates {{benchmarksBLASBenchmark-results.txt}} in \{SPARK_HOME}/mllib-local/
Bug: Recover updating build status in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/21 10:05
Affects Version/s: 3.2.0

https://github.com/apache/spark/runs/2709537998?check_suite_focus=true

Currently it fails to get the run id because apparently the author removed workflows. We should make it working even when they remove or disable GA in their forked repository back.
Bug: Make SparkR tests pass with R 4.1+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/21 07:44
Affects Version/s: 3.2.0

Currently, there exists 6 SparkR UT failures in R 4.1.0.
Until R 4.0.5, there was no errors.
{code}
══ Failed ══════════════════════════════════════════════════════════════════════
── 1. Failure (test_sparkSQL_arrow.R:71:3): createDataFrame/collect Arrow optimi
collect(createDataFrame(rdf)) not equal to `expected`.
Component “g”: 'tzone' attributes are inconsistent ('UTC' and '')

── 2. Failure (test_sparkSQL_arrow.R:143:3): dapply() Arrow optimization - type
collect(ret) not equal to `rdf`.
Component “b”: 'tzone' attributes are inconsistent ('UTC' and '')

── 3. Failure (test_sparkSQL_arrow.R:229:3): gapply() Arrow optimization - type
collect(ret) not equal to `rdf`.
Component “b”: 'tzone' attributes are inconsistent ('UTC' and '')

── 4. Error (test_sparkSQL.R:1454:3): column functions ─────────────────────────
Error: (converted from warning) cannot xtfrm data frames
Backtrace:
  1. base::sort(collect(distinct(select(df, input_file_name())))) test_sparkSQL.R:1454:2
  2. base::sort.default(collect(distinct(select(df, input_file_name()))))
  5. base::order(x, na.last = na.last, decreasing = decreasing)
  6. base::lapply(z, function(x) if (is.object(x)) as.vector(xtfrm(x)) else x)
  7. base:::FUN(X[[i]], ...)
 10. base::xtfrm.data.frame(x)

── 5. Failure (test_utils.R:67:3): cleanClosure on R functions ─────────────────
`actual` not equal to `g`.
names for current but not for target
Length mismatch: comparison on first 0 components

── 6. Failure (test_utils.R:80:3): cleanClosure on R functions ─────────────────
`actual` not equal to `g`.
names for current but not for target
Length mismatch: comparison on first 0 components
{code}
Bug: unionByName with null filling fails for some nested structs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/21 14:05
Affects Version/s: 3.1.1

We've encountered a few weird edge cases that seem to fail the new null filling unionByName (which has been a great addition!). It seems to stem from the fields being sorted by name and corrupted along the way. The simple reproduction is:
{code:java}
df = spark.createDataFrame([[]])
df1 = (df
    .withColumn('top', F.struct(
        F.struct(
            F.lit('ba').alias('ba')
        ).alias('b')
    ))
)
df2 = (df
    .withColumn('top', F.struct(
        F.struct(
            F.lit('aa').alias('aa')
        ).alias('a'),
        F.struct(
            F.lit('bb').alias('bb')
        ).alias('b'),
    ))
)
df1.unionByName(df2, True).printSchema()
{code}
This results in the exception:
{code:java}
pyspark.sql.utils.AnalysisException: Union can only be performed on tables with the compatible column types. struct<a:struct<aa:string>,b:struct<ba:string,bb:string>> <> struct<a:struct<aa:string>,b:struct<aa:string,bb:string>> at the first column of the second table;
{code}
You can see in the second schema that it has 
{code:java}
b:struct<aa:string,bb:string>
{code}
when it should be
{code:java}
b:struct<ba:string:bb:string>
{code}
It seems to happen somewhere during [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala#L73,] as everything seems correct up to that point from my testing. It's either modifying one expression during the transformUp then corrupts other expressions that are then modified, or the ExtractValue before the addFieldsInto is remembering the ordinal position in the struct that is then changing and causing issues.

 

I found that simply using sortStructFields instead of sortStructFieldsInWithFields gets things working correctly, but definitely has a performance impact. The deep expr unionByName test takes ~1-2 seconds normally but ~12-15 seconds with this change. I assume because the original method tried to rewrite existing expressions vs the sortStructFields just adds expressions on top of existing ones to project the new order.

I'm not sure if it makes sense to take the slower but works in the edge cases method (assuming it doesn't break other cases, all existing tests pass), or if there's a way to fix the existing method for cases like this.
Bug: Document the deprecation of Koalas Accessor after porting the documentation.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 31/May/21 01:29
Affects Version/s: 3.2.0

As we changed the name from 'koalas' to 'pandas_on_spark', the name of 'Koalas Accessor' is also changed to 'Pandas API on Spark Accessor'.

However, we keep the name of 'koalas' when use Pandas API on Spark Accessor for backward compatibility.

Thus, after porting the documentation, we should add the document that we recommend to use 'pandas_on_spark' rather than 'koalas' when users use the Pandas API on Spark Accessor since the Koalas Accessor is deprecated.
Bug: WARN TaskSchedulerImpl: Initial job has not accepted any resources;
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 21/May/21 18:40
Affects Version/s: 2.4.7

Hello,

I'm trying a select request on spark-sql, but I have "WARN TaskSchedulerImpl: Initial job has not accepted any resources;" message.

I can execute without error : show databases, use mydatabase, show tables ..

spark-env.sh
{quote}rity.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.client=false"


# MEMORY SETTINGS
export SPARK_DAEMON_MEMORY=2g
export SPARK_WORKER_MEMORY=4g

# Worker Directory
export SPARK_WORKER_DIR=$SPARK_HOME/tmp

# Environment variable for printing spark command everytime you run spark.Set to "1" to print.
# export SPARK_PRINT_LAUNCH_COMMAND=1

#UI
export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Djava.library.path=$SPARK_MAPR_HOME/lib"


export SPARK_MASTER_HOST=myhost
export SPARK_MASTER_IP=myhost
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=maprfs:///apps/spark -Djava.library.path=$SPARK_MAPR_HOME/lib"
#daniela
export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=4"
export SPARK_MEM=4g
h6. Thank you very much.

Best regards,

daniela
{quote}
Bug: Pass objects as parameters to SparkSQL UDFs
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/May/21 02:10
Affects Version/s: 3.1.1

You can pass class objects directly to UDFs using the UDF format:

df.select("*").filter(myFunc(classObj)(col("colName")))

but the format:

"""SELECT * FROM view WHERE myFunc(classObj, "colName")"""

or

"""SELECT * FROM view WHERE myFunc(classObj)("colName")"""

does not work. This would be a very useful feature to have, especially being that UDTs are being made public again in 3.2.0
Bug: Critical Vulnerabilities: log4j_log4j 1.2.17 shipped
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/May/21 14:26
Affects Version/s: 3.0.2

Vulnerabilities scanner is highlighting following CRITICAL vulnerabilities in {{spark-3.0.2-bin-hadoop3.2}} coming from obsolete {{log4j_log4j}} {{1.2.17}} :
 * [CVE-2019-17571|https://nvd.nist.gov/vuln/detail/CVE-2019-17571]

This package is shipped within {{jars/log4j-1.2.17.jar}}
Bug: Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/21 07:03
Affects Version/s: 3.2.0

In pyspark, user may create some threads, not via `Thread` object, but via some parallel helper function such as:
`thread_pool.imap_unordered`

In this case, we need create a function wrapper, used in pin thread mode, The wrapper function, before calling original thread target, it inherits the inheritable properties specific to JVM thread such as ``InheritableThreadLocal``, and after original thread target return, garbage-collects the Python thread instance and also closes the connection which finishes JVM thread correctly.
Bug: Sync with Apache Spark branch for all builds such as linter in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/May/21 00:47
Affects Version/s: 3.2.0

We currently just simply checking out https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L471-L472 for some build-only or linters.

This seems like easily casuing logical conflict when the PR is not synced against the latest changes.
Bug: PIP packaging test is skipped in GitHub Actions build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/21 03:29
Affects Version/s: 3.0.2

See https://github.com/apache/spark/runs/2568923639?check_suite_focus=true

{code}
========================================================================
Running PySpark packaging tests
========================================================================
Constructing virtual env for testing
Missing virtualenv & conda, skipping pip installability tests
Cleaning up temporary directory - /tmp/tmp.iILYWISPXW
{code}