av9ash/GitBugs huggingface dataset:
Bug: Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/22 13:33
Affects Version/s: 2.3.0

I'm trying to load data from Teradata, the code using is:
    

    sparkSession.read
          .format("jdbc")
          .options(
            Map(
              "url" -> "jdbc:teradata://hostname, user=$username, password=$password",
              "MAYBENULL" -> "ON",
              "SIP_SUPPORT" -> "ON",
              "driver" -> "com.teradata.jdbc.TeraDriver",
              "dbtable" -> $table_name
            )
          )
          .load()

However, some data lost its fractional part after loading. To be more concise, the column in Teradata is in the type of [Number][1] and after loading, the data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no digits after decimal point. 

Data in Teradata is something like,

    id column1 column2
    1   50.23    100.23
    2   25.8     20.669
    3   30.2     19.23

The `dataframe` of Spark is like,

    id column1 column2
    1   50     100
    2   26     21
    3   30     19

The meta data of the table in Teradata is like:

    CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER) PRIMARY INDEX (id);

The Spark version is 2.3.0/3.2.1 and Teradata is 16.20.32.59.
Bug: How to enable encryption for checkpoint data?
Status: Closed | Priority: Major | Resolution: Invalid
Created: 08/Apr/22 11:32
Affects Version/s: 3.2.1

Setting  spark.io.encryption.enabled to true as described here:
[https://spark.apache.org/docs/latest/security.html#local-storage-encryption |https://spark.apache.org/docs/latest/security.html#local-storage-encryption]has no effect at all on checkpoints.
Bug: Explicitly document the supported pandas version.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/22 05:12
Affects Version/s: 3.3.0

pandas has different behavior per each version for some APIs.

So, we should explicitly follow one pandas version for one pandas-on-Spark version and document it.

For example, if some APIs follow the behavior of pandas 1.3 whereas some APIs follow the behavior of pandas 1.4, users may be confused.
Bug: Unable to initialize the empty pyspark.pandas dataframe
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 06/Apr/22 13:36
Affects Version/s: 3.2.1

I am trying to replace pandas library with pyspark.pandas library. But after the replacement the below line of code failed - 
{code:java}
import pyspark.pandas as pd 

self._df = pd.DataFrame()
 {code}
 
It throws the below error : 
 
{code:java}
    self._df = pd.DataFrame()
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\frame.py", line 520, in __init__        
    internal = InternalFrame.from_pandas(pdf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\internal.py", line 1464, in from_pandas 
    sdf = default_session().createDataFrame(pdf, schema=schema)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\pandas\utils.py", line 477, in default_session 
    return builder.getOrCreate()
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\sql\session.py", line 228, in getOrCreate      
    sc = SparkContext.getOrCreate(sparkConf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 392, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 144, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\context.py", line 339, in _ensure_initialized  
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\site-packages\pyspark\java_gateway.py", line 101, in launch_gateway  
    proc = Popen(command, **popen_kwargs)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "C:\Users\eapasnr\Anaconda3\envs\oden2\lib\subprocess.py", line 1207, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified {code}
The code was working fine previously with Pandas
Bug: ISO-8859-1 encoding doesn't work for text format
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Apr/22 07:35
Affects Version/s: 3.2.1

When reading text files from spark which are not in UTF-8 charset it doesn't work well for foreign language characters (for ex. French chars like è and é). They are all replaced by �. In my case the text files were in ISO-8859-1 encoding.

After digging into docs, it seems that spark still uses Hadoop's LineRecordReader class for text format which only supports UTF-8. Here's the source code of that class: [LineRecordReader.java|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java#L154]

 

You can see this issue in the screenshot below:

!Screenshot 2022-04-06 at 09.29.24.png!

As you can see the French word *données* is read as {*}donn�es{*}. The work *Clôturé* is read as *Cl�tur�.*

 

I also read the same text file as CSV format while providing the correct charset value and it works fine in this case as you can see the screenshot below:

!Screenshot 2022-04-06 at 09.30.02.png!

 

So this issue is specifically for text format. Therefore reporting this issue.
Bug: Add StreamingQueryListener support in PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/22 10:07
Affects Version/s: 3.4.0

PySpark currently does not have the support of {{StreamingQueryListener}} in PySpark whereas DStream has it. This feature is important especially with {{Dataset.observe}} so users can monitor what's going on in their queries.
Bug: Generates a new dataframe instead of operating inplace in setitem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/22 12:44
Affects Version/s: 3.4.0

{code:java}
DataFrameTest.test_eval
DataFrameTest.test_update
DataFrameTest.test_inplace
DataFrameTest.test_fillna{code}
Bug: SparkRuntimeException when trying to get non-existent key in a map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/22 01:28
Affects Version/s: 3.3.0

[~maxgekk] submitted a [commit|https://github.com/apache/spark/commit/bc8c264851457d8ef59f5b332c79296651ec5d1e] that tries to convert the key to SQL but that part of the code is blowing up. 


{code:java}
scala> :pa
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.Row
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.types.StringType
import org.apache.spark.sql.types.DataTypes

val arrayStructureData = Seq(
Row(Map("hair"->"black", "eye"->"brown")),
Row(Map("hair"->"blond", "eye"->"blue")),
Row(Map()))

val mapType  = DataTypes.createMapType(StringType,StringType)

val arrayStructureSchema = new StructType()
.add("properties", mapType)



val mapTypeDF = spark.createDataFrame(
    spark.sparkContext.parallelize(arrayStructureData),arrayStructureSchema)

mapTypeDF.selectExpr("element_at(properties, 'hair')").show

// Exiting paste mode, now interpreting.

+----------------------------+
|element_at(properties, hair)|
+----------------------------+
|                       black|
|                       blond|
|                        null|
+----------------------------+

scala>     spark.conf.set("spark.sql.ansi.enabled", true)

scala> mapTypeDF.selectExpr("element_at(properties, 'hair')").show
22/04/25 18:26:01 ERROR Executor: Exception in task 6.0 in stage 5.0 (TID 23)
org.apache.spark.SparkRuntimeException: The feature is not supported: literal for 'hair' of class org.apache.spark.unsafe.types.UTF8String.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:240) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:101) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.errors.QueryErrorsBase.toSQLValue(QueryErrorsBase.scala:44) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.errors.QueryErrorsBase.toSQLValue$(QueryErrorsBase.scala:43) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.errors.QueryExecutionErrors$.toSQLValue(QueryExecutionErrors.scala:69) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]

{code}

Seems like it's trying to convert UTF8String to a sql literal
Bug: Nested column pruning should filter out attributes that do not belong to the current relation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/22 20:47
Affects Version/s: 3.3.0

`SchemaPruning` currently does not check if the root field of a nested column belongs to the current relation. This can happen when the filter contains correlated subqueries, where the children field can contain attributes from both the inner and the outer query.
Bug: Support spark.kubernetes.test.(driver|executor)RequestCores
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/22 08:23
Affects Version/s: 3.4.0

[https://github.com/apache/spark/pull/35830#pullrequestreview-929597027]

 

Support spark.kubernetes.test.(driver|executor)RequestCores to allow devs setting specific cpu for driver/executor.
Bug: Unpivot / melt function for Dataset API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/22 20:06
Affects Version/s: 3.4.0

As pointed out in SPARK-30273 and SPARK-37799, the Dataset API provides the {{pivot}} method but not its reverse operation {{melt}}. It is only provided as part of the PySpark Pandas API, implemented in Python. However, {{melt}} is not available via the PySpark DataFrame API, nor through Spark Scala Dataset API.

As mentioned in [#26912|https://github.com/apache/spark/pull/26912#pullrequestreview-332975715], it should be implemented in Scala and invoked from PySpark.
Bug: Implement sharing of cloud credentials among driver and executors
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Apr/22 21:44
Affects Version/s: 3.4.0

Currently Spark uses external implementations (e.g. hadoop-aws) to access cloud services like S3. In order to access the actual service, these implementations use credentials provider implementations that obtain credentials to allow access to the cloud service.
These credentials are typically session credentials, which means that they expire after a fixed time. Sometimes, this expiry can be only an hour and for a spark job that runs for many hours (or spark streaming job that runs continuously), the credentials have to be renewed periodically.
In many organizations, the process of getting credentials may multi-step. The organization has an identity provider service that provides authentication for the user, while the cloud service provider provides authorization for the roles the user has access to. Once the user is authenticated and her role verified, the credentials are generated for a new session.
In a large setup with hundreds of Spark jobs and thousands of executors, each executor is then spending a lot of time getting credentials and this may put unnecessary load on the backend authentication services.
The alleviate this, we can use Spark's architecture to obtain the credentials once in the driver and push the credentials to the executors. In addition, the driver can check the expiry of the credentials and push updated credentials to the executors. This is relatively easy to do since the rpc mechanism to implement this is already in place and is used similarly for Kerberos delegation tokens.
Bug: Refactor DataSourceScanExec code to 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/22 21:20
Affects Version/s: 3.2.2

Currently the code for `FileSourceScanExec` class, the physical node for the file scans is quite complex and lengthy. The class should be refactored into a trait `FileSourceScanLike` which implements basic functionality like metrics and file listing. The execution specific code can then live inside `FileSourceScanExec` which will subclass `FileSourceScanLike`.
Bug: Possible correctness issue on stream-stream join when handling edge case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/22 05:10
Affects Version/s: 3.2.1

There was an issue on NPE in stream-stream join. SPARK-35659 fixed the issue “partially”, and the part of fix is to ignore the null value from the last index on swapping elements in the list so the null value in the last index is going to be effectively dropped. If it is due to out of sync between numValues and the actual number of elements, this works effectively as a correction.

This unfortunately opens the possibility of another “correctness” issue; the reason we swap the value with last index is effectively to remove the value in the current index. Doing nothing in any case would mean “we don’t remove the value in the current index”, whereas the caller would expect the value as dropped, and even for outer join they may be emitted as left/right null join output while the value can be re-evaluated and emitted again.
Bug: Standardize Statistical Functions of pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Done
Created: 30/Apr/22 00:17
Affects Version/s: 3.4.0

Statistical functions are the most commonly-used functions in Data Engineering and Data Analysis.

Spark and pandas provide statistical functions in the context of SQL and Data Science separately.

pandas API on Spark implements the pandas API on top of Apache Spark. Although there may be semantic differences of certain functions due to the high cost of big data calculations, for example, median. We should still try to reach the parity from the API level.

However, critical parameters, such as `skipna`,  of statistical functions are missing of basic objects: DataFrame, Series, and Index are missing. 

There is even a larger gap between statistical functions of pandas-on-Spark GroupBy objects and those of pandas GroupBy objects. In addition, tests coverage is far from perfect.

With statistical functions standardized, pandas API coverage will be increased since missing parameters will be implemented. That would further improve the user adoption.

See details at https://docs.google.com/document/d/1IHUQkSVMPWiK8Jhe0GUtMHnDS6LB4_z9K2ktWmORSSg/edit?usp=sharing.
Bug: Add support for YARN decommissioning when ESS is Disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/22 06:55
Affects Version/s: 3.2.1

Subtask to handle Yarn Executor Decommissioning when Shuffle Service is Disabled.

This relates to [SPARK-30835|https://issues.apache.org/jira/browse/SPARK-30835]
Bug: Add Standalone backend support for Stage Level Scheduling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/22 01:16
Affects Version/s: 3.2.0

We should add Standalone backend support for Stage Level Scheduling:
* The Master should be able to generate executors for multiple ResouceProfiles, currently it only considers available CPUs;
* The Worker need let the executor know about its ResourceProfile.
Bug: when i clean data ,I hope one rdd spill two rdd according clean data rule
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 07/Apr/22 01:17
Affects Version/s: 3.2.1

when id do clean data,one rdd according one value(>or <) filter data, and then generate two different set，one is error data file， another is errorless data file.

Now I use filter, but this filter must have two spark dag job, that cost too much.

exactly some code like iterator.span(preidicate) and then return one tuple(iter1,iter2)

one dataset will be spilted tow dataset in one rule data clean progress.

i hope compute once not twice.
Bug: update guava to 30.1.1-jre
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Apr/22 01:38
Affects Version/s: 3.0.0

Update guava to 30.1.1-jre

guava 14.0.1 has risk:
 * [CVE-2020-8908|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908]
 * [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237]
Bug: PythonException - socke.timeout: timed out - socket.py line 707
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Apr/22 09:53
Affects Version/s: 3.2.1

I have a database of about 8 million residential addresses address. I perform 3 separate cleaning operations on the data using udf's and regular expressions (python re package). Then I create an additional column by splitting the 'cleaned' address by commas and then taking the object in the last index as the suburb and use this column as a key to joining the original data frame to a supplementary 1 which contains suburb and country pairs, joining on suburb and then finally create another column containing the final address with the 'unsplit clean' address column concatenated with the country column pulled in by the join. 

When I try to display the result by calling show, I get the desired result if I show only the first 1000 records or less, however if I try to show more records or I add an additional filter to only display records that have been modified, I get a socket timeout error.

I have tried to increase the socket's send and receive buffer sizes to the maximum of 1048576 bytes, as well as increasing the spark executor heartbeat interval (7200s )as well as the spark network timeout (3600s); and I have tried repartitioning the data to 16 and 32 partitions, all of which have had no impact on the result.
Bug: Simplify another conditionals case in predicate
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Apr/22 11:30
Affects Version/s: 3.4.0


{code:scala}
sql(
  """
    |CREATE TABLE t1 (
    |  id DECIMAL(18,0),
    |  event_dt DATE,
    |  cmpgn_run_dt DATE)
    |USING parquet
    |PARTITIONED BY (cmpgn_run_dt)
  """.stripMargin)

sql(
  """
    |select count(*)
    |from t1
    |where CMPGN_RUN_DT >= date_sub(EVENT_DT,2) and CMPGN_RUN_DT <= EVENT_DT
    |and EVENT_DT ='2022-04-05'

    |;
  """.stripMargin).explain(true)
{code}

Excepted:
{noformat}
== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#4L]
+- Project
   +- Filter (((isnotnull(CMPGN_RUN_DT#3) AND (CMPGN_RUN_DT#3 >= 2022-04-03)) AND (CMPGN_RUN_DT#3 <= 2022-04-05)) AND (EVENT_DT#2 = 2022-04-05))
      +- Relation default.t1[id#1,event_dt#2,cmpgn_run_dt#3] parquet

== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#4L])
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#31]
   +- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#7L])
      +- *(1) Project
         +- *(1) Filter (EVENT_DT#2 = 2022-04-05)
            +- *(1) ColumnarToRow
               +- FileScan parquet default.t1[event_dt#2,cmpgn_run_dt#3] Batched: true, DataFilters: [(event_dt#2 = 2022-04-05)], Format: Parquet, Location: InMemoryFileIndex[], PartitionFilters: [isnotnull(cmpgn_run_dt#3), (cmpgn_run_dt#3 >= 2022-04-03), (cmpgn_run_dt#3 <= 2022-04-05)], PushedFilters: [EqualTo(event_dt,2022-04-05)], ReadSchema: struct<event_dt:date>, UsedIndexes: []

{noformat}
Bug: DS V2 supports push down misc non-aggregate functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/22 10:20
Affects Version/s: 3.4.0

Currently, Spark have a lot misc non-aggregate functions of ANSI standard.
abs,
coalesce,
nullif,
when

DS V2 should supports push down these misc non-aggregate functions
Bug: DS V2 Top N push-down supports order by expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/22 06:45
Affects Version/s: 3.4.0

Currently, Spark DS V2 Top N push-down only supports order by column.
But the SQL show below is very useful and common.
SELECT CASE WHEN ("SALARY" > 8000.00) AND ("SALARY" < 10000.00) THEN "SALARY" ELSE 0.00 END AS key, dept, name FROM "test"."employee" ORDER BY key
Bug: DS V2 aggregate push-down supports group by expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/22 11:19
Affects Version/s: 3.4.0

Currently, Spark DS V2 aggregate push-down only supports group by column.
But the SQL show below is very useful and common.
SELECT CASE WHEN ("SALARY" > 8000.00) AND ("SALARY" < 10000.00) THEN "SALARY" ELSE 0.00 END AS key, SUM("SALARY") FROM "test"."employee"   GROUP BY key
Bug: DS V2 Limit push-down should avoid out of memory
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 29/Apr/22 08:32
Affects Version/s: 3.4.0

Currently, Spark DS V2 supports push down Limit operator to data source.
But the behavior only controlled by pushDownList option.
If the limit is very large, then Executor will pull all the result set from data source.
So it will cause the memory issue as you know.
Bug: Error when starting spark shell on Windows system
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/22 13:59
Affects Version/s: 3.2.1

Using the release version of spark-3.2.1  and the default configuration, starting spark shell on Windows system fails. (spark 3.1.2 doesn't show this issue)

Here is the stack trace of the exception:
{code:java}
22/04/06 21:47:45 ERROR SparkContext: Error initializing SparkContext.
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        ...
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.URISyntaxException: Illegal character in path at index 30: spark://192.168.X.X:56964/F:\classes
        at java.net.URI$Parser.fail(URI.java:2845)
        at java.net.URI$Parser.checkChars(URI.java:3018)
        at java.net.URI$Parser.parseHierarchical(URI.java:3102)
        at java.net.URI$Parser.parse(URI.java:3050)
        at java.net.URI.<init>(URI.java:588)
        at org.apache.spark.repl.ExecutorClassLoader.<init>(ExecutorClassLoader.scala:57)
        ... 70 more
22/04/06 21:47:45 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
            ... {code}