av9ash/GitBugs huggingface dataset:
Bug: pyspark.pandas.read_csv parses dates, but docs state otherwise
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/22 10:45
Affects Version/s: 3.3.1

from [https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.read_csv.html] :
{quote}parse_dates:
boolean or list of ints or names or list of lists or dict, default False.
Currently only False is allowed.
{quote}
This documentation suggests that dates are never parsed, but apparently they are always parsed (and it can not be disabled):
{code:python}
import pyspark.pandas
df = pyspark.pandas.read_csv("data.csv", parse_dates=False)
print(df)
print(df.dtypes)
{code}
with this data
{code:java}
date,feature_index,band_0,band_1,band_2
2021-01-05T01:00:00.000+01:00,2,5.0,4.5,3.75
2021-01-05T01:00:00.000+01:00,0,5.0,1.0,2.25
2021-01-05T01:00:00.000+01:00,1,5.0,3.5,4.0
2021-01-15T01:00:00.000+01:00,2,15.0,4.5,3.75
2021-01-15T01:00:00.000+01:00,0,15.0,1.0,2.25
{code}
gives
{code:java}
                 date  feature_index  band_0  band_1  band_2
0 2021-01-05 01:00:00              2     5.0     4.5    3.75
1 2021-01-05 01:00:00              0     5.0     1.0    2.25
2 2021-01-05 01:00:00              1     5.0     3.5    4.00
3 2021-01-15 01:00:00              2    15.0     4.5    3.75
4 2021-01-15 01:00:00              0    15.0     1.0    2.25
date             datetime64[ns]
feature_index             int32
band_0                  float64
band_1                  float64
band_2                  float64
dtype: object
{code}
Notice how the dates are parsed (e.g.  dtype {{datetime64[ns]}} for {{date}})
Bug: Need to Speed ​​up unpacking when reading the next group in VectorizedRleValuesReader 
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/22 07:56
Affects Version/s: 3.4.0

In VectorizedRleValuesReader, BytePacker#unpack8Values() with ByteBuffer input API will bring performance degradation, compared to this function with Array input API.
Bug: Cleanup github action warning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/22 14:46
Affects Version/s: 3.4.0

* The `save-state` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
* The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
* Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout, docker/setup-qemu-action, docker/setup-buildx-action, docker/build-push-action, docker/build-push-action, docker/setup-buildx-action, actions/checkout
* ....

like: https://github.com/apache/spark/actions/runs/3288503290

In order to reduce pontential impact on infra, we'd better address them one by one according actions type.
Bug: Fix broadcasts in Python UDFs when encryption is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/22 14:08
Affects Version/s: 3.4.0

The following Pyspark script:
{noformat}
bin/pyspark --conf spark.io.encryption.enabled=true

...

bar = {"a": "aa", "b": "bb"}
foo = spark.sparkContext.broadcast(bar)
spark.udf.register("MYUDF", lambda x: foo.value[x] if x else "")
spark.sql("SELECT MYUDF('a') AS a, MYUDF('b') AS b").collect()
{noformat}
fails with:
{noformat}
22/10/21 17:14:32 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/Users/petertoth/git/apache/spark/python/lib/pyspark.zip/pyspark/worker.py", line 811, in main
    func, profiler, deserializer, serializer = read_command(pickleSer, infile)
  File "/Users/petertoth/git/apache/spark/python/lib/pyspark.zip/pyspark/worker.py", line 87, in read_command
    command = serializer._read_with_length(file)
  File "/Users/petertoth/git/apache/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 173, in _read_with_length
    return self.loads(obj)
  File "/Users/petertoth/git/apache/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 471, in loads
    return cloudpickle.loads(obj, encoding=encoding)
EOFError: Ran out of input
{noformat}
Bug: Upgrade fabric8io - kubernetes-client to 6.2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/22 08:25
Affects Version/s: 3.4.0

[Release notes|https://github.com/fabric8io/kubernetes-client/releases/tag/v6.2.0]

[Snakeyaml version should be updated to mitigate CVE-2022-28857|https://github.com/fabric8io/kubernetes-client/issues/4383]
Bug: Change default serialization from 'broken' CSV to Spark DF JSON
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/22 08:12
Affects Version/s: 3.4.0

Currently, Spark Connect uses a mediocre version of CSV serialization for the results from the server side without proper batching.

 

We should at least migrate to the existing Spark DF JSON serialization and batch the results.
Bug: Enhance JDBC Connector to use PreparedStatement.getMetaData() to resolve schema instead of PreparedStatement.executeQuery()
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Oct/22 19:45
Affects Version/s: 2.1.0

Currently, Spark JDBC Connector uses *PreparedStatement.executeQuery()* to resolve the JDBCRelation's schema. The schema query is like *s"SELECT * FROM $table_or_query WHERE 1=0".*

But it is not necessary to execute the query. It's enough to *prepare* the query. With preparing the statement, the query is parsed and compiled, but is not executed. It will be more efficient.

So, it's better to use PreparedStatement.getMetaData() to resolve schema.
Bug: Infer schema for CSV files - wrong behavior using header + merge schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/22 08:24
Affects Version/s: 3.2.2

Hello. 
I am writing unit-tests to some functionality in my application that reading data from CSV files using Spark.

I am reading the data using:


{code:java}
header=True
mergeSchema=True
inferSchema=True{code}
When I am reading this single file:
{code:java}
File1:
"int_col","string_col","decimal_col","date_col"
1,"hello",1.43,2022-02-23
2,"world",5.534,2021-05-05
3,"my name",86.455,2011-08-15
4,"is ohad",6.234,2002-03-22{code}
I am getting this schema:
{code:java}
int_col=int
string_col=string
decimal_col=double
date_col=string{code}




When I am duplicating this file, I am getting the same schema.

The strange part is when I am adding new int column, it looks like spark is getting confused and think that the column that already identified as int are now string:


{code:java}
File1:
"int_col","string_col","decimal_col","date_col"
1,"hello",1.43,2022-02-23
2,"world",5.534,2021-05-05
3,"my name",86.455,2011-08-15
4,"is ohad",6.234,2002-03-22
File2:
"int_col","string_col","decimal_col","date_col","int2_col"
1,"hello",1.43,2022-02-23,234
2,"world",5.534,2021-05-05,5
3,"my name",86.455,2011-08-15,32
4,"is ohad",6.234,2002-03-22,2
{code}
result:
{code:java}
int_col=string
string_col=string
decimal_col=string
date_col=string
int2_col=int{code}




When I am reading only the second file, it looks fine:
{code:java}
File2:
"int_col","string_col","decimal_col","date_col","int2_col"
1,"hello",1.43,2022-02-23,234
2,"world",5.534,2021-05-05,5
3,"my name",86.455,2011-08-15,32
4,"is ohad",6.234,2002-03-22,2{code}
result:
{code:java}
int_col=int
string_col=string
decimal_col=double
date_col=string
int2_col=int{code}
For conclusion, it looks like there is a bug mixing the two features: header recognition and merge schema.
Bug: Force grouped import onto single line with Scalafmt
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/22 07:10
Affects Version/s: 3.4.0

When using the scalafmt.conf file as provided by the project repository to organize and optimize imports there is a case in which un-desired behavior appears.

If the import group does not fit on a single line, Scalafmt will by default try to bin-pack and then break out over mulitple lines.

For example:
import org.apache.spark.sql.catalyst.analysis.\{UnresolvedAlias, UnresolvedAttribute, UnresolvedFunction, UnresolvedRelation, UnresolvedStar}{{{}{}}}will become
import org.apache.spark.sql.catalyst.analysis.\{
  UnresolvedAlias,
  UnresolvedAttributed,
  ...
}{{{}{}}}In previous code reviews this has been marked as departing from the consistency of the Spark code base even when using the provided scalafmt.conf.
 
Scalafmt allows to configure this by setting an option 
 
{code:java}
importSelectors = "singleLine"{code}
Bug: Upgrade the guava defined in `plugins.sbt` to `31.0.1-jre`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/22 05:55
Affects Version/s: 3.4.0

SPARK-40071 upgrade checkstyle to 9.3 and checkstyle 9.3uses guava 31.0.1-jre
Bug: How to set  BlockManager info of hostname as ipaddress
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Oct/22 17:57
Affects Version/s: 2.4.3

 
{code:java}
2022-10-09 17:22:42.517 [INFO ] [YARN application state monitor          ] o.a.s.u.SparkUI (54) [logInfo] - Stopped Spark web UI at http://linkis-demo-cg-engineconnmanager-76778ff4b5-sf9xz.linkis-demo-cg-engineconnmanager-headless.linkis.svc.cluster.local:4040
2022-10-09 17:46:09.854 [INFO ] [main                                    ] o.a.s.s.BlockManager (54) [logInfo] - Initialized BlockManager: BlockManagerId(driver, linkis-demo-cg-engineconnmanager-76778ff4b5-sf9xz.linkis-demo-cg-engineconnmanager-headless.linkis.svc.cluster.local, 38798, None) 

{code}
I want to  repleace canonicalHostName(linkis-demo-cg-engineconnmanager-76778ff4b5-sf9xz.linkis-demo-cg-engineconnmanager-headless.linkis.svc.cluster.loca) with ipaddress such as 10.10.10.10
Bug: spark项目bin/beeline对于distribute by sort by语句支持不好，输出结果错误
Status: Reopened | Priority: Major | Resolution: 
Created: 11/Oct/22 02:27
Affects Version/s: 2.4.5

The results are inconsistent in part with the same sql statement that contains 'distribute by .... sort by ....' between spark-beeline($SPARK_HOME/bin/beeline) and hive-beeline, the result of hive-beeline is expected and the result of spark-beeline is error.sql statement:

select count(1) all_cnt,
sum(if(cmc_cellvoltage != new_cell_voltage, 1, 0)) ne_cnt
from (
select vin,
samplingtimesec,
cmc_cellvoltage,
concat('[', concat_ws(',', collect_list(cell_voltage)), ']') new_cell_voltage
from (
select vin, samplingtimesec, cmc_cellvoltage, cell_index, cell_voltage
from (
select vin, samplingtimesec, cmc_cellvoltage, cell_index, cell_voltage
from (
select vin,
samplingtimesec,
cmc_cellvoltage,–[1,2,3,4...,111,112]

row_number() over (partition by vin,samplingtimesec order by samplingtimesec) r
from table_name
WHERE dt = '20221007'
and samplingtimesec <= 1665079200000
) tmp
lateral view posexplode(split(replace(replace(cmc_cellvoltage, '[', ''), ']', ''), ',')) v0 as cell_index, cell_voltage
where r = 1
) tmp
distribute by vin
, samplingtimesec sort by cell_index
) tmp
group by vin, samplingtimesec, cmc_cellvoltage
) tmp;

hive-beeline result:  5682904 , 0

spark-beeline result: 5682904 , 5613492
Bug: Pyspark incompatible with Pypy
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/22 15:11
Affects Version/s: 3.3.0

Starting Spark with a recent Pypy (>3.6) fails because of an incompatibility between their pickle implementation and cloudpickle:
{quote}{{% PYSPARK_PYTHON=pypy3 ./bin/pyspark}}

{{...}}

{{ModuleNotFoundError: No module named '_pickle'}}
{quote}
 

It seems to be related to [this cloudpickle issue|https://github.com/cloudpipe/cloudpickle/issues/455], which has been fixed upstream. I was able to work around by replacing the Spark-provided cloudpickle (python/pyspark/cloudpickle) with the code from their git repo (and deleting pyspark.zip to purge that copy).
Bug: Add .asf.yaml to apache/spark-docker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/22 07:02
Affects Version/s: 3.3.0

We will initial the repo with .asf.yaml as first commit


{code:yaml}
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features
---
github:
  description: "Official Dockerfile for Apache Spark"
  homepage: https://spark.apache.org/
  labels:
    - python
    - scala
    - r
    - java
    - big-data
    - jdbc
    - sql
    - spark
  enabled_merge_buttons:
    merge: false
    squash: true
    rebase: true

notifications:
  pullrequests: reviews@spark.apache.org
  issues: reviews@spark.apache.org
  commits: commits@spark.apache.org
{code}
Bug: mypy complains accessing the variable defined in the class method 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/22 01:07
Affects Version/s: 3.4.0

This is one example: 

for SparkConnectTestsPlanOnly, those unit tests access the mock remote session by `self.connect`, however mypy complains in as `error: "SparkConnectTestsPlanOnly" has no attribute "connect"  [attr-defined]`
Bug: Shade more dependency to be able to run separately
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/22 12:25
Affects Version/s: 3.4.0

https://github.com/apache/spark/pull/38109 separated the component but found out that there were several more jars to be shaded. See also https://github.com/apache/spark/pull/38109#issuecomment-1269836435
Bug:  Scala 2.12 + Hadoop 2 + JDK 8 Daily Test failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/22 14:43
Affects Version/s: 3.4.0

[https://github.com/apache/spark/actions/runs/3164718086]

 
{code:java}
[error]     org.apache.spark.deploy.yarn.YarnShuffleAlternateNameConfigWithLevelDBBackendSuite
[error]     org.apache.spark.deploy.yarn.YarnShuffleIntegrationWithLevelDBBackendSuite
[error]     org.apache.spark.deploy.yarn.YarnClusterSuite
[error]     org.apache.spark.deploy.yarn.YarnShuffleAuthWithLevelDBBackendSuite
[error]     org.apache.spark.deploy.yarn.YarnShuffleAlternateNameConfigWithRocksDBBackendSuite
[error]     org.apache.spark.deploy.yarn.YarnShuffleIntegrationWithRocksDBBackendSuite
[error]     org.apache.spark.deploy.yarn.YarnShuffleAuthWithRocksDBBackendSuite {code}
Bug: Throw exception for Collect() and recommend to use toPandas()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/22 03:18
Affects Version/s: 3.4.0

Current connect `Collect()` return Pandas DataFrame, which does not match with PySpark DataFrame API: https://github.com/apache/spark/blob/ceb8527413288b4d5c54d3afd76d00c9e26817a1/python/pyspark/sql/connect/data_frame.py#L227.

The underlying implementation has been generating Pandas DataFrame though. In this case, we can choose to use to `toPandas()` and throw exception for `Collect()`.
Bug: V2 file scans have duplicative descriptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/22 21:10
Affects Version/s: 3.3.0

V2 file scans have duplication in the description. This is because FileScan uses the metadata to create the description, but each file type overrides metadata and the description adding the same metadata.

Example from a parquet agg pushdown explain:

{{ *+- BatchScan parquet file:/...[min(_3)#814, max(_3)#815, min(_1)#816, max(_1)#817, count(*)#818L, count(_1)#819L, count(_2)#820L, count(_3)#821L] ParquetScan DataFilters: [], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/..., PartitionFilters: [], PushedAggregation: [MIN(_3), MAX(_3), MIN(_1), MAX(_1), COUNT(*), COUNT(_1), COUNT(_2), COUNT(_3)], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct<min(_3):int,max(_3):int,min(_1):int,max(_1):int,count(*):bigint,count(_1):bigint,count(_2)..., PushedFilters: [], PushedAggregation: [MIN(_3), MAX(_3), MIN(_1), MAX(_1), COUNT(*), COUNT(_1), COUNT(_2), COUNT(_3)], PushedGroupBy: [] RuntimeFilters: []*}}
Bug: SPJ: Introduce a new DataSource V2 interface SupportsPushDownClusterKeys
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Oct/22 04:26
Affects Version/s: 3.4.0

A mix-in interface for ScanBuilder. Data sources can implement this interface to push down all the join or aggregate keys to data sources. A return value true indicates that data source will return input partitions  following the clustering keys. Otherwise, a false return value indicates the data source doesn't make such a guarantee, even though it may still report a partitioning that may or may not be compatible with the given clustering keys, and it's Spark's responsibility to group the input partitions whether it can be applied.
Bug: Support List[Column] for Join's on argument.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/22 03:42
Affects Version/s: 3.4.0

Right now Join's on does not support a list of ColumnRef: [df.age == df2.age, df.name == df2.name], we can improve the expression system to figure out a way to support it.
Bug: Fallback to original shuffle block when a push-merged shuffle chunk is zero-size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/22 12:08
Affects Version/s: 3.2.2

A large number of shuffle tests in our cluster show that bad nodes with chunk corruption appear have a probability of fetching zero-size shuffleChunks. In this case, we can fall back to original shuffle blocks.
Bug: Add support for compiled classes (Java classes)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/22 20:57
Affects Version/s: 3.3.0

For some users, it is more convenient to provide compiled classes rather than a descriptor file. 

We can support java compiled classes. Python could also use the same since all the processing happens in Scala.

Supporting python compiled classes is out-of-scope for this. It is not clear how well we can support that, short of using python UDF.
Bug: Fix the unsupported ops checker to allow chaining of stateful operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/22 19:59
Affects Version/s: 3.4.0

This is follow up ticket on https://issues.apache.org/jira/browse/SPARK-40925 - once we allow chaining of stateful operators in Spark SS, we need to fix the unsupported ops checker to allow these (currently they are blocked and require setting spark.sql.streaming.unsupportedOperationCheck to false
Bug: Alter partition should verify value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/22 09:59
Affects Version/s: 3.4.0

 
{code:java}
CREATE TABLE t (c int) USING PARQUET PARTITIONED BY(p int);

-- This DDL should fail but worked:
ALTER TABLE t ADD PARTITION(p='aaa'); {code}
Bug: Migrate type check failures of number formatting onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 14:56
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the number formatting or parsing expressions:
1. ToNumber (1):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala#L83
2. ToCharacter (1):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala#L227
3. ToNumberParser (1):
https://github.com/apache/spark/blob/5556cfc59aa97a3ad4ea0baacebe19859ec0bcb7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala#L262
Bug: Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/22 16:20
Affects Version/s: 3.2.2

Spark Config: spark.driver.log.dfsDir doesn't support absolute URI hadoop based path. It currently only supports URI path and writes only to fs.defaultFS and does not write logs to any other configured filesystem.
Bug: Upgrade jackson-module-scala to 2.14.0
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Oct/22 12:49
Affects Version/s: 3.3.0

This 2.14.0 release is still a few weeks a way. There is an rc2 release but there will probably be an rc3 before a full release.

The reason I marked the Jira component as 'Java API' is that this issue will affect Java users more than Scala users.

I raised this separately to SPARK-40666 because I have a different reason for this issue. SPARK-40666 can probably already be closed as the CVE is fixed in jackson-databind 2.13.4.2.

There are performance issues in jackson-module-scala 2.13.x that may affect some Spark users. Specifically, the per issue is [https://github.com/FasterXML/jackson-module-scala/issues/576]

Scala3 support added in jackson-module-scala 2.13.0 means that if you use Scala 2.13, you should be able to use Scala3 compiled classes with jackson-module-scala. Scala3 compiled classes are harder to recognise using runtime reflection (and Jackson is built around runtime reflection). Scala2 compiled classes have specific annotations. With Scala3 compiled classes, we need to look for .tasty files. This lookup can be slow if you have a lot of jars (or big jars). Issue [576|https://github.com/FasterXML/jackson-module-scala/issues/576] fixes an issue where this .tasty lookup is done every time you try to serialize/deserialize a Java class with an ObjectMapper that has the DefaultScalaModule registered. I will also disable the .tasty file lookups for Scala 2.11/2.12 as they are not useful for those users.

For Spark usage, it may be worth turning off this .tasty file support altogether. This is another enhancement in jackson-module-scala (but not in the RC2 release).

I will follow up and update this issue when the v2.14.0 release is ready. This change will require updating all Jackson jars to v2.14.0 (as Jackson does not support using version mismatches - except at patch version level).
Bug: isRemoteAddressMaxedOut performance overhead on scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/22 08:53
Affects Version/s: 3.2.2

On scala 2.13 the blocks in FetchRequest is sometimes backed by a `List` while in 2.12 it would be ArrayBuffer. This means that calculating the length of the blocks which is done in isRemoteAddressMaxedOut and other places now much more expensive.  This is because in 2.13 `Seq` is can no longer be backed by a mutable collection.