av9ash/GitBugs huggingface dataset:
Bug: Reduce the number of doMergeApplicationListing invocations
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Dec/22 08:43
Affects Version/s: 3.4.0

When restarting the history server, In history server log,
we can see many {{INFO FsHIstoryProvider: Finished parsing application_xxx}} followed by
{{INFO FsHIstoryProvider: Deleting expired event log for application_xxx}}.
This is caused by the expired and unlisted log files also be parsed, and then execute {{checkAndCleanLog}} to delete parsed info, this means that the parsing is unnecessary.

If there are a large number of expired log files in the log directory, it will affect the speed of replay.

In order to avoid this, we can add a logic to clean up these expired log files before calling {{doMergeApplicationListing}}.
Bug: Multi sparkSession should share single SQLAppStatusStore
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Dec/22 00:49
Affects Version/s: 3.1.1

In spark , if we create multi sparkSession in the program, we will get multi-SQLTab in UI, 

At the same time, we will get muti-SQLAppStatusListener object, it is waste of memory.

code like this:

 
{code:java}
// code placeholder
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf()
.setAppName("demo")
.setMaster("local[*]")

val spark = SparkSession.builder()
.config(sparkConf)
.getOrCreate()

setDefaultSession(null)
setActiveSession(null)

val spark2 = SparkSession.builder()
.config(sparkConf)
.getOrCreate()

import spark.implicits._
val testData = spark.sparkContext
.parallelize((1 to 3).map(i => TestData(i, i.toString))).toDF()
testData.createOrReplaceTempView("testTable")
val testData2 = spark.sparkContext.parallelize(
TestData2(1, "1") ::
TestData2(1, "2") ::
TestData2(2, "1") ::
TestData2(2, "2") ::
TestData2(3, "1") ::
TestData2(3, "2") ::
Nil, 2).toDF()

testData2.createOrReplaceTempView("testTable2")
val query = "select ind2,count(*) from ( select * from testTable2 join testTable on testTable.ind = testTable2.ind2 where testTable.name <> '1') group by ind2"
spark.sql(query).collect()

Thread.sleep(500000)
spark.stop()
}
case class TestData(ind: Int, name: String)
case class TestData2(ind2: Int, name: String) {code}
Bug: Add assertion on end offset range for Kafka data source with Trigger.AvailableNow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/22 07:20
Affects Version/s: 3.4.0

Although there are lots of benefits Trigger.AvailableNow provides, we figure out one caveat of Trigger.AvailableNow, very sensitive on the offset range.

Trigger.AvailableNow stops the query when the start offset and end offset are being same, producing no data from data source. Given the semantic of Trigger.AvailableNow, the implementation of data source is expected to retrieve the final offset at the start of the query, and gradually increase the offset range to eventually reach the final offset.

Any bug breaking this leads to infinity run of the query, hence all data source implementations supporting Trigger.AvailableNow are encouraged to have some assertion to prevent such case in prior.

For built-in data sources, only Kafka data source is something supporting Trigger.AvailableNow but don't have some assertion on the offset range. We'd like to add some assertion against Kafka data source, for Trigger.AvailableNow.
Bug: Expose SQL confs and extraOptions separately in o.a.s.sql.execution.streaming.state.RocksDBConf
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Dec/22 06:03
Affects Version/s: 3.4.0

Currently the usage of *StateStoreConf* is via [*{{confs}}*|https://github.com/huanliwang-db/spark/blob/7671bc975f2d88ab929e4982abfe3e166fa72e35/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala#L77-L78], which composes both SQL confs and extraOptions into one. The name of config for extraOptions shouldn't have to follow the name prefix of SQL conf, because it's not bound to the context of SQL conf.

After differentiate SQL conf and extraOptions in {*}StateStoreConf{*}, we should be able to adopt more use case on operator level configs by using the {*}extraOptions{*}.
Bug: Pin versions-maven-plugin version
Status: Open | Priority: Minor | Resolution: 
Created: 14/Dec/22 21:58
Affects Version/s: 3.4.0

`versions-maven-plugin` release `2.14.0` broke the following this command in Spark:
{noformat}
build/mvn versions:set -DnewVersion=3.4.0-jz-0 -DgenerateBackupPoms=false
{noformat}
See [https://github.com/mojohaus/versions/issues/848.]

The plugin will be fixed in 2.14.1. However it may be desirable to pin plugin version.
Bug: Support easy way for user defined PYTHONPATH in workers
Status: Open | Priority: Minor | Resolution: 
Created: 13/Dec/22 12:11
Affects Version/s: 3.3.1

When working interactively with Spark through notebooks in various envs - Databricks/YARN I often encounter a very frustrating process of trying to add new python modules and even change their code without starting a new spark session/cluster.

In the driver side it is easy to add things like `sys.path.append()` but if for example, if a UDF code is importing a function from a local module, then the pickle boundaries will assume that the module exists in the workers, and fail on "python module does not exist..".

To update the code "online" I can add NFS volume to the workers' PYTHONPATH.

However, setting the PYTHONPATH in the workers is not easy as it gets overridden by someone (databricks/spark) along the way. a few ugly workarounds are suggested like running a "dummy" UDF on the workers to add the folder to the sys.path.

I think all of that could easily be solved if we just add a dedicated `spark.conf` the will get merged into the worker's PYTHONPATH, just here:

[https://github.com/apache/spark/blob/0e2d604fd33c8236cfa8ae243eeaec42d3176a06/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala#L94]

 

please tell me what you think, and I will make the PR.

thanks.
Bug: Update R version to 4.1.2 in Dockerfile comment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Dec/22 03:28
Affects Version/s: 3.4.0

The comment is outdated. We are using R 4.1.2 for Apache Spark 3.4.0.
{code}
$ docker run -it --rm apache/spark-r:3.3.1 R --version | grep 'R version'
R version 4.0.4 (2021-02-15) -- "Lost Library Book"

$ docker run -it --rm kubespark/spark-r:dev R --version | grep 'R version'
R version 4.1.2 (2021-11-01) -- "Bird Hippie"
{code}
Bug: protoc-3.21.9-linux-x86_64.exe requires GLIBC_2.14
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/22 07:13
Affects Version/s: 3.4.0

Compile Spark Core module on CentOS release 6.3, the default glibc version is 2.12, this will cause compilation to fail as follows:
{code:java}
[ERROR] /home/disk1/spark-ut/spark/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto [0:0]: /tmp/protoc5792311590243222147.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/protoc5792311590243222147.exe)
[ERROR] /home/disk1/spark-ut/spark/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto [0:0]: /tmp/protoc5792311590243222147.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /tmp/protoc5792311590243222147.exe)
[ERROR] /home/disk1/spark-ut/spark/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto [0:0]: /tmp/protoc5792311590243222147.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /tmp/protoc5792311590243222147.exe)
[ERROR] /home/disk1/spark-ut/spark/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto [0:0]: /tmp/protoc5792311590243222147.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /tmp/protoc5792311590243222147.exe){code}
Bug: There are some small files when using rebalance(column)
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Dec/22 07:15
Affects Version/s: 3.3.1

*Problem ( REBALANCE(column)* {*}){*}:

 SparkSession config:
{noformat}
config("spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled", "true") 
config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "20m") 
config("spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor", "0.5"){noformat}
so, we except that files size should be bigger than 20m*0.5=10m at least. 

but in fact , we got some small files like the following:
{noformat}
-rw-r--r--   1 jp28948 staff     12.1 M 2022-12-07 13:13 .../part-00000-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet
-rw-r--r--   1 jp28948 staff     12.1 M 2022-12-07 13:13 .../part-00001-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet
-rw-r--r--   1 jp28948 staff     12.1 M 2022-12-07 13:13 .../part-00002-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet
-rw-r--r--   1 jp28948 staff     12.1 M 2022-12-07 13:13 .../part-00003-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet
-rw-r--r--   1 jp28948 staff      9.1 M 2022-12-07 13:13 .../part-00004-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet
-rw-r--r--   1 jp28948 staff      3.0 M 2022-12-07 13:13 .../part-00005-1ece1aae-f4f6-47ac-abe2-170ccb61f60e.c000.snappy.parquet{noformat}
9.1 M and 3.0 M is smaller than 10M. we have to handle these small files in another way.
Bug: Update the script used to generate register function in UDFRegistration 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/22 11:11
Affects Version/s: 3.4.0

SPARK-35065 use {{QueryCompilationErrors.invalidFunctionArgumentsError}} instead of {{throw new AnalysisException(...)}} for {{register}} function in {{{}UDFRegistration{}}}, but the script used to generate xx has not been updated, so this pr update the script.
Bug: Workaround hive table name validation issue
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Dec/22 01:36
Affects Version/s: 3.4.0

For example:
 * We want to create a table called {{tAb_I}}
 * Hive metastore will check if the table name is valid by {{MetaStoreUtils.validateName(tbl.getTableName())}}
 * Hive will call {{HiveStringUtils.normalizeIdentifier(tbl.getTableName())}} and then save the save the table name to lower case, *but after setting the local to "tr", it will be {{tab_ı}} which is not a valid table name*
 * When we run alter table command, we will first get the hive table from hive metastore which is not a valid table name.
 * Update some properties or other, and then try to save it to hive metastore.
 * Hive metastore will check if the table name is valid and then throw exception {{org.apache.hadoop.hive.ql.metadata.HiveException: [tab_ı]: is not a valid table name}}
Bug: Add data frame equality check for testing purposes.
Status: Open | Priority: Minor | Resolution: 
Created: 02/Dec/22 15:07
Affects Version/s: 3.3.0

We woud like to have the functionality as suggested in https://issues.apache.org/jira/browse/SPARK-28172 . It got closed by an unrelated story. The comment on the story 

> Wouldn't this require to execute both DataFrames and collect the data into driver side? When the datasets are large, it's very easy for users to shoot them in the foot. I won't do that without an explicit plan and design doc for all other operators.

does not make sense since this will be used for unit testing purposes and the dataframes compared will be small.

We are currently using Pandas `assert_frame_equal` https://github.com/pandas-dev/pandas/blob/v1.5.2/pandas/_testing/asserters.py#L1135-L1358 functionality which works very well. However, we are having issues with pandas since it only supports only a subset of the timestamps supported by Spark [https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-timestamp-limits.] This is the first time this became a blocker for us so therefore we would like to have the functionality to validate equality and get feedback on what is different similar to that in Spark. As far as I could see there is nothing in PySpark which currently supports does this.

Please let me know any feedback you have, thanks!
Bug: Invalid call toAttribute on unresolved object exception caused by WidenSetOperationTypes
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Dec/22 08:06
Affects Version/s: 3.2.0

The problem can be reproduced in the following way:
{code:java}
spark-sql> CREATE OR REPLACE TEMPORARY VIEW t1 AS VALUES (1, 'a'), (2, 'b') tbl(c1, c2);

spark-sql> CREATE OR REPLACE TEMPORARY VIEW t2 AS VALUES (1.0, 1), (2.0, 4) tbl(c1, c2); 

spark-sql> SELECT
         > TRANSFORM(*) USING 'cat' AS (a)
         > FROM
         > (
         > SELECT c2 AS c from t2
         > UNION
         > SELECT c2 AS c from t1);
Invalid call to toAttribute on unresolved object{code}
Bug: Remove def test_vectorized_udf_unsupported_types
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Dec/22 18:01
Affects Version/s: 3.4.0

https://github.com/apache/spark/blob/18488158beee5435f99899f99b2e90fb6e37f3d5/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py#L603

{code:java}
def test_vectorized_udf_wrong_return_type(self):
        with QuietTest(self.sc):
            for udf_type in [PandasUDFType.SCALAR, PandasUDFType.SCALAR_ITER]:
                with self.assertRaisesRegex(
                    NotImplementedError,
                    "Invalid return type.*scalar Pandas UDF.*ArrayType.*TimestampType",
                ):
                    pandas_udf(lambda x: x, ArrayType(TimestampType()), udf_type)
{code}

is the same code as 

https://github.com/apache/spark/blob/18488158beee5435f99899f99b2e90fb6e37f3d5/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py#L679


{code:java}
def test_vectorized_udf_unsupported_types(self):
        with QuietTest(self.sc):
            for udf_type in [PandasUDFType.SCALAR, PandasUDFType.SCALAR_ITER]:
                with self.assertRaisesRegex(
                    NotImplementedError,
                    "Invalid return type.*scalar Pandas UDF.*ArrayType.*TimestampType",
                ):
                    pandas_udf(lambda x: x, ArrayType(TimestampType()), udf_type)
{code}


So we can remove one or fix the typo. 


I found this using [sonor|https://sonarcloud.io/project/issues?languages=py&resolved=false&rules=python%3AS4144&id=spark-python&open=AYQdnW-FRrJbVxW9ZDO0]
Bug: input_file_positon
Status: Open | Priority: Trivial | Resolution: 
Created: 17/Dec/22 05:15
Affects Version/s: 3.3.1

As for now, we have 3 built-in UDFs related to input files and blocks.  So can we provide a new UDF to return current record position of a file or block? Sometimes, it's useful and we can consider this position (called ROWID in oracle) as a physical primary key.

 
|input_file_block_length()|Returns the length of the block being read, or -1 if not available.|
|input_file_block_start()|Returns the start offset of the block being read, or -1 if not available.|
|input_file_name()|Returns the name of the file being read, or empty string if not available.|