av9ash/GitBugs huggingface dataset:
Bug: Remove duplicated test_freqItems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/23 20:04
Affects Version/s: 3.4.0

def test_freqItems(self)
is duplicated

https://github.com/apache/spark/blob/e6c01cedbecebdc40863e552862beee1bd959710/python/pyspark/sql/tests/connect/test_connect_plan.py#L316
        
        https://github.com/apache/spark/blob/e6c01cedbecebdc40863e552862beee1bd959710/python/pyspark/sql/tests/connect/test_connect_plan.py#L334
Bug: Data duplicate or data lost with non-deterministic function
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/23 03:44
Affects Version/s: 3.0.0

When write table with shuffle data and non-deterministic function, data may be duplicate or lost due to retry task attempt.

For example:
{quote}
insert overwrite table target_table partition(ds)
select ... from a join b join c...
ditributed by ds, cast(rand()*10 as int)
{quote}

As rand() is non deterministic, the order of input to shuffle data may change in retry task. a row that is already present in another shuffe output might get distributed again to a new shuffle output (causing data duplication) or some row might not get any shuffle out as the designated shuffle output might have already finished (causing data loss).
Bug: DataFrame.unionByName output is wrong
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:46
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 584, in pyspark.sql.connect.dataframe.DataFrame.unionByName
Failed example:
    df1.unionByName(df2).show()
Expected:
    +----+----+----+
    |col0|col1|col2|
    +----+----+----+
    |   1|   2|   3|
    |   6|   4|   5|
    +----+----+----+
Got:
    +----+----+----+
    |col0|col1|col2|
    +----+----+----+
    |   1|   2|   3|
    |   4|   5|   6|
    +----+----+----+
    <BLANKLINE>{code}
Bug: DataFrameReader should support list of paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 02:14
Affects Version/s: 3.4.0

for example,

{code:java}
    def load(
        self,
        path: Optional[PathOrPaths] = None,
        format: Optional[str] = None,
        schema: Optional[Union[StructType, str]] = None,
        **options: "OptionalPrimitiveType",
    ) -> "DataFrame":
{code}

in PySpark support a list of pathes
Bug: Avoid losing exception info in Protobuf errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 18:03
Affects Version/s: 3.4.0

Protobuf connector related error handlers incorrectly report the exception. This is makes it hard for users to see actual issue. E.g. if there is a {{FileNotFoundException}} these error handlers use pass {{exception.getCause()}} rather than passing {{{}exception{}}}. As result, we lose the information that it was a {{FileNotFoundException}}
Bug: Unexpected schema set to DefaultSource plan (ReadwriterTests.test_save_and_load)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 02:46
Affects Version/s: 3.4.0

{code}
                                                                                
pyspark/sql/tests/test_readwriter.py:28 (ReadwriterParityTests.test_save_and_load)
self = <pyspark.sql.tests.connect.test_parity_readwriter.ReadwriterParityTests testMethod=test_save_and_load>

    def test_save_and_load(self):
        df = self.df
        tmpPath = tempfile.mkdtemp()
        shutil.rmtree(tmpPath)
        df.write.json(tmpPath)
        actual = self.spark.read.json(tmpPath)
        self.assertEqual(sorted(df.collect()), sorted(actual.collect()))
    
        schema = StructType([StructField("value", StringType(), True)])
        actual = self.spark.read.json(tmpPath, schema)
>       self.assertEqual(sorted(df.select("value").collect()), sorted(actual.collect()))

../test_readwriter.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/dataframe.py:1246: in collect
    query = self._plan.to_proto(self._session.client)
../../connect/plan.py:93: in to_proto
    plan.root.CopyFrom(self.plan(session))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyspark.sql.connect.plan.DataSource object at 0x7fe0d09c22b0>
session = <pyspark.sql.connect.client.SparkConnectClient object at 0x7fe0d069b5b0>

    def plan(self, session: "SparkConnectClient") -> proto.Relation:
        plan = proto.Relation()
        if self.format is not None:
            plan.read.data_source.format = self.format
        if self.schema is not None:
>           plan.read.data_source.schema = self.schema
E           TypeError: StructType([StructField('value', StringType(), True)]) has type StructType, but expected one of: bytes, unicode

../../connect/plan.py:246: TypeError
{code}
Bug: Support data type int8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:38
Affects Version/s: 3.4.0

{code:java}
import numpy as np
from pyspark.sql.functions import lit

dtype_to_spark_dtypes = [
    (np.int8, [("CAST(1 AS TINYINT)", "tinyint")]),
    (np.int16, [("CAST(1 AS SMALLINT)", "smallint")]),
    (np.int32, [("CAST(1 AS INT)", "int")]),
    (np.int64, [("CAST(1 AS BIGINT)", "bigint")]),
    (np.float32, [("CAST(1.0 AS FLOAT)", "float")]),
    (np.float64, [("CAST(1.0 AS DOUBLE)", "double")]),
    (np.bool_, [("true", "boolean")]),
]
for dtype, spark_dtypes in dtype_to_spark_dtypes:
    self.assertEqual(self.spark.range(1).select(lit(dtype(1))).dtypes, spark_dtypes){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 1064, in test_lit_np_scalar
    self.assertEqual(self.spark.range(1).select(lit(dtype(1))).dtypes, spark_dtypes)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 332, in wrapped
    return getattr(functions, f.__name__)(*args, **kwargs)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 198, in lit
    return Column(LiteralExpression._from_value(col))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/expressions.py", line 266, in _from_value
    return LiteralExpression(value=value, dataType=LiteralExpression._infer_type(value))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/expressions.py", line 262, in _infer_type
    raise ValueError(f"Unsupported Data Type {type(value).__name__}")
ValueError: Unsupported Data Type int8
{code}
Bug: Support aware datetimes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:19
Affects Version/s: 3.4.0

{code}

pyspark/sql/tests/test_serde.py:71 (SerdeParityTests.test_filter_with_datetime_timezone)
self = <pyspark.sql.tests.connect.test_parity_serde.SerdeParityTests testMethod=test_filter_with_datetime_timezone>

    def test_filter_with_datetime_timezone(self):
        dt1 = datetime.datetime(2015, 4, 17, 23, 1, 2, 3000, tzinfo=UTCOffsetTimezone(0))
        dt2 = datetime.datetime(2015, 4, 17, 23, 1, 2, 3000, tzinfo=UTCOffsetTimezone(1))
        row = Row(date=dt1)
>       df = self.spark.createDataFrame([row])

../test_serde.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:278: in createDataFrame
    _table = pa.Table.from_pylist([row.asDict(recursive=True) for row in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   NotImplementedError: a tzinfo subclass must implement tzname()

pyarrow/error.pxi:144: NotImplementedError

{code}
Bug: Introduce shuffle on SinglePartition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 11:56
Affects Version/s: 3.4.0

{code:scala}
spark.range(100000000L).selectExpr("id as a", "id as b").write.saveAsTable("t1")

sql(
  """
    |WITH base
    |     AS (select *, ROW_NUMBER() OVER(ORDER BY a) AS new_a FROM t1)
    |SELECT * FROM base t1 JOIN base t2 ON t1.a = t2.b
    |""".stripMargin).explain()
{code}

The output:
{noformat}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [a#10L], [b#26L], Inner
   :- Filter isnotnull(a#10L)
   :  +- Window [row_number() windowspecdefinition(a#10L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS new_a#8], [a#10L ASC NULLS FIRST]
   :     +- Sort [a#10L ASC NULLS FIRST], false, 0
   :        +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=50]
   :           +- FileScan parquet spark_catalog.default.t1[a#10L,b#11L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>
   +- Sort [b#26L ASC NULLS FIRST], false, 0
      +- Filter isnotnull(b#26L)
         +- Window [row_number() windowspecdefinition(a#25L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS new_a#27], [a#25L ASC NULLS FIRST]
            +- Sort [a#25L ASC NULLS FIRST], false, 0
               +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=54]
                  +- FileScan parquet spark_catalog.default.t1[a#25L,b#26L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>

{noformat}
Bug: DataFrame.createDataFrame datatype conversion error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 22:28
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1594, in pyspark.sql.connect.functions.to_json
Failed example:
    df = spark.createDataFrame(data, ("key", "value"))
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.to_json[3]>", line 1, in <module>
        df = spark.createDataFrame(data, ("key", "value"))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", line 252, in createDataFrame
        table = pa.Table.from_pandas(pdf)
      File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas
      File "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, in dataframe_to_arrays
        arrays = [convert_column(c, f)
      File "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, in <listcomp>
        arrays = [convert_column(c, f)
      File "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, in convert_column
        raise e
      File "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, in convert_column
        result = pa.array(col, type=type_, from_pandas=True, safe=safe)
      File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
      File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
      File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
    pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: tried to convert to int64", 'Conversion failed for column 1 with type object'){code}
Bug: DataFrame.show() fix map printing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 22:32
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1472, in pyspark.sql.connect.functions.posexplode_outer
Failed example:
    df.select("id", "a_map", posexplode_outer("an_array")).show()
Expected:
    +---+----------+----+----+
    | id|     a_map| pos| col|
    +---+----------+----+----+
    |  1|{x -> 1.0}|   0| foo|
    |  1|{x -> 1.0}|   1| bar|
    |  2|        {}|null|null|
    |  3|      null|null|null|
    +---+----------+----+----+
Got:
    +---+------+----+----+
    | id| a_map| pos| col|
    +---+------+----+----+
    |  1| {1.0}|   0| foo|
    |  1| {1.0}|   1| bar|
    |  2|{null}|null|null|
    |  3|  null|null|null|
    +---+------+----+----+
    <BLANKLINE>{code}
Bug: Implement `transform_keys` function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 22:21
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1611, in pyspark.sql.connect.functions.transform_keys
Failed example:
    df.select(transform_keys(
        "data", lambda k, _: upper(k)).alias("data_upper")
    ).show(truncate=False)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.transform_keys[1]>", line 1, in <module>
        df.select(transform_keys(
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 534, in show
        print(self._show_string(n, truncate, vertical))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 423, in _show_string
        ).toPandas()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 619, in _handle_error
        raise SparkConnectAnalysisException(
    pyspark.sql.connect.client.SparkConnectAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data type mismatch: Parameter 1 requires the "MAP" type, however "data" has the type "STRUCT<bar: DOUBLE, foo: DOUBLE>".
    Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496]
    +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493]
       +- LocalRelation [0#4488L, 1#4489] {code}
Bug: PYARROW_IGNORE_TIMEZONE warning can break application logging setup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 15:53
Affects Version/s: 3.2.3

in {code}python/pyspark/pandas/__init__.py{code}  there is currently a warning when {{PYARROW_IGNORE_TIMEZONE}} env var is not set (https://github.com/apache/spark/blob/187c4a9c66758e973633c5c309b551b1d9094e6e/python/pyspark/pandas/__init__.py#L44-L59):

{code:python}
    import logging

    logging.warning(
        "'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to "...
{code}

The {{logging.warning()}} call  will silently do a {{logging.basicConfig()}} call (at least in python 3.9, which I tried).
(FYI: Something like {{logging.getLogger(...).warning()}} would not do this silent call)


This has the following very hard to figure out side-effect:
importing `pyspark.pandas` (directly or indirectly somewhere)  might break your logging setup (if PYARROW_IGNORE_TIMEZONE is not set).

Very basic  example (assuming PYARROW_IGNORE_TIMEZONE is not set):

{code:python}
import logging
import pyspark.pandas

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger("test")
logger.warning("I warn you")
logger.debug("I debug you")
{code}

Will only produce the warning, not the debug line.
By removing the {{import pyspark.pandas}}, the debug line is produced
Bug: Optimize logs for removeBlocks and removeShuffleMerge PRC
Status: Open | Priority: Major | Resolution: 
Created: 12/Jan/23 08:13
Affects Version/s: 3.4.0

1. Change the error logs level for RemoveBlocks RPC.
2. Add some error logs for RemoveShuffleMerge RPC.

Discuss link: https://github.com/apache/spark/pull/37922#discussion_r1067551485
Bug: Support DataFrame hint parameter to be list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 02:50
Affects Version/s: 3.4.0

{code:java}
df = self.spark.range(10e10).toDF("id")
such_a_nice_list = ["itworks1", "itworks2", "itworks3"]
hinted_df = df.hint("my awesome hint", 1.2345, "what", such_a_nice_list){code}
Bug: Migrate the official Spark History Server helm chart to a non-archived repository
Status: Open | Priority: Major | Resolution: 
Created: 11/Jan/23 19:46
Affects Version/s: 3.4.0

* The Spark History server Helm chart is currently published in the [`helm/charts` repository|https://github.com/helm/charts/tree/master/stable/spark-history-server].  This repository has been [deprecated|https://github.com/helm/charts#%EF%B8%8F-deprecation-and-archive-notice] in favor of having chart maintainers own their own repository.
 * We need to create a new GitHub repository to host this chart so that it can be updated/modified/maintained.
Bug: DiskBlockManager can't clear all of the tmp dirs in k8s
Status: Open | Priority: Major | Resolution: 
Created: 09/Jan/23 06:17
Affects Version/s: 3.2.0

I have a test for spark on k8s  that my job had completed, but some tmp dirs are still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.

The DiskBlockManager may be clear the dirs  just one iteration ， then the process be killed.

The shuffle data will be getting larger and   fill the disk space.

!image-2023-01-09-14-03-01-840.png!
Bug: Flaky Test: StreamingQueryStatusListenerSuite.test small retained queries
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Jan/23 20:51
Affects Version/s: 3.4.0

I saw this failures on master branch frequently.

https://github.com/apache/spark/runs/10560025461
https://github.com/apache/spark/runs/10556299549
https://github.com/apache/spark/runs/10551101022
Bug: Collapse percentile functions if possible
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Jan/23 08:50
Affects Version/s: 3.4.0

*percentile* function will put all the target elements into a Hashset and then compute the result. We can try to combine percentile functions into one and compute the result with only one Hashset.
For example:
{code:sql}
SELECT max(value1) as max_value1, percentile(value2, 0.3) as p1,
        percentile(value3, 0.4) + percentile(value3, 0.5) as p2,
        percentile(value2, 0.6) as p3
FROM t1
{code}
can be optimized to 
{code:sql}
SELECT max_value1, _combined_percentile_0[0] as p1, p2, _combined_percentile_0[1] as p3
FROM (
   SELECT  max(value1) as max_value1,
           percentile(value3, 0.4) + percentile(value3, 0.5) as p2,
           percentile(value2, array(0.3, 0.6)) as _combined_percentile_0
   FROM t1) as t1
{code}
Bug: `DataFrame.collect` should support nested types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 07:09
Affects Version/s: 3.4.0


{code:java}
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1578, in pyspark.sql.connect.functions.struct
Failed example:
    df.select(struct('age', 'name').alias("struct")).collect()
Expected:
    [Row(struct=Row(age=2, name='Alice')), Row(struct=Row(age=5, name='Bob'))]
Got:
    [Row(struct={'age': 2, 'name': 'Alice'}), Row(struct={'age': 5, 'name': 'Bob'})]

{code}
Bug: `DataFrame.intersect` doctest output has different order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 01:23
Affects Version/s: 3.4.0

not sure whether this needs to be fix:


{code:java}
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 609, in pyspark.sql.connect.dataframe.DataFrame.intersect
Failed example:
    df1.intersect(df2).show()
Expected:
    +---+---+
    | C1| C2|
    +---+---+
    |  b|  3|
    |  a|  1|
    +---+---+
Got:
    +---+---+
    | C1| C2|
    +---+---+
    |  a|  1|
    |  b|  3|
    +---+---+
    <BLANKLINE>
**********************************************************************
   1 of   3 in pyspark.sql.connect.dataframe.DataFrame.intersect

{code}
Bug: SparkSession.createDataFrame error parity
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 05:22
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame(
    [
        (1, 10, 1.0, "one"),
        (2, 20, 2.0, "two"),
        (3, 30, 3.0, "three"),
    ],
    ["id", "int", "double", "str"],
)

with self.subTest(desc="with none identifier"):
    with self.assertRaisesRegex(AssertionError, "ids must not be None"):
        df.unpivot(None, ["int", "double"], "var", "val"){code}
Error:
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 575, in test_unpivot
    with self.assertRaisesRegex(AssertionError, "ids must not be None"):
AssertionError: AssertionError not raised{code}
Bug: SBT unable to resolve particular packages from the imported maven build
Status: Open | Priority: Major | Resolution: 
Created: 10/Jan/23 13:41
Affects Version/s: 3.4.0

An SBT issue causes the resolution from the imported maven build for particular packages to not work for an unknown reason. This affects Spark-Connect-related projects (see [here|https://github.com/apache/spark/blob/6cae6aa5156655c79eb3f20292ccec6c479c3b1b/project/SparkBuild.scala#L667-L668] and [here|https://github.com/apache/spark/blob/6cae6aa5156655c79eb3f20292ccec6c479c3b1b/project/SparkBuild.scala#L902-L904] for example) by forcing duplicate deps.

The pom build works fine when removing the affected dep (like guava for example) but the sbt build then fails. Thus, we are forced to explicitly mention the versions of the affected packages so that SBT can then parse the version(s) to manually include them (and they're also added as a dep in maven to ensure version consistency with sbt)
Bug: Make stage scheduling support local-cluster mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/23 09:08
Affects Version/s: 3.4.0

Make stage scheduling support local-cluster mode.


This is useful in testing, especially for test code of third-party python libraries that depends on pyspark, many tests are written with pytest, but pytest is hard to integrate with a standalone spark cluster.
Bug: Function `sampleby` return parity
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:57
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame([Row(a=i, b=(i % 3)) for i in range(100)])
sampled = df.stat.sampleBy("b", fractions={0: 0.5, 1: 0.5}, seed=0)
self.assertTrue(sampled.count() == 35){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 202, in test_sampleby
    self.assertTrue(sampled.count() == 35)
AssertionError: False is not true {code}