av9ash/GitBugs huggingface dataset:
Bug: Inline eval path cannot handle null structs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/22 23:29
Affects Version/s: 3.1.3

This issue is somewhat similar to SPARK-39061, but for the eval path rather than the codegen path.

Example:
{noformat}
set spark.sql.codegen.wholeStage=false;
select inline(array(named_struct('a', 1, 'b', 2), null));
{noformat}
This results in a NullPointerException:
{noformat}
22/06/16 15:10:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
{noformat}
The next example doesn't require setting {{spark.sql.codegen.wholeStage}} to {{{}false{}}}:
{noformat}
val dfWide = (Seq((1))
  .toDF("col0")
  .selectExpr(Seq.tabulate(99)(x => s"$x as col${x + 1}"): _*))

val df = (dfWide
  .selectExpr("*", "array(named_struct('a', 1, 'b', 2), null) as struct_array"))

df.selectExpr("*", "inline(struct_array)").collect
{noformat}
The result is similar:
{noformat}
22/06/16 15:18:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:80)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_8$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
{noformat}
Bug: Fix broken function links in the auto-generated pandas API support list documentation.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/22 15:13
Affects Version/s: 3.4.0

In the auto-generated documentation on pandas API support list, there are cases where the link of the function property provided in the document is not connected, so it needs to be corrected.

The current 'supported API generation' function dynamically compares the modules of `{{{}PySpark.pandas`{}}} and `{{{}pandas`{}}} to find the difference. At this time, the inherited class is also aggregated, and the link is not generated correctly (such as {{`CategoricalIndex.all()`}} is used internally by inheriting `{{{}Index.all()`{}}}.) because it does not match the pattern of each API document.
Bug: Spark Sql create table using jbdc add preSql option
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/22 02:47
Affects Version/s: 3.2.1

In my recent process of using Spark Sql, I tried to use create using jdbc to create a spark table that I need to use, but I may need to consider that before inserting, I need to delete some of the previous data, so I want to be able to expose This option, the use example can refer to the following picture.

And I can submit a pullrequest to solve this problem. please assign to me. Thanks.

!image-2022-06-09-10-48-05-347.png!
Bug: Sphinx build fails with "node class 'meta' is already registered, its visitors will be overridden"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/22 00:53
Affects Version/s: 3.2.1

{code}
Moving to python/docs directory and building sphinx.
Running Sphinx v3.0.4
WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
/__w/spark/spark/python/pyspark/pandas/supported_api_gen.py:101: UserWarning: Warning: Latest version of pandas(>=1.4.0) is required to generate the documentation; however, your version was 1.3.5
  warnings.warn(
Warning, treated as error:
node class 'meta' is already registered, its visitors will be overridden
make: *** [Makefile:35: html] Error 2
                    ------------------------------------------------
      Jekyll 4.2.1   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
{code}

Sphinx build fails apparently with the latest docutils (see also https://issues.apache.org/jira/browse/FLINK-24662). we should pin the version.
Bug:  Exclude rules in analyzer
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 08/Jun/22 04:13
Affects Version/s: 3.2.1

I have found that we can use `spark.sql.optimizer.excludedRules` to exclude rules in the optimizer. I'd like to have a similar capability in the analyzer.

I don't know if it is possible or if it breaks the design of catalyst?
Bug: Flaky Test: BloomFilterAggregateQuerySuite
Status: Open | Priority: Major | Resolution: 
Created: 06/Jun/22 05:24
Affects Version/s: 3.3.0

During Apache Spark 3.3.0 RC5 tests, I found that this test case is very flaky in my environment.
{code:java}
     [info] - Test bloom_filter_agg and might_contain *** FAILED *** (20 seconds, 370 milliseconds)
     [info]   Results do not match for query:
     [info]   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
     [info]   Timezone Env: 
...

  == Results ==
     [info]   !== Correct Answer - 1 ==   == Spark Answer - 1 ==
     [info]   !struct<>                   struct<positive_membership_test:boolean,negative_membership_test:boolean>
     [info]   ![true,false]               [true,true] (QueryTest.scala:244)
     [info]   org.scalatest.exceptions.TestFailedException:
     [info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
     [info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
     [info]   at org.apache.spark.sql.QueryTest$.newAssertionFailedException(QueryTest.scala:234)
     [info]   at org.scalatest.Assertions.fail(Assertions.scala:933)
     [info]   at org.scalatest.Assertions.fail$(Assertions.scala:929)
     [info]   at org.apache.spark.sql.QueryTest$.fail(QueryTest.scala:234)
     [info]   at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:244)
     [info]   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:151)
     [info]   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:155)
     [info]   at org.apache.spark.sql.BloomFilterAggregateQuerySuite.$anonfun$new$4(BloomFilterAggregateQuerySuite.scala:98)
{code}
Bug: Improve PySpark structured streaming page more readable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/22 00:43
Affects Version/s: 3.4.0

Similar to SPARK-39253, the PySpark documentation for "Structured Streaming" part is not-well organized so it's a bit uncomfortable to be read.

So we might need to refine the document to make it more readable so that users can easily find the methods they want.
Bug: Support R 4.2.0 in SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 08:34
Affects Version/s: 3.4.0

This JIRA aims to support R 4.2.0 with upgrading R version from 4.0.2 to 4.2.0 in CI.

One test fails with R 4.2.0:

{code}
-- 1. Error (test_mllib_classification.R:245:3): spark.logit -------------------
Error in `if (class(upperBoundsOnCoefficients) != "matrix") {
    stop("upperBoundsOnCoefficients must be a matrix.")
}`: the condition has length > 1
{code}
Bug: Refactor and merge all related job selection logic into precondition 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/22 07:33
Affects Version/s: 3.4.0

Currently there are three logics that choose which build to run.

First is configure-jobs
Second is precondition
Third is the type of job (if it's scheduled or not).

We should merge all to precondition.
Bug: ExpressionSetSuite test failure with Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/22 01:25
Affects Version/s: 3.4.0

https://github.com/apache/spark/runs/6912247219?check_suite_focus=true#step:9:21351

{code}
[info] - remove multiple elements to set with non-deterministic expressions *** FAILED *** (32 milliseconds)
[info]   Set() had size 0 instead of expected size 1 (ExpressionSetSuite.scala:239)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.catalyst.expressions.ExpressionSetSuite.$anonfun$new$6(ExpressionSetSuite.scala:239)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:203)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
{code}
Bug: Recover PySpark jobs for branch-3.2 w/ Scala 2.13
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 00:37
Affects Version/s: 3.2.1

We now enabled it (https://github.com/apache/spark/runs/7016817224?check_suite_focus=true). We should recover it or skip it for a valid reason.
Bug: NumPy Input Support in PySpark
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 07/Jun/22 18:22
Affects Version/s: 3.4.0

NumPy is the fundamental package for scientific computing with Python. It is very commonly used, especially in the data science world. For example, Pandas is backed by NumPy, and Tensors also supports interchangeable conversion from/to NumPy arrays.

However, PySpark only supports Python native types with the exception of pandas data in “SparkSession.createDataFrame(pandas.DataFrame)”.  “DataFrame.toPandas()”. NumPy input support  has been requested multiple times, see SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857. 

The ticket proposes to support NumPy input in PySpark.
 
See more design [design|https://docs.google.com/document/d/e/2PACX-1vTD70k1_Ohd2QCY-dbVrqDTmyfmlyDgxA-0SVl0DcDyeePzc9JiaIjG9B_FFI2_MfGGyjPrWYCTXk5F/pub].
Bug: Expose the information of catalog table to the logical plan in streaming query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/22 02:28
Affects Version/s: 3.4.0

In Spark 3.1.0, we added ability to read and write from/to catalog table for streaming query. (Read: [SPARK-32885|https://issues.apache.org/jira/browse/SPARK-32885] / Write: [SPARK-32896|https://issues.apache.org/jira/browse/SPARK-32896])

When the streaming query is being executed with such API, the table identifier for source and sink is "conditionally" exposed to the logical plan, depending on the DSv1 vs DSv2 and source vs sink.

Since this is the only way to get the information of table, we would like to make some changes on relevant logical nodes to carry out and expose the information of table, specifically, table identifier.
Bug: Unable to query _metadata in streaming if getBatch returns multiple logical nodes in the DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/22 18:11
Affects Version/s: 3.2.1

Here: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L585]

 

We should probably `transform` instead of `match`
Bug: Python API Parity in Structure Streaming
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Jun/22 20:59
Affects Version/s: 3.3.0

New APIs in Structured Streaming tend to get added to Java/Scala first.  This creates a situation where the Python API have fallen behind.  For example map/flatMapGroupsWithState is not supported in the Pyspark.  We need Pyspark API to catch up with the Java/Scala APIs and, where necessary, provide tighter integrations with native python data processing frameworks such as Pandas.
Bug: Infer date type for CSV schema inference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/22 18:21
Affects Version/s: 3.2.1

1. If a column contains only dates, it should be of “date” type in the inferred schema
 * If the date format and the timestamp format are identical (e.g. both are yyyy/mm/dd), entries will default to being interpreted as Date

2. If a column contains dates and timestamps, it should be of “timestamp” type in the inferred schema

 

A similar issue was opened in the past but was reverted due to the lack of strict pattern matching.
Bug: Support `createDataFrame` from a list of scalars when schema is not provided
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Jun/22 21:46
Affects Version/s: 3.4.0

Currently, DataFrame creation from a list of native Python scalars is unsupported in PySpark, for example,

{{>>> spark.createDataFrame([1, 2]).collect()}}
{{Traceback (most recent call last):}}
{{...}}
{{TypeError: Can not infer schema for type: <class 'int'>}}

{{However, Spark DataFrame Scala API supports that:}}

{{scala> Seq(1, 2).toDF().collect()}}
{{res6: Array[org.apache.spark.sql.Row] = Array([1], [2])}}

To maintain API consistency, we propose to support DataFrame creation from a list of scalars. 

See more [here]([https://docs.google.com/document/d/1Rd20PVbVxNrLfOmDtetVRxkgJQhgAAtJp6XAAZfGQgc/edit?usp=sharing]).
Bug: Do not output duplicated columns in star expansion of subquery alias of NATURAL/USING JOIN
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 18:16
Affects Version/s: 3.2.0

A bug was introduced in https://issues.apache.org/jira/browse/SPARK-34527 such that the duplicated columns within a NATURAL/USING JOIN were output from the qualified star of a subquery alias. For example:

{code:java}
val df1 = Seq((3, 8)).toDF("a", "b") 
val df2 = Seq((8, 7)).toDF("b", "d") 
val joinDF = df1.join(df2, "b")
joinDF.alias("r").select("r.*")
{code}

Outputs two duplicate `b` columns, instead of just one.
Bug: Add session catalog name for v1 database table and function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/22 07:52
Affects Version/s: 3.4.0

To make it more clearer that this table or function comes from which catalog. It affects:
 * the scan table(view) of the query explain
 * the target table(view) of the data writing
 * desc database
 * desc formatted table(view)
 * show create table(view)
 * desc function
Bug: Support IPv6-only environment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/22 18:24
Affects Version/s: 3.4.0

Spark doesn't fully work in pure IPV6 environment that doesn't have IPV4 at all. This is an umbrella jira tracking the support of pure IPV6 deployment.

 

The scope of this JIRA doesn't include ancient resource managers like Mesos(deprecated in Spark) or YARN(because Apache Spark still supports Hadoop 2). We are going to focus on only `Spark Standalone` and `K8s`

 

K8s IPv4/IPv6 dual-stack Feature reached `Stable` stage at v1.23.
 - [https://kubernetes.io/docs/concepts/services-networking/dual-stack/]
 -- v1.16 [alpha]
 -- v1.21 [beta]
 -- v1.23 [stable]

 

Currently, EKS supports IPv6 with v1.21.13-eks-84b4fe6.

- https://aws.amazon.com/blogs/containers/amazon-eks-launches-ipv6-support/
Bug: Driver cores must be a positive number
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jun/22 11:38
Affects Version/s: 3.2.1

When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Driver cores must be a positive number
	at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
	at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
	at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
	at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
Obviously I set the driver-cores property but it doesn't matter, it is not used. I'm suspecting that it is something about the Mesos Dispatcher, it receives the property because it is being displayed correctly in the Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and therefore fails.

 

Update:

It happens when submitting a *Spark 3.2.1* job with *Mesos* in {*}Cluster mode{*}.
Bug: Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been restarted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/22 21:06
Affects Version/s: 3.2.0

We saw these exceptions during block push:
{code:java}
22/06/24 13:29:14 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_170_568_174, and will not retry (0 retries)
org.apache.spark.network.shuffle.BlockPushException: !application_1653753500486_3193550shuffle_170_568_174java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration
	at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:92)
	at org.apache.spark.network.shuffle.RemoteBlockPushResolver.getActiveLocalDirs(RemoteBlockPushResolver.java:300)
	at org.apache.spark.network.shuffle.RemoteBlockPushResolver.getFile(RemoteBlockPushResolver.java:290)
	at org.apache.spark.network.shuffle.RemoteBlockPushResolver.getMergedShuffleFile(RemoteBlockPushResolver.java:312)
	at org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$getOrCreateAppShufflePartitionInfo$1(RemoteBlockPushResolver.java:168)

22/06/24 13:29:14 WARN UnsafeShuffleWriter: Pushing block shuffle_170_568_174 to BlockManagerId(, node-x, 7337, None) failed.
{code}
Note: The NodeManager on node-x (node against which this exception was seen) was not restarted.

The reason this happened is because the executor registers the block manager with {{BlockManagerMaster}} before it registers with the ESS. In push-based shuffle, a block manager is selected by the driver as a merger for the shuffle push. However, the ESS on that node can successfully merge the block only if it has received the metadata about merged directories from the local executor (sent when the local executor registers with the ESS). If this local executor registration is delayed, but the ESS host got picked up as a merger then it will fail to merge the blocks pushed to it which is what happened here.

The local executor on node-x is executor 754 and the block manager registration happened at 13:28:11
{code:java}
22/06/24 13:28:11 INFO ExecutorAllocationManager: New executor 754 has registered (new total is 1200)

22/06/24 13:28:11 INFO BlockManagerMasterEndpoint: Registering block manager node-x:16747 with 2004.6 MB RAM, BlockManagerId(754, node-x, 16747, None)
{code}
The application got registered with shuffle server at node-x at 13:29:40
{code:java}
2022-06-24 13:29:40,343 INFO org.apache.spark.network.shuffle.RemoteBlockPushResolver: Updated the active local dirs [/grid/i/tmp/yarn/, /grid/g/tmp/yarn/, /grid/b/tmp/yarn/, /grid/e/tmp/yarn/, /grid/h/tmp/yarn/, /grid/f/tmp/yarn/, /grid/d/tmp/yarn/, /grid/c/tmp/yarn/] for application application_1653753500486_3193550
 {code}

node-x was selected as a merger by the driver after 13:28:11 and when the executors started pushing to it, all those pushes failed until 13:29:40

We can fix by having the executor register with ESS before it registers the block manager with the {{BlockManagerMaster}}
Bug: Add RangePartitioning to DataSource V2
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Jun/22 17:23
Affects Version/s: 3.4.0

DataSourceV2 allows data sources to report existing partitioning of read data (org.apache.spark.sql.connector.read.partitioning). Currently, there is only KeyGroupedPartitioning and UnknownPartitioning. Data sources should be able to report global ordered data so that downstream operations can exploit this.

The following is required for this to work:
- Define RangePartitioning as a new implementation of Partitioning
- Add Catalyst rules that handle this partitioning
- Add a test source that reports ordering to proof that subsequent operations that require order do not invoke sorting the data.
Bug: Rework MISSING_COLUMN error class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/22 17:07
Affects Version/s: 3.3.1

"MISSING_COLUMN" : {
"message" : [
"Column '<columnName>' does not exist. Did you mean one of the following? [<proposal>]"
],
"sqlState" : "42000"

Is unfortunately named. It is more accurate to talk about an UNRESOLVED_COLUMN or an UNRESOLVED_COLUMN_IDENTIFIER since we could refer to an alias, a SQL UDF parameter, a field, or, in the future, a variable.
Bug: PySpark need to support pypy3.8 to avoid "No module named '_pickle"
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/22 02:02
Affects Version/s: 3.4.0

{code:java}
Starting test(pypy3): pyspark.sql.tests.test_arrow (temp output: /tmp/pypy3__pyspark.sql.tests.test_arrow__jx96qdzs.log)
Traceback (most recent call last):
  File "/usr/lib/pypy3.8/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/pypy3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/__w/spark/spark/python/pyspark/__init__.py", line 59, in <module>
    from pyspark.rdd import RDD, RDDBarrier
  File "/__w/spark/spark/python/pyspark/rdd.py", line 54, in <module>
    from pyspark.java_gateway import local_connect_and_auth
  File "/__w/spark/spark/python/pyspark/java_gateway.py", line 32, in <module>
    from pyspark.serializers import read_int, write_with_length, UTF8Deserializer
  File "/__w/spark/spark/python/pyspark/serializers.py", line 68, in <module>
    from pyspark import cloudpickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/__init__.py", line 4, in <module>
    from pyspark.cloudpickle.cloudpickle import *  # noqa
  File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle.py", line 57, in <module>
    from .compat import pickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/compat.py", line 13, in <module>
    from _pickle import Pickler  # noqa: F401
ModuleNotFoundError: No module named '_pickle'
Had test failures in pyspark.sql.tests.test_arrow with pypy3; see logs. {code}
Build latest dockerfile pypy3 upgrade to 3.8 (original is 3.7), but it seems cloudpickle has a bug.

This may related: https://github.com/cloudpipe/cloudpickle/commit/8bbea3e140767f51dd935a3c8f21c9a8e8702b7c, but I try to apply this, also failed. Need a deeper look, if you guys know the reason of this, pls let me know.