av9ash/GitBugs huggingface dataset:
Bug: Support separate buffer size configuration in UnsafeShuffleWriter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Aug/24 04:42
Affects Version/s: 4.0.0

{{UnsafeShuffleWriter#mergeSpillsWithFileStream}} uses {{spark.shuffle.file.buffer}} as the buffer for reading spill files, and this buffer is an off-heap buffer.

In the spill process, we hope that the buffer size is larger, but once there are too many files in the spill, {{UnsafeShuffleWriter#mergeSpillsWithFileStream}} needs to create a lot of off-heap memory, which makes the executor easily killed by YARN.

 

[https://github.com/apache/spark/blob/e72d21c299a450e48b3cf6e5d36b8f3e9a568088/core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java#L372-L375]

 
{code:java}
       for (int i = 0; i < spills.length; i++) {
        spillInputStreams[i] = new NioBufferedFileInputStream(
          spills[i].file,
          inputBufferSizeInBytes);{code}
Bug: Lower Spark Connect client log messages to debug level
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/24 11:42
Affects Version/s: 4.0.0

Spark Connect client produces a fair amount of logs but is incorrectly
configured at INFO level. When clients set the global log level to INFO,
a lot of debug level logs are produced.

Lower the Spark Connect client log level to debug to avoid these spurious messages at INFO level.
Bug: Update `Dockerfile` to use `alpine/java:17-jdk` instead of `gradle:8.9.0-jdk17-jammy`
Status: Closed | Priority: Minor | Resolution: Invalid
Created: 16/Aug/24 22:43
Affects Version/s: kubernetes-operator-0.1.0

`alpine` Java 21 image is 65% smaller. And, we currently use grade wrapper to download newer image always.
{code}
$ docker images | head -n3
REPOSITORY                                TAG                                        IMAGE ID       CREATED         SIZE
alpine/java                               17-jdk                                     c6b1a03c388c   6 days ago      318MB
gradle                                    8.9.0-jdk17-jammy                          28e51bcaedf1   5 weeks ago     710MB
{code}
Bug: Fix null type non-codegen ordering exception
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Aug/24 02:43
Affects Version/s: 3.5.2


{code:java}
set spark.sql.codegen.factoryMode=NO_CODEGEN;
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.EliminateSorts;

select * from range(10) order by array(null);
{code}


{code:java}

org.apache.spark.SparkIllegalArgumentException: Type PhysicalNullType does not support ordered operations.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.orderedOperationUnsupportedByDataTypeError(QueryExecutionErrors.scala:352)
    at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:246)
    at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:243)
    at org.apache.spark.sql.catalyst.types.PhysicalArrayType$$anon$1.<init>(PhysicalDataType.scala:283)
    at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering$lzycompute(PhysicalDataType.scala:281)
    at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering(PhysicalDataType.scala:281)
    at org.apache.spark.sql.catalyst.types.PhysicalArrayType.ordering(PhysicalDataType.scala:277)
    at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:67)
    at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:40)
    at org.apache.spark.sql.execution.UnsafeExternalRowSorter$RowComparator.compare(UnsafeExternalRowSorter.java:254)
    at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:70)
    at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:44)

{code}
Bug: Sub categorize cannot load state store errors and catch out of memory errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/24 17:39
Affects Version/s: 3.4.3

* Reclassify a OOM error for both state store providers in loading state store and throw an informative error message

 * Within the loading state store call path, reclassify any non-error-class errors with error-class errors under the {{CANNOT_LOAD_STATE_STORE.}} category
Bug: make JdbcUtils methods makeGetter and makeSetter public to extend custom dialect support for complex types
Status: Open | Priority: Minor | Resolution: 
Created: 08/Aug/24 09:57
Affects Version/s: 4.0.0

Currently, in Apache Spark's {{JdbcUtils}} (across all versions, including Spark 4), there is no support for complex types like {{{}StructType{}}}, {{{}MapType{}}}, except for {{{}ArrayType{}}}. This limitation is evident in the private methods [makeGetter|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L400] and [makeSetter|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L620].

As Spark provides the ability to implement custom dialects, users cannot implement custom handling for these complex types because the methods are private and they encounter the following error:

 

{{org.apache.spark.SparkSQLException: Unsupported type <complex_type>.}}

The suggestion is to change the way these methods are called. Specifically, the getter/setter methods should be able to be provided by the `JdbcDialect`, rather than only being hardcoded in `JdbcUtils`.

I suppose it can be done in one of two ways:

1. Modify the makeGetter and makeSetter methods to delegate to the dialect
    
   {code}
    
    class JdbcUtils {
      def makeGetter(args): Option[Getter] = {
        dialect.makeGetter(args).orElse {
          ...
        }
      }
    
      def makeSetter(args): Option[Setter] = {
        dialect.makeSetter(args).orElse {
          ...
        }
      }
    }
   {code}
    
2. Move the makeGetter/makeSetter methods out of JdbcUtils into JdbcDialect, allowing custom dialects to override them. 
    
    {code}
    
    abstract class JdbcDialect {
      def makeGetter(args): Option[Getter]
      def makeSetter(args): Option[Setter]
    }
    
    object JdbcUtils {
      def getGetter(args): Option[Getter] = {
        // logic to fetch getter from JdbcDialect
      }
    
      def getSetter(args): Option[Setter] = {
        // logic to fetch setter from JdbcDialect
      }
    }
   {code}
Bug: Add support for StructType and MapType in JdbcUtils
Status: Open | Priority: Minor | Resolution: 
Created: 08/Aug/24 10:39
Affects Version/s: 4.0.0

Currently, in Apache Spark's {{JdbcUtils}} (across all versions, including Spark 4), there is no support for {{{}StructType{}}}, {{MapType.}}

The current implementation in {{JdbcUtils}} supports [{{ArrayType}}|https://github.com/apache/spark/blob/6246b18635f073e872e32dd05a699007446aee94/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L687]

To extend the functionality and enable users to read and write {{StructType}} and {{MapType}} from their databases analog, it is proposed to add similar handling for these complex types in the {{makeGetter}} and {{makeSetter}} methods.

For {{MapType}} it might look like:

{code:java}
case MapType(keyType, valueType, valueContainsNull) =>
  (rs: ResultSet, row: InternalRow, pos: Int) =>
    val mapData = rs.getObject(pos + 1).asInstanceOf[Map[Any, Any]]
    if (mapData == null) {
      row.setNullAt(pos)
    } else {
      val keys = mapData.keys.map(getCatalystType(keyType).getJavaValue).toArray
      val values = mapData.values.map(getCatalystType(valueType).getJavaValue).toArray
      row.update(pos, new ArrayBasedMapData(new GenericArrayData(keys), new GenericArrayData(values)))
    } {code}
Bug: Use the latest PMD 6.x rules instead of the deprecated ones
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/24 22:31
Affects Version/s: kubernetes-operator-0.1.0

There are too many warnings on PMD like the following.
{code:java}
Use Rule name category/java/errorprone.xml/BrokenNullCheck instead of the deprecated Rule name rulesets/java/basic.xml/BrokenNullCheck. PMD 7.0.0 will remove support for this deprecated Rule name usage.
Use Rule name category/java/errorprone.xml/CheckSkipResult instead of the deprecated Rule name rulesets/java/basic.xml/CheckSkipResult. PMD 7.0.0 will remove support for this deprecated Rule name usage. {code}
 
{code:java}
$ ./gradlew clean build | grep PMD | wc -l
     204 {code}
 

We had better fix it before we do the release.
Bug: spark-kubernetes-operator-dynamic-configuration ConfigMap Data Overriding Failure
Status: Closed | Priority: Trivial | Resolution: Duplicate
Created: 29/Aug/24 03:53
Affects Version/s: 4.0.0

I have a value.yaml as below:
operatorConfiguration:
    dynamicConfig:
    enable: true
    create: true
    data:
        spark.kubernetes.operator.watchedNamespaces: "default, spark-1"

helm install spark-kubernetes-operator --create-namespace -f build-tools/helm/spark-kubernetes-operator/values.yaml -f tests/e2e/helm/dynamic-config-values.yaml build-tools/helm/spark-kubernetes-operator/

The generated configmap data field does not contains the line spark.kubernetes.operator.watchedNamespaces: "default, spark-1". Note that if you run helm install --dry-run, the record exist
Bug: PySpark Python UDF return inconsistent results when applying 2 UDFs with different return type to 2 columns together
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 25/Feb/21 17:29
Affects Version/s: 3.0.0

Python UDF returns inconsistent results between evaluating 2 columns together and evaluating one by one.

The issue occurs after we upgrading to spark3, so seems it doesn't exist in spark2.

How to reproduce it?

{code:python}
df = spark.createDataFrame([([(1.0, "1"), (1.0, "2"), (1.0, "3")], [(1, "1"), (1, "2"), (1, "3")]), ([(2.0, "1"), (2.0, "2"), (2.0, "3")], [(2, "1"), (2, "2"), (2, "3")]), ([(3.1, "1"), (3.1, "2"), (3.1, "3")], [(3, "1"), (3, "2"), (3, "3")])], ['c1', 'c2'])

from pyspark.sql.functions import udf
from pyspark.sql.types import *

def getLastElementWithTimeMaster(data_type):
    def getLastElementWithTime(list_elm):
        # x should be a list of (val, time)
        y = sorted(list_elm, key=lambda x: x[1]) # default is ascending
        return y[-1][0]
    return udf(getLastElementWithTime, data_type)

# Add 2 columns whcih apply Python UDF
df = df.withColumn("c3", getLastElementWithTimeMaster(DoubleType())("c1"))
df = df.withColumn("c4", getLastElementWithTimeMaster(IntegerType())("c2"))

# Show the results
df.select("c3").show()
df.select("c4").show()
df.select("c3", "c4").show()
{code}

Results:
{noformat}
>>> df.select("c3").show()
+---+                                                                           
| c3|
+---+
|1.0|
|2.0|
|3.1|
+---+
>>> df.select("c4").show()
+---+
| c4|
+---+
|  1|
|  2|
|  3|
+---+
>>> df.select("c3", "c4").show()
+---+----+
| c3|  c4|
+---+----+
|1.0|null|
|2.0|null|
|3.1|   3|
+---+----+
{noformat}

The test was done in branch-3.1 local mode.
Bug: Custom functions can't be used in temporary views with CTEs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 11/Feb/21 06:32
Affects Version/s: 3.1.0

The following query works in Spark 3.0 not Spark 3.1.
  
 Start with:
 {{spark.udf.registerJavaFunction("custom_func", "com.stuff.path.custom_func", LongType())}}
  
 Works: * {{select custom_func()}}
 * {{create temporary view blaah as select custom_func()}}
 * {{with step_1 as ( select custom_func() ) select * from step_1}}

Broken:
 {{create temporary view blaah as with step_1 as ( select custom_func() ) select * from step_1}}
  
 followed by:
 {{select * from blaah}}
  
 Error:
 {{Error in SQL statement: AnalysisException: No handler for UDF/UDAF/UDTF '}}{{com.stuff.path.custom_func}}{{';}}
Bug: io.file.buffer.size set by spark.buffer.size will override by hive-site.xml may cause perf regression
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Feb/21 14:13
Affects Version/s: 3.0.2

In many real-world cases, when interacting with hive catalog through Spark SQL, users may just share the `hive-site.xml` for their hive jobs and make a copy to `SPARK_HOME`/conf w/o modification. In Spark, when we generate Hadoop configurations, we will use `spark.buffer.size(65536)` to reset  `io.file.buffer.size(4096)`. But when we load the hive-site.xml, we may ignore this behavior and reset `io.file.buffer.size` again according to `hive-site.xml`.

1. The configuration priority for setting Hadoop and Hive config here is not right, while literally, the order should be `spark > spark.hive > spark.hadoop > hive > hadoop`

2. This breaks `spark.buffer.size` congfig's behavior for tuning the IO performance w/ HDFS if there is an existing `io.file.buffer.size` in hive-site.xml
Bug: No python3 in docker images 
Status: Resolved | Priority: Critical | Resolution: Cannot Reproduce
Created: 04/Feb/21 00:16
Affects Version/s: 3.0.1

The spark-py container image doesn't receive the instruction to use python3 and defaults to python 2.7

 

The worker container was build using the following commands
{code:java}
mkdir ./tmp
wget -qO- https://www.mirrorservice.org/sites/ftp.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz | tar -C ./tmp/ -xzf -
cd ../spark-3.0.1-bin-hadoop3.2/
./bin/docker-image-tool.sh -r docker.io/timhughes -t spark-3.0.1-bin-hadoop3.2 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
docker push docker.io/timhughes/spark-py:spark-3.0.1-bin-hadoop3.2{code}
 

This is the code I am using to initialize the workers

 
{code:java}
import os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession# Create Spark config for our Kubernetes based cluster manager
sparkConf = SparkConf()
sparkConf.setMaster("k8s://https://kubernetes.default.svc.cluster.local:443")
sparkConf.setAppName("spark")
sparkConf.set("spark.kubernetes.container.image", "docker.io/timhughes/spark-py:spark-3.0.1-bin-hadoop3.2")
sparkConf.set("spark.kubernetes.namespace", "spark")
sparkConf.set("spark.executor.instances", "2")
sparkConf.set("spark.executor.cores", "1")
sparkConf.set("spark.driver.memory", "1024m")
sparkConf.set("spark.executor.memory", "1024m")
sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")
sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")
sparkConf.set("spark.driver.port", "29413")
sparkConf.set("spark.driver.host", "my-notebook-deployment.spark.svc.cluster.local")
# Initialize our Spark cluster, this will actually
# generate the worker nodes.
spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext
{code}
Bug: scala.MatchError: 2.3 (of class java.lang.String) for Hive on Google Dataproc 1.4
Status: Resolved | Priority: Critical | Resolution: Won't Fix
Created: 04/Feb/21 16:15
Affects Version/s: 2.4.7

According to Google Dataproc 1.4 release notes:
 [https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-1.4]
 2.4.7 is supported spark version.

Use *spark-hive_2.11-2.4.7.jar* for Hive on Dataproc 1.4 causes:
{noformat}
scala.MatchError: 2.3 (of class java.lang.String) scala.MatchError: 2.3 
(of class java.lang.String) 
at org.apache.spark.sql.hive.client.IsolatedClientLoader$.hiveVersion(IsolatedClientLoader.scala:89) 
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:300) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:287) at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66) 
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
{noformat}
Bug: docker-image-tool.sh debconf trying to configure kerberos
Status: In Progress | Priority: Critical | Resolution: 
Created: 01/Feb/21 20:39
Affects Version/s: 3.0.1

When building the docker containers using the docker-image-tool.sh there is RUN `apt install -y bash tini libc6 libpam-modules krb5-user libnss3`  Which leads to `debconf` trying to configure kerberos. I have tried putting nothing, EXAMPLE.COM, my corporate kerberos realm and none of them work. it just hangs after enter is pressed

 

{{Setting up krb5-config (2.6) ...}}
{{debconf: unable to initialize frontend: Dialog}}
{{debconf: (TERM is not set, so the dialog frontend is not usable.)}}
{{debconf: falling back to frontend: Readline}}
{{debconf: unable to initialize frontend: Readline}}
{{debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)}}
{{debconf: falling back to frontend: Teletype}}
{{Configuring Kerberos Authentication}}
{{-----------------------------------}}

{{When users attempt to use Kerberos and specify a principal or user name without}}
{{specifying what administrative Kerberos realm that principal belongs to, the}}
{{system appends the default realm. The default realm may also be used as the}}
{{realm of a Kerberos service running on the local machine. Often, the default}}
{{realm is the uppercase version of the local DNS domain.}}

{{Default Kerberos version 5 realm: EXAMPLE.ORG}}
{{^CFailed to build Spark JVM Docker image, please refer to Docker build output for details.}}

 

 

{{## Steps to reproduce}}

{{```}}

{{wget -qO- https://www.mirrorservice.org/sites/ftp.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz | tar -xzf -}}

{{cd spark-3.0.1-bin-hadoop3.2/}}

{{./bin/docker-image-tool.sh build}}

{{```}}
Bug: Pushdown scalar-subquery filter to FileSourceScan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/21 07:06
Affects Version/s: 3.2.0

We can pushdown {{a < (select max(d) from t2)}} to FileSourceScan:
{code:scala}
sql("CREATE TABLE t1 using parquet AS SELECT id AS a, id AS b FROM range(500000000L)")
sql("CREATE TABLE t2 using parquet AS SELECT id AS d FROM range(20)")
sql("SELECT * FROM t1 WHERE b = (select max(d) from t2)").show
{code}
Bug: pyspark toPandas() should return pd.DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/21 16:01
Affects Version/s: 3.1.1

Right now {{toPandas()}} returns {{DataFrameLike}}, which is an incomplete "view" of pandas {{DataFrame}}. Which leads to cases like mypy reporting that certain pandas methods are not present in {{DataFrameLike}}, even tho those methods are valid methods on pandas {{DataFrame}}, which is the actual type of the object. This requires type ignore comments or asserts.
Bug: Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/21 14:20
Affects Version/s: 3.2.0

Snapshots of Spark 3.2 built against Scala 2.13 are available at [https://repository.apache.org/content/repositories/snapshots/org/apache/spark/,] but they seem to depend on Scala 2.12. Specifically if I look at [https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-parent_2.13/3.2.0-SNAPSHOT/spark-parent_2.13-3.2.0-20210223.010629-29.pom] I see:
{code:java}
<scala.version>2.12.10</scala.version>
<scala.binary.version>2.13</scala.binary.version{code}
It looks like [https://github.com/apache/spark/blob/8f994cbb4a18558c2e81516ef1e339d9c8fa0d41/dev/change-scala-version.sh#L65] needs to be updated to also change the `scala.version` and not just the `scala.binary.version`.
Bug: Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Feb/21 00:55
Affects Version/s: 3.0.1

In case Cluster does not have sufficient resource (CPU/ Memory ) for minimum number of executors , the executors goes in Pending State for indefinite time until the resource gets free.

Suppose, Cluster Configurations are:


total Memory=204Gi
used Memory=200Gi
free memory= 4Gi
SPARK.EXECUTOR.MEMORY=10G
SPARK.DYNAMICALLOCTION.MINEXECUTORS=4
SPARK.DYNAMICALLOCATION.MAXEXECUTORS=8

Rather, the job should be cancelled if requested number of minimum executors are not available at that point of time because of resource unavailability.

Currently it is doing partial scheduling or no scheduling and waiting indefinitely. And the job got stuck.
Bug: When we submit spark core tasks frequently, the submitted nodes will have a lot of resource pressure
Status: Reopened | Priority: Major | Resolution: 
Created: 09/Feb/21 03:44
Affects Version/s: 3.0.1

When we submit spark core tasks frequently, the submitted node will have a lot of resource pressure, because spark will create a process instead of a thread for each submitted task. In fact, there is a lot of resource consumption. When the QPS of the submitted task is very high, the submission will fail due to insufficient resources. I would like to ask how to optimize the amount of resources consumed by spark core submission
Bug: Invalid ID for offset-based ZoneId since Spark 3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/21 08:31
Affects Version/s: 3.0.0

How to reproduce this issue:
{code:sql}
select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");
{code}
Spark 2.4:
{noformat}
spark-sql> select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");
2020-02-07 08:00:00
Time taken: 0.089 seconds, Fetched 1 row(s)
{noformat}

Spark 3.x:
{noformat}
spark-sql> select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");
21/02/07 01:24:32 ERROR SparkSQLDriver: Failed in [select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00")]
java.time.DateTimeException: Invalid ID for offset-based ZoneId: GMT+8:00
	at java.time.ZoneId.ofWithPrefix(ZoneId.java:437)
	at java.time.ZoneId.of(ZoneId.java:407)
	at java.time.ZoneId.of(ZoneId.java:359)
	at java.time.ZoneId.of(ZoneId.java:315)
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.getZoneId(DateTimeUtils.scala:53)
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.toUTCTime(DateTimeUtils.scala:814)
{noformat}
Bug: Missing pyspark 3.1.1 doc migration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/21 22:09
Affects Version/s: 3.1.1

The link on the page [https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-docs/_site/pyspark-migration-guide.html]

is not wokring.

Also there is nothing on 3.0 to 3.1.1 on this page

https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-docs/_site/api/python/migration_guide/index.html