av9ash/GitBugs huggingface dataset:
Bug: HiveOrcHadoopFsRelationSuite fails with seed 610710213676
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/21 20:00
Affects Version/s: 3.0.2

The test "test all data types" in HiveOrcHadoopFsRelationSuite fails with:
{code:java}
== Results ==
!== Correct Answer - 20 ==    == Spark Answer - 20 ==
 struct<index:int,col:date>   struct<index:int,col:date>
 [1,1582-10-15]               [1,1582-10-15]
 [2,null]                     [2,null]
 [3,1970-01-01]               [3,1970-01-01]
 [4,1681-08-06]               [4,1681-08-06]
 [5,1582-10-15]               [5,1582-10-15]
 [6,9999-12-31]               [6,9999-12-31]
 [7,0583-01-04]               [7,0583-01-04]
 [8,6077-03-04]               [8,6077-03-04]
![9,1582-10-06]               [9,1582-10-15]
 [10,1582-10-15]              [10,1582-10-15]
 [11,9999-12-31]              [11,9999-12-31]
 [12,9722-10-04]              [12,9722-10-04]
 [13,0243-12-19]              [13,0243-12-19]
 [14,9999-12-31]              [14,9999-12-31]
 [15,8743-01-24]              [15,8743-01-24]
 [16,1039-10-31]              [16,1039-10-31]
 [17,9999-12-31]              [17,9999-12-31]
 [18,1582-10-15]              [18,1582-10-15]
 [19,1582-10-15]              [19,1582-10-15]
 [20,1582-10-15]              [20,1582-10-15]
{code}
Bug: Recognize `spark_catalog` in new identifier while view/table renaming
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Feb/21 09:13
Affects Version/s: 3.2.0

Currently, v1 ALTER TABLE .. RENAME TO doesn't recognize spark_catalog in new view/table identifiers. The example below demonstrates the issue:
{code:scala}
spark-sql> CREATE DATABASE db;
spark-sql> CREATE TABLE spark_catalog.db.tbl (c0 INT) USING parquet;
spark-sql> INSERT INTO spark_catalog.db.tbl SELECT 0;
spark-sql> SELECT * FROM spark_catalog.db.tbl;
0
spark-sql> ALTER TABLE spark_catalog.db.tbl RENAME TO spark_catalog.db.tbl2;
Error in query: spark_catalog.db.tbl2 is not a valid TableIdentifier as it has more than 2 name parts.
{code}
Bug: Deprecate spark.sql.legacy.replaceDatabricksSparkAvro.enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/21 18:22
Affects Version/s: 3.2.0

Mark spark.sql.legacy.replaceDatabricksSparkAvro.enabled as deprecated, and recommend to use `.format("avro")` in `DataFrameWriter` or `DataFrameReader` instead.
Bug: Add alternatives for datetime rebasing SQL configs and deprecate legacy configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/21 12:06
Affects Version/s: 3.2.0

The rebasing SQL configs like spark.sql.legacy.parquet.datetimeRebaseModeInRead can be used not only for migration from previous Spark versions but also to read/write datatime columns saved by other systems/frameworks/libs. The ticket aims to move the configs from the legacy namespace by introducing alternatives (like spark.sql.parquet.datetimeRebaseModeInRead) and deprecating the legacy configs (spark.sql.legacy.parquet.datetimeRebaseModeInRead).
Bug: Fix PostgresDialect to handle money types properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 17:20
Affects Version/s: 3.2.0

In the current master, PostgresDialect supports money and money[] types.
But those types seems not to be able to handle those types properly. 

Error of money type.
{code}
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.204 executor driver): org.postgresql.util.PSQLException: Bad value for type double : 1,000.00
[info] 	at org.postgresql.jdbc.PgResultSet.toDouble(PgResultSet.java:3104)
[info] 	at org.postgresql.jdbc.PgResultSet.getDouble(PgResultSet.java:2432)
[info] 	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$5(JdbcUtils.scala:418)
{code}

Error of money[] type.
{code}
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.204 executor driver): org.postgresql.util.PSQLException: Bad value for type double : $2,000.00
[info] 	at org.postgresql.jdbc.PgResultSet.toDouble(PgResultSet.java:3104)
[info] 	at org.postgresql.jdbc.ArrayDecoding$5.parseValue(ArrayDecoding.java:235)
[info] 	at org.postgresql.jdbc.ArrayDecoding$AbstractObjectStringArrayDecoder.populateFromString(ArrayDecoding.java:122)
[info] 	at org.postgresql.jdbc.ArrayDecoding.readStringArray(ArrayDecoding.java:764)
[info] 	at org.postgresql.jdbc.PgArray.buildArray(PgArray.java:310)
[info] 	at org.postgresql.jdbc.PgArray.getArrayImpl(PgArray.java:171)
[info] 	at org.postgresql.jdbc.PgArray.getArray(PgArray.java:111)
{code}

These errors seem to related to the following issues.
https://github.com/pgjdbc/pgjdbc/issues/100
https://github.com/pgjdbc/pgjdbc/issues/1405
Bug: Add driver and executors POD logs to integration tests log when the test fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/21 20:57
Affects Version/s: 3.2.0

Now both the driver and executors logs are lost.

In https://spark.apache.org/developer-tools.html there is a hint:
"Getting logs from the pods and containers directly is an exercise left to the reader."

But when the test is executed by Jenkins and there is failure we really need the POD logs to analyze problem.
Bug: Make `spark.sql.legacy.replaceDatabricksSparkAvro.enabled` as non-internal
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Feb/21 07:55
Affects Version/s: 3.2.0

The SQL config spark.sql.legacy.replaceDatabricksSparkAvro.enabled has been already documented in Spark SQL guide, in fact. Need to make it non-internal as it is documented publically.
Bug: Session window support in SS
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Feb/21 23:24
Affects Version/s: 3.2.0

Currently structured streaming supports two kinds of windows: tumbling window and sliding window. Another useful window function is session window. Which is not supported by SS. We have user requirement to use session window. We'd like to have this support in the upstream.

About session window, there is some info: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows.
Bug: Support parquet datasource options to control datetime rebasing in read
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/21 14:50
Affects Version/s: 3.2.0

Add new parquet options similar to the SQL configs {{spark.sql.legacy.parquet.datetimeRebaseModeInRead}} and {{spark.sql.legacy.parquet.int96RebaseModeInRead.}}
Bug: CSV(/JSON?) files with corrupt row + Permissive mode can yield wrong partial result row
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Feb/21 13:37
Affects Version/s: 2.4.7

(This was actually found and fixed in spark-xml, which copied some Spark code for handling bad records. See https://github.com/databricks/spark-xml/issues/517 )

When CSV parsing (or, I think JSON?) encounters a bad record, in Permissive mode, it can return a partial result of values that were successfully parsed, along with the problem input in a new 'corrupt record' column.

However the logic in FailureSafeParser that copies the partial results to the resulting Row has an off-by-one error that arises when the catalyst projection puts the 'corrupt record' column anywhere but the last column, which can readily happen. This could mean the resulting partial results are wrong, or, that processing the bad record in permissive mode fails entirely, if the resulting elements don't happen to match the schema of the result.

The partial results are usually not that useful, so being wrong isn't a huge deal, but, failing entirely in permissive mode is a problem.
Bug: CatalogImpl.uncacheTable should invalidate in cascade for temp views 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/21 18:08
Affects Version/s: 3.2.0

When {{spark.sql.legacy.storeAnalyzedPlanForView}} is false, {{CatalogImpl.uncacheTable}} should invalidate caches for temp view in cascade.
Bug: PostgresDialect can't treat arrays of some types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/21 13:33
Affects Version/s: 3.2.0

PostgresDialect implements the logic to treat array of some types but it's not enough.
Currently, the following types can't be treated.

* xml
* tsvector
* tsquery
* macaddr
* txid_snapshot
* point
* line
* lseg
* box
* path
* polygon
* circle
* pg_lsn
Bug: Format DateLiteral and TimestampLiteral toString
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/21 09:42
Affects Version/s: 3.2.0

Format DateLiteral and TimestampLiteral toString. For example:
 SELECT * FROM date_dim WHERE d_date BETWEEN (cast('2000-03-11' AS DATE) - INTERVAL 30 days) AND (cast('2000-03-11' AS DATE) + INTERVAL 30 days)
 Current:

 

{{Condition : (((isnotnull(d_date#18) AND (d_date#18 >= 10997)) AND (d_date#18 <= 11057)) AND isnotnull(d_date_sk#17))}}

 

Expected:

 

{{Condition : (((isnotnull(d_date#14) AND (d_date#14 >= 2000-02-10)) AND (d_date#14 <= 2000-04-10)) AND isnotnull(d_date_sk#13))}}
Bug: Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/21 20:49
Affects Version/s: 2.3.1

This came up in https://github.com/apache/spark/pull/31133#issuecomment-773567152.


The use case is the following there is a partitioned Hive table with Avro data. The schema is specified via "avro.schema.url".
With time the schema is evolved and the new schema is set for the table "avro.schema.url" when data is read from the old partition this new evolved schema must be used.
Bug: "Proleptic" date off by 10 days when returned by .collectAsList
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 06/Feb/21 18:52
Affects Version/s: 3.0.1

Run the following commands using Spark 3.0.1:

{{scala> spark.sql("select to_timestamp('1582-10-05 02:12:34.997') as data_console").show(false)}}
{{+-----------------------+}}
{{|data_console           |}}
{{+-----------------------+}}
{{|*1582-10-05 02:12:34.997*|}}
{{+-----------------------+}}

{{scala> spark.sql("select to_timestamp('1582-10-05 02:12:34.997') as data_console")}}
{{res3: org.apache.spark.sql.DataFrame = [data_console: timestamp]}}

{{scala> res3.collectAsList}}
{{res4: java.util.List[org.apache.spark.sql.Row] = [[*1582-10-{color:#FF0000}15{color} 02:12:34.997*]]}}

Notice that the returned date is off by 10 days compared to the date returned by the first command.
Bug: applyInPandas doesn't seem to work with StructType output schema 
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 03/Feb/21 18:29
Affects Version/s: 3.0.1

 
{code:java}
df = spark.createDataFrame(
    [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
    ("id", "v"))

def subtract_mean(pdf):
    # pdf is a pandas.DataFrame
    pdf['count'] = pdf.shape[0]
    return pdf{code}
 

 

Using a DDL-formatted string for output schema works fine:
{code:java}
df.groupby("id").applyInPandas(subtract_mean, schema="id long, v double, count int").show()

+---+----+-----+
| id|   v|count|
+---+----+-----+
|  1| 1.0|    2|
|  1| 2.0|    2|
|  2| 3.0|    3|
|  2| 5.0|    3|
|  2|10.0|    3|
+---+----+-----+
{code}
 

 

But using StructType schema (appending a integer count column) fails:
{code:java}
df.groupby("id").applyInPandas(subtract_mean, schema=df.schema.add(StructField('count', IntegerType(), False))).show()

AnalysisException: Cannot resolve column name "count" among (id, v);

{code}
It appears to be looking for the new return field in the input schema?

As a workaround, is there a toDDL method I can use to get the current schema as a DDL string to which I can append the new return fields?
Bug: Support referencing subquery with column aliases by table alias
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 20:42
Affects Version/s: 3.1.2

Allow referencing subquery with column aliases by table alias. For example
{code:sql}
-- This is supported
SELECT t.c1, t.c2 FROM (SELECT 1 AS c1, 1 AS c2) t

-- AnalysisException: cannot resolve '`t.c1`' given input columns: [c1, c2];
SELECT t.c1, t.c2 FROM (SELECT 1 AS a, 1 AS b) t(c1, c2)
{code}
Bug: Introduce relationTypeMismatchHint to UnresolvedTable for a better error message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/21 20:56
Affects Version/s: 3.2.0

The relationTypeMismatchHint in UnresolvedTable can be used to give a hint if the resolved relation is a view. For example, for "ALTER TABLE t ...", if "t" is resolved a view, the error message will also contain a hint, "Please use ALTER VIEW instead."
Bug: Use GenericData as Avro serialization data model can improve Avro write/read performance
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Feb/21 21:48
Affects Version/s: 3.1.2

We found that using "org.apache.avro.generic.GenericData" as Avro serialization data model in Avro writer can significantly improve Avro write performance and slightly improve Avro read performance.

This optimization was originally put up by [~samkhan]  in this PR https://github.com/apache/spark/pull/29354.

We re-evaluated the change "Use GenericData instead of ReflectData when writing Avro data" in that PR and verified it can provide performance improvement in Avro write/read benchmarks.

The base branch is today(2/2/21)'s branch-3.1.

Besides current Avro read/write benchmarks, I also ran some extra benchmarks for nested structs and arrays read/write, these benchmarks were put up in this PR https://github.com/apache/spark/pull/29352 but haven't been merged.

Benchmark results are added in the comment.
Bug: FileTable should not list TRUNCATE in capabilities by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 06:24
Affects Version/s: 3.2.0

abstract class {{FileTable}} now lists {{TRUNCATE}} in its {{capabilities}}, but {{FileTable}} does not know if an implementation really supports truncation or not. Specifically, we can check existing {{FileTable}} implementations including {{AvroTable}}, {{CSVTable}}, {{JsonTable}}, etc. No one implementation really implements {{SupportsTruncate}} in its writer builder.
Bug: When refreshing a view, also refresh its underlying tables
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Feb/21 06:11
Affects Version/s: 3.0.1

For a view, there might be several underlying tables.

In long-running spark server use case, such as zeppelin, kyuubi, livy.

If a table updated, we need refresh this table in current long running spark session.

But if the table is a view, we need refresh the underlying tables one by one.
Bug: Fix the guarantee of foreachBatch
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Feb/21 05:43
Affects Version/s: 3.2.0

Similar to SPARK-28650, {{foreachBatch}} API document also documents the guarantee:

The batchId can be used to deduplicate and transactionally write the output (that is, the provided Dataset) to external systems. The output Dataset is guaranteed to be exactly the same for the same batchId

But like the reason of fixing the document of {{ForeachWriter}} in SPARK-28650, it is not hard to break the guarantee by changing the partition number.
Bug: Refactor spark.udf.register to share the same path to generate UDF instance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Feb/21 12:34
Affects Version/s: 3.2.0

Currently we have two places that create UDFs. One is _create_udf, and another one is {{UserDefinedFunction(...)}}. We should better combine the code path and have single place that create UDF instance
Bug: Add API for all built-in expresssion functions
Status: Open | Priority: Minor | Resolution: 
Created: 04/Feb/21 13:05
Affects Version/s: 3.0.1

From the [SQL functions|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html] documentation:

{quote}Commonly used functions available for DataFrame operations. Using functions defined here provides a little bit more compile-time safety to make sure the function exists. {quote}

Functions such as "inline_outer" are actually commonly used, but are not currently included in the API, meaning that we lose compile-time safety for those invocations. We should implement the required function definitions for the remaining built-in functions when applicable.
Bug: ./build/mvn error output on aarch64
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Feb/21 07:53
Affects Version/s: 3.1.2

You can see the below error ouput in every spark arm jenkins job [1]:
{quote}/build/zinc-0.3.15/bin/nailgun: line 50: /root/dev/spark/build/zinc-0.3.15/bin/ng/linux32/ng: cannot execute binary file: Exec format error
{quote}
And you can also reproduce this error output using the simple cmd: `build/mvn -v` on aarch64:
{quote}root@yikun-arm:~/dev/spark# uname -a
 Linux yikun-arm 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:10 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
 root@yikun-arm:~/dev/spark# uname -m
 aarch64
 root@yikun-arm:~/dev/spark# build/mvn -v
 */root/dev/spark/build/zinc-0.3.15/bin/nailgun: line 50: /root/dev/spark/build/zinc-0.3.15/bin/ng/linux32/ng: cannot execute binary file: Exec format error*
 */root/dev/spark/build/zinc-0.3.15/bin/nailgun: line 50: /root/dev/spark/build/zinc-0.3.15/bin/ng/linux32/ng: cannot execute binary file: Exec format error*
 Using `mvn` from path: /root/dev/spark/build/apache-maven-3.6.3/bin/mvn
 Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
 Maven home: /root/dev/spark/build/apache-maven-3.6.3
 Java version: 1.8.0_222, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-arm64/jre
 Default locale: en, platform encoding: UTF-8
 OS name: "linux", version: "4.15.0-70-generic", arch: "aarch64", family: "unix"
 */root/dev/spark/build/zinc-0.3.15/bin/nailgun: line 50: /root/dev/spark/build/zinc-0.3.15/bin/ng/linux32/ng: cannot execute binary file: Exec format error*
{quote}
The reason of this error output is that the zinc is not supported well on aarch64, unfortunately, the stand alone zinc is deprecated now[2], so that we have no chance to support it on aarch64 in stand alone verison.

Looks like we should skip the zinc related command on aarch64 platform first, and then upgrade the sbt/zinc[3] in future.
  

[1] [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/547/consoleFull]
 [2] [https://github.com/typesafehub/zinc]
 [3] [https://github.com/sbt/zinc]
Bug: Escape meta-characters in printSchema
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Feb/21 06:19
Affects Version/s: 3.2.0

Similar to SPARK-33690, the output layout of printSchema can break if column names contain meta characters.
{code:java}
scala> val df1 = spark.sql("SELECT 'aaa\nbbb\tccc\rddd\feee\bfff\u000Bggg\u0007hhh'")
scala> df1.printSchema
root
 |-- aaa
ddd	ccc
   eefff
        ggghhh: string (nullable = false)
{code}
Bug: Rename GITHUB_API_TOKEN to GITHUB_OAUTH_KEY in translate-contributors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/21 09:26
Affects Version/s: 3.2.0

{code}
dev/create-release/translate-contributors.py:GITHUB_OAUTH_KEY = os.environ.get("GITHUB_OAUTH_KEY", os.environ.get("GITHUB_API_TOKEN", None))
dev/create-release/translate-contributors.py:if not GITHUB_OAUTH_KEY:
dev/create-release/translate-contributors.py:    sys.exit("GITHUB_OAUTH_KEY must be set")
dev/create-release/translate-contributors.py:github_client = Github(GITHUB_OAUTH_KEY)
dev/github_jira_sync.py:GITHUB_OAUTH_KEY = os.environ.get("GITHUB_OAUTH_KEY")
dev/github_jira_sync.py:        request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY)
dev/github_jira_sync.py:        request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY)
dev/merge_spark_pr.py:GITHUB_OAUTH_KEY = os.environ.get("GITHUB_OAUTH_KEY")
dev/merge_spark_pr.py:        if GITHUB_OAUTH_KEY:
dev/merge_spark_pr.py:            request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY)
dev/run-tests-jenkins.py:    github_oauth_key = os.environ["GITHUB_OAUTH_KEY"]
{code}

Spark uses {{GITHUB_OAUTH_KEY}} for GitHub token, but translate-contributors script alone uses {{GITHUB_API_TOKEN}}. We should better match to make it easier to run the script
Bug: Use Caffeine instead of Guava Cache
Status: In Progress | Priority: Minor | Resolution: 
Created: 01/Feb/21 08:11
Affects Version/s: 3.2.0

Caffeine is a high performance, near optimal caching library based on Java 8, it is used in a similar way to guava cache, but with better performance. The comparison results are as follow are on the [caffeine benchmarks |https://github.com/ben-manes/caffeine/wiki/Benchmarks]

At the same time, caffeine has been used in some open source projects like Cassandra, Hbase, Neo4j, Druid, Spring and so on.
Bug: from_json documentation is wrong about malformed JSONs output
Status: Open | Priority: Minor | Resolution: 
Created: 15/Feb/21 21:20
Affects Version/s: 3.0.0

The documentation of the `from_json` function states that malformed json will return a `null` value, which is not the case anymore after https://issues.apache.org/jira/browse/SPARK-25243.
Bug: Ignore or reject wrong config when start sparksession
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Feb/21 02:38
Affects Version/s: 3.2.0

When use 
{code:java}
SparkSession.builder().config()
{code}
In this method user may config `spark.driver.memory`. But when we run this code, jvm is started,  so this configuration won't work and in Spark UI, it will show as this configuration. 

So we should ignore such as wrong way.
Bug: Use randomization as a possibly better technique than grid search in optimizing hyperparameters
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Feb/21 08:48
Affects Version/s: 3.0.1

Randomization can be a more effective techinique than a grid search in finding optimal hyperparameters since min/max points can fall between the grid lines and never be found. Randomisation is not so restricted although the probability of finding minima/maxima is dependent on the number of attempts.

Alice Zheng has an accessible description on how this technique works at [https://www.oreilly.com/library/view/evaluating-machine-learning/9781492048756/ch04.html]

(Note that I have a PR for this work outstanding at [https://github.com/apache/spark/pull/31535] )
Bug: Zinc standalone server is useless after scala-maven-plugin 4.x
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/21 12:37
Affects Version/s: 3.2.0

Zinc is the incremental compiler to speed up builds of compilation.

scala-maven-plugin is the mave plugin, which is used by Spark, one of the function is to integrate the Zinc to enable the incremental compiler.

Since Spark v3.0.0 (SPARK-28759), the scala-maven-plugin is upgraded to v4.X, that means Zinc v0.3.13 standalone server is useless anymore. 

*However, we still [download|https://github.com/apache/spark/blob/4a3200b08ac3e7733b5a3dc7271d35e6872c5967/build/mvn#L107], [install|https://github.com/apache/spark/blob/4a3200b08ac3e7733b5a3dc7271d35e6872c5967/build/mvn#L140], [start|https://github.com/apache/spark/blob/4a3200b08ac3e7733b5a3dc7271d35e6872c5967/build/mvn#L153] the standalone Zinc server. we should remove all zinc standalone server code, and all related configuration.*

 

See more in https://docs.google.com/document/d/1u4kCHDx7KjVlHGerfmbcKSB0cZo6AD4cBdHSse-SBsM
Bug: Add convert_dtypes to the DataFrameLike protocol
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 25/Feb/21 12:38
Affects Version/s: 3.0.1

We have hit an issue of missing `convert_dtypes` in the {{DataFrameLike}} protocol, leading to mypy error. Is there a recommended change path to add a new method to {{DataFrameLike}}?