av9ash/GitBugs huggingface dataset:
Bug: Improve performance for last_value over unbounded preceding window frame
Status: Resolved | Priority: Major | Resolution: Later
Created: 28/Jan/21 04:13
Affects Version/s: 3.2.0

The current implement of `last_value` over unbounded preceding window frame will execute `updateExpressions` multiple times.
In fact, `last_value` only execute `updateExpressions` once on the last row.
Bug: Add metrics for data loss and offset out range for KafkaMicroBatchStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/21 07:53
Affects Version/s: 3.2.0

When testing SS, I found it is hard to track data loss of SS reading from Kafka. The micro scan node has only one metric, number of output rows. Users have no idea how many times offsets to fetch are out of Kafak now, how many times data loss happens.
Bug: Two-tier encryption key management
Status: Open | Priority: Major | Resolution: 
Created: 03/Jan/21 14:48
Affects Version/s: 3.2.0

Columnar data formats (Parquet and ORC) have recently added a column encryption capability. The data protection follows the practice of envelope encryption, where the Data Encryption Key (DEK) is freshly generated for each file/column, and is encrypted with a master key (or an intermediate key, that is in turn encrypted with a master key). The master keys are kept in a centralized Key Management Service (KMS) - meaning that each Spark worker needs to interact with a (typically slow) KMS server. 

This Jira (and its sub-tasks) introduce an alternative approach, that on one hand preserves the best practice of generating fresh encryption keys for each data file/column, and on the other hand allows Spark clusters to have a scalable interaction with a KMS server, by delegating it to the application driver. This is done via two-tier management of the keys, where a random Key Encryption Key (KEK) is generated by the driver, encrypted by the master key in the KMS, and distributed by the driver to the workers, so they can use it to encrypt the DEKs, generated there by Parquet or ORC libraries. In the workers, the KEKs are distributed to the executors/threads in the write path. In the read path, the encrypted KEKs are fetched by workers from file metadata, decrypted via interaction with the driver, and shared among the executors/threads.

The KEK layer further improves scalability of the key management, because neither driver or workers need to interact with the KMS for each file/column.

Stand-alone Parquet/ORC libraries (without Spark) and/or other frameworks (e.g., Presto, pandas) must be able to read/decrypt the files, written/encrypted by this Spark-driven key management mechanism - and vice-versa. [of course, only if both sides have proper authorisation for using the master keys in the KMS]

A link to a discussion/design doc is attached.
Bug: Jars or file paths which contain spaces are generating FileNotFoundException exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/21 13:26
Affects Version/s: 3.0.1

 Whenever I try to use jars or files whose paths contain spaces FileNotFoundException exception is generated

just run spark-shell command having spaces in paths

See below
{code:java}
c:>spark-shell --files "c:\Program Files\...\myjar.jar"
or
c:>spark-shell --jars"c:\Program Files\...\myjar.jar"
or 
c:>spark-shell --conf spark.jars="c:\Program Files\...\myjar.jar"

any combination produce the same exception

 java.io.FileNotFoundException: Jar c:\Program%20Files\........ not found
        at org.apache.spark.SparkContext.addLocalJarFile$1(SparkContext.scala:1833)
        at org.apache.spark.SparkContext.addJar(SparkContext.scala:1887)
        at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:490)
        at org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:490)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:490)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)
        at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
{code}
I have noticed that the following function is causing the issue.
{code:java}
def addJar(path: String): Unit = {
...
} else {
  val uri = new Path(path).toUri
  {code}
the path as the string is 

"file:///C:/Program%20Files/Nokia/....jar"

this call generates the following uri

"file:///C:/Program*%25*20Files/Nokia/....jar"

which results in an invalid file name.

Using 
{code:java}
val uri = Utils.resolveURI(path){code}
seems to resolve the issue.

 Until a fix is provided are there any workarounds to overcome this issue?
Bug: Filter predicate reorder
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Jan/21 06:40
Affects Version/s: 3.2.0

Reorder filter predicate to improve query performance:
{noformat}
others < In < Like < UDF/CaseWhen/If < Inset < LikeAny/LikeAll
{noformat}
[https://www.ibm.com/support/knowledgecenter/SSSHTQ_8.1.0/com.ibm.netcool_OMNIbus.doc_8.1.0/omnibus/wip/admin/reference/omn_adm_per_optimizationrules.html#omn_adm_per_optimizationrules__reorder]
 [https://docs.oracle.com/en/database/oracle/oracle-database/21/addci/extensible-optimizer-interface.html#GUID-28A4EDA6-19DD-4773-B3B8-1802C3B01E21]
 [https://docs.oracle.com/cd/B10501_01/server.920/a96533/hintsref.htm#13676]

https://issues.apache.org/jira/browse/HIVE-21857
https://issues.apache.org/jira/browse/IMPALA-2805
Bug: DataSource V2: support static partitioning on required distribution and ordering
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/21 06:23
Affects Version/s: 3.2.0

SPARK-34026 addressed the functionality of requiring repartition and sort order from data source, but left the number of partitions during repartition as depending on the config (default number of shuffle partitions).

Some special data sources may require the "static number of partitions" during repartition - for example, state data source. Spark stores the state via partitioned by "hash(group key) % default number of shuffle partitions", which means state data source should do the same to rewrite the state data. And the data source is required to "change" the default number of shuffle partitions, as the value is not guaranteed to be same, and also there's a chance we change the number of partitions to non-static one (like letting AQE decides it, SPARK-34230).

This issue tracks the effort to support static number of partitions during repartition.
Bug: Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/21 23:31
Affects Version/s: 3.2.0

MapReduce jobs can instruct YARN to skip renewal of tokens obtained from certain hosts by specifying the hosts with configuration mapreduce.job.hdfs-servers.token-renewal.exclude=<host1>,<host2>,..,<hostN>.

But seems Spark lacks of similar option. So the job submission fails if YARN fails to renew DelegationToken for any of the remote HDFS cluster.  The failure in DT renewal can happen due to many reason like Remote HDFS does not trust Kerberos identity of YARN etc.
Bug: MemoryConsumer's memory mode should respect MemoryManager's memory mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/21 12:48
Affects Version/s: 2.4.7

Currently, the memory mode always set to ON_HEAP for memory consumer when it's not explicitly set.

However, we actually can know the specific memory mode by taskMemoryManager.getTungstenMemoryMode().

 

[https://github.com/apache/spark/blob/3a299aa6480ac22501512cd0310d31a441d7dfdc/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java#L43-L45]
Bug: Support ZSTD compression in ORC data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/21 06:29
Affects Version/s: 3.2.0

h3. What changes were proposed in this pull request?

This PR aims to support ZSTD compression in ORC data source.
h3. Why are the changes needed?

Apache ORC 1.6 supports ZSTD compression to generate more compact files and save the storage cost.

*BEFORE*
{code:java}
scala> spark.range(10).write.option("compression", "zstd").orc("/tmp/zstd")
 java.lang.IllegalArgumentException: Codec [zstd] is not available. Available codecs are uncompressed, lzo, snappy, zlib, none. {code}
*AFTER*
{code:java}
scala> spark.range(10).write.option("compression", "zstd").orc("/tmp/zstd") {code}
{code:java}
 $ orc-tools meta /tmp/zstd 
 Processing data file file:/tmp/zstd/part-00011-a63d9a17-456f-42d3-87a1-d922112ed28c-c000.orc [length: 230]
 Structure for file:/tmp/zstd/part-00011-a63d9a17-456f-42d3-87a1-d922112ed28c-c000.orc
 File Version: 0.12 with ORC_14
 Rows: 1
 Compression: ZSTD
 Compression size: 262144
 Calendar: Julian/Gregorian
 Type: struct<id:bigint>
Stripe Statistics:
 Stripe 1:
 Column 0: count: 1 hasNull: false
 Column 1: count: 1 hasNull: false bytesOnDisk: 6 min: 9 max: 9 sum: 9
File Statistics:
 Column 0: count: 1 hasNull: false
 Column 1: count: 1 hasNull: false bytesOnDisk: 6 min: 9 max: 9 sum: 9
Stripes:
 Stripe: offset: 3 data: 6 rows: 1 tail: 35 index: 35
 Stream: column 0 section ROW_INDEX start: 3 length 11
 Stream: column 1 section ROW_INDEX start: 14 length 24
 Stream: column 1 section DATA start: 38 length 6
 Encoding column 0: DIRECT
 Encoding column 1: DIRECT_V2
File length: 230 bytes
 Padding length: 0 bytes
 Padding ratio: 0%
User Metadata:
 org.apache.spark.version=3.2.0{code}
Bug: AggregateWindowFunction#frame should not always use UnboundedPreceding
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Jan/21 07:13
Affects Version/s: 3.0.2

How to reproduce this issue:
{code:sql}
CREATE TABLE test_unboundedpreceding(SELLER_ID INT) using parquet;

SELECT
  DENSE_RANK() OVER (
    ORDER BY SELLER_ID ROWS BETWEEN 10 PRECEDING
    AND CURRENT ROW
  ) AS SELLER_RANK
FROM
  test_unboundedpreceding
{code}

It will throw:
{noformat}
Error: Error running query: org.apache.spark.sql.AnalysisException: Window Frame specifiedwindowframe(RowFrame, -10, currentrow$()) must match the required frame specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$()); (state=,code=0)
{noformat}

Related code:
https://github.com/apache/spark/blob/cde697a479a2f67c6bc4281f39a1ab2ff6a9d17d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L514
Bug: Eliminate distinct from more aggregates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/21 15:59
Affects Version/s: 3.2.0

The EliminateDistinct rule removes distinct from min and max aggregates.
It could remove it from several more aggregates:
* BitAndAgg
* BitOrAgg
* First
* Last
* HyperLogLogPlusPlus
* CollectSet
Bug: Upgrade Scala to avoid Scala's bug about illegal cyclic reference
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 27/Jan/21 07:57
Affects Version/s: 3.2.0

There is a regression in Scala 2.13. We should probably upgrade maintenance version of Scala 2.13 before Spark 3.2.0 release. Otherwise:

{code}
scalaVersion := "2.13.2"

libraryDependencies ++= Seq(
  "org.scala-lang" % "scala-reflect" % scalaVersion.value,
  "com.google.protobuf" % "protobuf-java" % "3.11.4"
)
{code}

{code}
package foo

import scala.reflect.runtime.universe
import com.google.protobuf.DescriptorProtos.FileDescriptorProto

object Hello {
  def fromJava(t: FileDescriptorProto): Unit = {}
}

object Main {
  def main(args: Array[String]): Unit = {
    val mirror = universe.runtimeMirror(getClass.getClassLoader())

    println(
      mirror.reflectModule(
        mirror.staticModule("foo.Hello$"))
      .instance)
  }
{code}

fails with the error below:

{code}
[info] running foo.Main
[error] (run-main-b) scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving type BuilderType
{code}

See https://github.com/scala/bug/issues/12038 for more details.
Bug: Master may not remove the finished executor when Worker fails to send ExecutorStateChanged
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 15:04
Affects Version/s: 2.4.7

If the Worker fails to send ExecutorStateChanged to the Master due to some errors, e.g., temporary network error, then the Master can't remove the finished executor normally and think the executor is still alive. In the worst case, if the executor is the only one executor for the application, the application can get hang.
Bug: Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/21 07:09
Affects Version/s: 3.0.0

Both AQE and DPP cannot be applied at the same time. This PR will enable AQE and DPP when the join is Broadcast hash join at the beginning.
Bug: Combines all adjacent 'Union' operators into a single 'Union' when using 'Dataset.union.distinct.union.distinct'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/21 03:06
Affects Version/s: 3.2.0

Problem:

Currently when using 'Dataset.union.distinct.union.distinct' to union some datasets, Optimizer can't combine all adjacent 'Union' operators into a single 'Union', but it can handle this case when using sql.

For example:

!image-2021-01-29-11-12-44-112.png!

The 'Physical Plan' is shown below:

!image-2021-01-29-11-13-42-055.png!

But using sql:

!image-2021-01-29-11-14-08-822.png!

The 'Physical Plan' is shown below:

!image-2021-01-29-11-14-42-700.png!

 

Root cause:

When using 'Dataset.union.distinct.union.distinct', the operator is  'Deduplicate(Keys, Union)', but AstBuilder transform sql 'Union' to operator 'Distinct(Union)', the rule 'CombineUnions' in Optimizer only handle 'Distinct(Union)' operator but not Deduplicate(Keys, Union).
Bug: Fix the test code build error for docker-integration-tests in branch-3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/21 08:58
Affects Version/s: 3.0.0

Currently, building test code for docker-integration-tests in branch-3.0 seems to fail due to ojdbc6:11.2.0.1.0 and jdb2jcc4:10.5.0.5 are not available
Bug: The tree string does not contain statistics for nested scalar sub queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/21 14:56
Affects Version/s: 3.2.0

How to reproduce:
{code:scala}
spark.sql("create table t1 using parquet as select id as a, id as b from range(1000)")
spark.sql("create table t2 using parquet as select id as c, id as d from range(2000)")

spark.sql("ANALYZE TABLE t1 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("ANALYZE TABLE t2 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("set spark.sql.cbo.enabled=true")

spark.sql(
  """
    |WITH max_store_sales AS
    |  (SELECT max(csales) tpcds_cmax
    |  FROM (SELECT
    |    sum(b) csales
    |  FROM t1 WHERE a < 100 ) x),
    |best_ss_customer AS
    |  (SELECT
    |    c
    |  FROM t2
    |  WHERE d > (SELECT * FROM max_store_sales))
    |
    |SELECT c FROM best_ss_customer
    |""".stripMargin).explain("cost")
{code}

Output:
{noformat}
== Optimized Logical Plan ==
Project [c#4263L], Statistics(sizeInBytes=31.3 KiB, rowCount=2.00E+3)
+- Filter (isnotnull(d#4264L) AND (d#4264L > scalar-subquery#4262 [])), Statistics(sizeInBytes=46.9 KiB, rowCount=2.00E+3)
   :  +- Aggregate [max(csales#4260L) AS tpcds_cmax#4261L]
   :     +- Aggregate [sum(b#4266L) AS csales#4260L]
   :        +- Project [b#4266L]
   :           +- Filter ((a#4265L < 100) AND isnotnull(a#4265L))
   :              +- Relation default.t1[a#4265L,b#4266L] parquet, Statistics(sizeInBytes=23.4 KiB, rowCount=1.00E+3)
   +- Relation default.t2[c#4263L,d#4264L] parquet, Statistics(sizeInBytes=46.9 KiB, rowCount=2.00E+3)
{noformat}

Another case is TPC-DS q23a.
Bug: Support table valued command
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jan/21 10:09
Affects Version/s: 3.2.0

Some command used to display some metadata, such as: SHOW TABLES, SHOW TABLE EXTENDED,SHOW TBLPROPERTIES and so no.

If the output rows much than screen height, the output very unfriendly to developers.
So we should have a way to filter the output like the behavior of SELECT ... FROM ... WHERE.

We could adopt the implement of table valued function.
Bug: Add rule WindowFrameCoercion into ANSI implicit cast rules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 16:10
Affects Version/s: 3.1.2

To make review easier, the rule WindowFrameCoercion is not added to ANSI implicit cast rules in PR https://github.com/apache/spark/pull/31349
We should add it back and add new test cases.
Bug: Major slowdown in spark streaming after 6 days
Status: Open | Priority: Major | Resolution: 
Created: 11/Jan/21 00:02
Affects Version/s: 3.0.0

Spark streaming application runs at 60s batch intervals.

The application runs fine processing batches around 40s. After ~8600 batches (around 6 days), the application all of a sudden hits a wall and processing time jumps to 2-2.4 minutes, and eventually dies with exit code 137. This happens consistently every 6 days, regardless of data. 

Looking at the application logs, it seems like when the issue begins, tasks are being completed by executors, however the driver is taking a while to acknowledge. I have taken numerous memory dumps of the driver (before it hits the 6 day wall) using *jcmd* and can see the *org.apache.spark.scheduler.AsyncEventQueue* is growing in size despite the fact that the application is able to keep up with batches. I have yet to take a snapshot of the application in the broken state.
Bug: ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/21 10:50
Affects Version/s: 3.0.1

Here is the example to reproduce the issue:
{code:sql}
spark-sql> create table tbl (col int, part int) using parquet partitioned by (part);
spark-sql> insert into tbl partition (part=0) select 0;
spark-sql> cache table tbl;
spark-sql> select * from tbl;
0	0
spark-sql> show table extended like 'tbl' partition(part=0);
default	tbl	false	Partition Values: [part=0]
Location: file:/Users/maximgekk/proj/recover-partitions-refresh-cache/spark-warehouse/tbl/part=0
...
{code}
Add new partition by copying the existing one:
{code}
cp -r /Users/maximgekk/proj/recover-partitions-refresh-cache/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/recover-partitions-refresh-cache/spark-warehouse/tbl/part=1
{code}
 Recover and select the table:
{code}
spark-sql> alter table tbl recover partitions;
spark-sql> select * from tbl;
0	0
{code}
We see only old data.
Bug: Impove preformace for char varchar padding and length check with StaticInvoke
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/21 16:55
Affects Version/s: 3.1.0

This could reduce the `generate.java` size to prevent codegen fallback which causes performance regression.

here is a case from tpcds that could be fixed by this improvementent
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133964/testReport/org.apache.spark.sql.execution/LogicalPlanTagInSparkPlanSuite/q41/
Bug: Use available offset range obtained during polling when checking offset validation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 07:24
Affects Version/s: 2.4.7

We support non-consecutive offsets for Kafka since 2.4.0. In `fetchRecord`, we do offset validation by checking if the offset is in available offset range. But currently we obtain latest available offset range to do the check. It looks not correct as the available offset range could be changed during the batch, so the available offset range is different than the one when we polling the records from Kafka.

It is possible that an offset is valid when polling, but at the time we do the above check, it is out of latest available offset range. We will wrongly consider it as data loss case and fail the query or drop the record.
Bug: FileSource table null partition can not be dropped
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/21 11:19
Affects Version/s: 3.0.1

*case to reproduce*

create table ppp(i string, j string) using parquet partitioned by (j);

INSERT INTO ppp PARTITION (j=null) VALUES ('1');
 alter table ppp drop partition(j=null); --- success

INSERT OVERWRITE ppp VALUES ('1', null);  --- error
Error in query: No partition is dropped. One partition spec 'Map(j -> null)' does not exist in table 'ppp' database 'default';

alter table ppp drop partition(j='__HIVE_DEFAULT_PARTITION__'); --- then, this is need
Bug: Adding partitions to fully partitioned v2 table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/21 10:14
Affects Version/s: 3.2.0

The test below fails:
{code:scala}
    withNamespaceAndTable("ns", "tbl") { t =>
      sql(s"CREATE TABLE $t (p0 INT, p1 STRING) $defaultUsing PARTITIONED BY (p0, p1)")
      sql(s"ALTER TABLE $t ADD PARTITION (p0 = 0, p1 = 'abc')")
      checkPartitions(t, Map("p0" -> "0", "p1" -> "abc"))
      checkAnswer(sql(s"SELECT * FROM $t"), Row(0, "abc"))
    }
{code}
Bug: ExtractGenerator analyzer should handle lazy projectlists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/21 23:56
Affects Version/s: 3.2.0

With the dataframe api it is possible to have a lazy sequence as the output field on a LogicalPlan class. When exploding a column on this dataframe using the withColumn method, the ExtractGenerator does not extract the generator.
Bug: LDAP authentication of spark thrift server support user id mapping
Status: Open | Priority: Major | Resolution: 
Created: 16/Jan/21 07:31
Affects Version/s: 3.0.1

I'm trying to configure LDAP authentication of spark thrift server, and would like to implement user id mapping to mail address.

My scenario is, "uid" is the key of our LDAP system, and "mail"(email address) is one of attributes. Now we want users to input email address, i.e. "mail" when they login thrift client. That is to map "username" input to mail attribute query. e.g.

{code:none}
hive.server2.authentication.ldap.customLDAPQuery="(&(objectClass=person)(mail=${uid}))"
{code}
Bug: SparkR Daemon Initialization
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jan/21 20:16
Affects Version/s: 3.2.0

Provide a way for users to initialize the sparkR daemon before it forks.

I'm a contractor to Target, where we have several projects doing ML with sparkR. The changes proposed here result in weeks of compute-time saved with every run.

Please see [docs/sparkr.md#daemon-initialization|https://github.com/WamBamBoozle/spark/blob/daemon_init/docs/sparkr.md#daemon-initialization].
Bug: Remove view checks in v1 alter table commands
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/21 15:45
Affects Version/s: 3.2.0

After migration on new resolution framework in SPARK-29900, view checks are not needed in the commands:
  - AlterTableAddPartitionCommand
  - AlterTableDropPartitionCommand
  - AlterTableRenamePartitionCommand
  - AlterTableRecoverPartitionsCommand
  - AlterTableSerDePropertiesCommand

So, the checks DDLUtils.verifyAlterTableType can be removed from the v1 commands.
Bug: [AVRO] Improve error messages when matching Catalyst-to-Avro schemas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/21 21:24
Affects Version/s: 2.4.0

Broken off as a separate task from SPARK-34133. Improve the error messages thrown by {{AvroSerializer}}/{{AvroDeserializer}} when an incompatible schema is detected. Examples to follow in the PR.