av9ash/GitBugs huggingface dataset:
Bug: Convert HiveException to SparkException
Status: Open | Priority: Minor | Resolution: 
Created: 26/Jun/23 11:47
Affects Version/s: 3.5.0

1.Trigger scenario:

[https://github.com/apache/spark/blob/67abc430140558e60c785b158e9199dc884fb15c/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/Hive_2_1_DDLSuite.scala#L102-L113]

 

2.HiveException -> SparkException

[https://github.com/apache/spark/blob/67abc430140558e60c785b158e9199dc884fb15c/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L100-L108]

 

3.How to add a suitable name for this scenario？
Bug: Enable dynamicPartitionOverwrite in SaveAsHiveFile for insert overwrite
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jun/23 11:53
Affects Version/s: 3.4.1

Currently in InsertIntoHiveTable.scala , there is no way to pass dynamicPartitionOverwrite to true , when calling  saveAsHiveFile . When dynamicPartitioOverwrite is true , spark will use  built-in FileCommitProtocol instead of Hadoop FileOutputCommitter , which is more performant. 

 

Here is the solution . 

When inserting overwrite into Hive table

 

Current code 

 
{code:java}
val writtenParts = saveAsHiveFile(
  sparkSession = sparkSession,
  plan = child,
  hadoopConf = hadoopConf,
  fileFormat = fileFormat,
  outputLocation = tmpLocation.toString,
  partitionAttributes = partitionColumns,
  bucketSpec = bucketSpec,
  options = options)
       {code}
 

 

Proposed code.  

enableDynamicPartitionOverwrite 
{code:java}
val USE_FILECOMMITPROTOCOL_DYNAMIC_PARTITION_OVERWRITE =
    buildConf("spark.sql.hive.filecommit.dynamicPartitionOverwrite"){code}
 
{code:java}
 val enableDynamicPartitionOverwrite =
      SQLConf.get.getConf(HiveUtils.USE_FILECOMMITPROTOCOL_DYNAMIC_PARTITION_OVERWRITE)
    logWarning(s"enableDynamicPartitionOverwrite: $enableDynamicPartitionOverwrite"){code}
 

 

Now if enableDynamicPartitionOverwrite is true and numDynamicPartitions > 0 and overwrite is true , pass dynamicPartitionOverwrite true. 

 
{code:java}
val writtenParts = saveAsHiveFile( sparkSession = sparkSession, plan = child, hadoopConf = hadoopConf, fileFormat = fileFormat, outputLocation = tmpLocation.toString, partitionAttributes = partitionColumns, bucketSpec = bucketSpec, options = options, dynamicPartitionOverwrite =
        enableDynamicPartitionOverwrite && numDynamicPartitions > 0 && overwrite)       {code}
 

 

In saveAs File 
{code:java}
val committer = FileCommitProtocol.instantiate(
      sparkSession.sessionState.conf.fileCommitProtocolClass,
      jobId = java.util.UUID.randomUUID().toString,
      outputPath = outputLocation,
      dynamicPartitionOverwrite = dynamicPartitionOverwrite) {code}
This will internal call  with dynamicPartitionOverwrite value true. 

 
{code:java}
class SQLHadoopMapReduceCommitProtocol(
    jobId: String,
    path: String,
    dynamicPartitionOverwrite: Boolean = false)
  extends HadoopMapReduceCommitProtocol(jobId, path, dynamicPartitionOverwrite) {code}
Bug: KafkaDataConsumer to log some stats
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/23 03:27
Affects Version/s: 3.4.0

For each task, Kafka source should report fraction of time spent in KafkaConsumer to fetch records. It should also report overall read bandwidth (bytes or records read / time spent fetching).

This will be useful in verifying if fetching is the bottleneck.
Bug: Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jun/23 07:08
Affects Version/s: 3.5.0

- Refactor `PreWriteCheck` to use error framework.
- Make `INSERT_COLUMN_ARITY_MISMATCH` more generic & avoiding to embed error's text in source code.
- Assign name to _LEGACY_ERROR_TEMP_1170.
Bug: Update some maven plugins to newest version
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jun/23 09:25
Affects Version/s: 3.5.0

include:
 * versions-maven-plugin from 2.15.0 to 2.16.0
 * maven-source-plugin from 3.2.1 to 3.3.0
 * maven-surefire-plugin from 3.1.0 to 3.1.2
 * maven-dependency-plugin from 3.5.0 to 3.6.0
Bug: Mark all `local-cluster` tests as `ExtendedSQLTest`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/23 18:04
Affects Version/s: 3.5.0

This issue aims to mark all `local-cluster` tests as `ExtendedSQLTest`

https://pipelines.actions.githubusercontent.com/serviceHosts/03398d36-4378-4d47-a936-fba0a5e8ccb9/_apis/pipelines/1/runs/251144/signedlogcontent/12?urlExpires=2023-06-14T17%3A11%3A50.2399742Z&urlSigningMethod=HMACV1&urlSignature=%2FHTlrgaHtF2Jv65vw%2Fj4SzT69etebI0swSSM6dXC0tk%3D

{code}
$ git grep local-cluster sql/core/
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala:    val session = SparkSession.builder().master("local-cluster[3, 1, 1024]").getOrCreate()
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala:    val session = SparkSession.builder().master("local-cluster[3, 1, 1024]").getOrCreate()
sql/core/src/test/scala/org/apache/spark/sql/execution/BroadcastExchangeSuite.scala:// Additional tests run in 'local-cluster' mode.
sql/core/src/test/scala/org/apache/spark/sql/execution/BroadcastExchangeSuite.scala:      .setMaster("local-cluster[2,1,1024]")
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala:      "--master", "local-cluster[1,1,1024]",
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCommitterSuite.scala:   * Create a new [[SparkSession]] running in local-cluster mode with unsafe and codegen enabled.
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCommitterSuite.scala:      .master("local-cluster[2,1,1024]")
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala: * Tests in this suite we need to run Spark in local-cluster mode. In particular, the use of
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala:   * Create a new [[SparkSession]] running in local-cluster mode with unsafe and codegen enabled.
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala:      .master("local-cluster[2,1,512]")
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala:          .config(sparkConf.setMaster("local-cluster[2, 1, 1024]"))
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala:  // Create a new [[SparkSession]] running in local-cluster mode.
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala:      .master("local-cluster[2,1,1024]")
{code}
Bug: Upgrade zstd-jni to 1.5.5-4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jun/23 11:02
Affects Version/s: 3.5.0

New version includes a bug fixed
[Fix bug in ZstdBufferDecompressingStream when array backing ByteBuffer|https://github.com/luben/zstd-jni/commit/355b8511a2967d097a619047a579930cac2ccd9d]
Bug: Assign names to the error class _LEGACY_ERROR_TEMP_[0050|0057|0058|0059]
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jun/23 08:15
Affects Version/s: 3.5.0

_LEGACY_ERROR_TEMP_0050 => LOCAL_MUST_WITH_SCHEMA_FILE

_LEGACY_ERROR_TEMP_0057 => UNSUPPORTED_DEFAULT_VALUE.WITHOUT_SUGGESTION

_LEGACY_ERROR_TEMP_0058 => UNSUPPORTED_DEFAULT_VALUE.WITH_SUGGESTION

_LEGACY_ERROR_TEMP_0059 => REF_DEFAULT_VALUE_IS_NOT_ALLOWED_IN_PARTITION
Bug: Use toSet to deduplicate the iterator data to prevent the creation of large Array
Status: Reopened | Priority: Minor | Resolution: 
Created: 01/Jun/23 08:23
Affects Version/s: 3.4.0

When SubqueryBroadcastExec reuses the keys of Broadcast HashedRelation for dynamic partition pruning, it will put all the keys in an Array, and then call the distinct of the Array to remove the duplicates.

In general, Broadcast HashedRelation may have many rows, and the repetition rate of this key is high. Doing so will cause this Array to occupy a large amount of memory (and this memory is not managed by MemoryManager), which may trigger OOM.
Bug: Use `checkError()` to check `Exception` in `*Insert*Suite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jun/23 11:05
Affects Version/s: 3.5.0

Include:

1.sql/core/src/test/scala/org/apache/spark/sql/SQLInsertTestSuite

2.sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite

3.sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite

4.sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite
Bug: Incorrect SparkException when missing config in resources in Stage-Level Scheduling
Status: Open | Priority: Minor | Resolution: 
Created: 02/Jun/23 07:31
Affects Version/s: 3.4.0

[ResourceUtils.listResourceIds|https://github.com/apache/spark/blob/807abf9c53ee8c1c7ef69646ebd8a266f60d5580/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala#L152-L155] can throw an exception for any missing config, not just `amount`.

{code:scala}
      val index = key.indexOf('.')
      if (index < 0) {
        throw new SparkException(s"You must specify an amount config for resource: $key " +
          s"config: $componentName.$RESOURCE_PREFIX.$key")
      }
{code}
Bug: Incorrect SparkException for Stage-Level Scheduling in local mode
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jun/23 08:55
Affects Version/s: 3.4.0

While in `local[*]` mode, the following `SparkException` is thrown:

```text
org.apache.spark.SparkException: TaskResourceProfiles are only supported for Standalone cluster for now when dynamic allocation is disabled.
  at org.apache.spark.resource.ResourceProfileManager.isSupported(ResourceProfileManager.scala:71)
  at org.apache.spark.resource.ResourceProfileManager.addResourceProfile(ResourceProfileManager.scala:126)
  at org.apache.spark.rdd.RDD.withResources(RDD.scala:1802)
  ... 42 elided
```

This happens for the following snippet:

```scala
val rdd = sc.range(0, 9)

import org.apache.spark.resource.ResourceProfileBuilder
val rpb = new ResourceProfileBuilder
val rp1 = rpb.build()

rdd.withResources(rp1)
```
Bug: Expose amount of shuffle data available on the node
Status: Open | Priority: Trivial | Resolution: 
Created: 27/Jun/23 11:41
Affects Version/s: 3.4.1

[ShuffleMetrics|https://github.com/apache/spark/blob/43f7a86a05ad8c7ec7060607e43d9ca4d0fe4166/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java#L318] doesn't have metrics like 
"totalShuffleDataBytes" and "numAppsWithShuffleData", these metrics are per node published by External Shuffle Service.
 
Adding these metrics would help in - 
1. Deciding if we can decommission the node if no shuffle data present
2. Better live monitoring of customer's workload to see if there is skewed shuffle data present on the node
Bug: Improve Error message for SQL Window functions
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Jun/23 18:44
Affects Version/s: 3.5.0

Right now, if window spec is used with a stream query, the error message looks like following:
Non-time-based windows are not supported on streaming DataFrames/Datasets;
Window [... <Sub Logic Plan>
The message isn't very helpful to identify what's the problem is and some customers and even support engineers got confused by this. It is suggested that we call out aggregation function over the window spec so that the users can locate the part of the query that caused the problem easier.
Bug: Change to use `map` where `unzip` used to extract a single element 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Jun/23 03:09
Affects Version/s: 3.5.0

For example:

 

Seq((1, 11), (2, 22)).unzip._1

 

should change to 

 

Seq((1, 11), (2, 22)).map(_._1)
Bug: Fix Flaky Test: ThriftServerQueryTestSuite. subquery_scalar_subquery_scalar_subquery_select_sql
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 28/Oct/20 15:56
Affects Version/s: 3.1.0

- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130369/testReport/org.apache.spark.sql.hive.thriftserver/ThriftServerQueryTestSuite/subquery_scalar_subquery_scalar_subquery_select_sql/

{code}
[info] - subquery/scalar-subquery/scalar-subquery-select.sql *** FAILED *** (3 seconds, 877 milliseconds)
[info]   Expected "[1]0	2017-05-04 01:01:0...", but got "[]0	2017-05-04 01:01:0..." Result did not match for query #3
[info]   SELECT (SELECT min(t3d) FROM t3) min_t3d,
[info]          (SELECT max(t2h) FROM t2) max_t2h
[info]   FROM   t1
[info]   WHERE  t1a = 'val1c' (ThriftServerQueryTestSuite.scala:197)
{code}
Bug: ANSI mode: runtime errors instead of returning null on invalid inputs
Status: Resolved | Priority: Major | Resolution: Done
Created: 28/Oct/20 17:15
Affects Version/s: 3.1.0

We should respect the ANSI mode in more places. What we have done so far are mostly the overflow check in various operators. This ticket is to track a category of ANSI mode behaviors: operators should throw runtime errors instead of returning null when the input is invalid.
Bug: Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 20:47
Affects Version/s: 3.0.1

Hadoop 3.x+ offers shaded client jars: hadoop-client-api and hadoop-client-runtime, which shade 3rd party dependencies such as Guava, protobuf, jetty etc. This Jira switches Spark to use these jars instead of hadoop-common, hadoop-client etc. Benefits include:
 * It unblocks Spark from upgrading to Hadoop 3.2.2/3.3.0+. The newer versions of Hadoop have migrated to Guava 27.0+ and in order to resolve Guava conflicts, Spark depends on Hadoop to not leaking dependencies.
 * It makes Spark/Hadoop dependency cleaner. Currently Spark uses both client-side and server-side Hadoop APIs from modules such as hadoop-common, hadoop-yarn-server-common etc. Moving to hadoop-client-api allows use to only use public/client API from Hadoop side.
 * Provides a better isolation from Hadoop dependencies. In future Spark can better evolve without worrying about dependencies pulled from Hadoop side (which used to be a lot).

*There are some behavior changes introduced with this JIRA, when people use Spark compiled with Hadoop 3.x:*
- Users now need to make sure class path contains `hadoop-client-api` and `hadoop-client-runtime` jars when they deploy Spark with the `hadoop-provided` option. In addition, it is high recommended that they put these two jars before other Hadoop jars in the class path. Otherwise, conflicts such as from Guava could happen if classes are loaded from the other non-shaded Hadoop jars.
- Since the new shaded Hadoop clients no longer include 3rd party dependencies. Users who used to depend on these now need to explicitly put the jars in their class path.

Ideally the above should go to release notes.
Bug: Support PyArrow 2.0.0+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 06:24
Affects Version/s: 3.1.0

Some tests fail with PyArrow 2.0.0 in PySpark:

{code}
======================================================================
ERROR [0.774s]: test_grouped_over_window_with_key (pyspark.sql.tests.test_pandas_grouped_map.GroupedMapInPandasTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/sql/tests/test_pandas_grouped_map.py", line 595, in test_grouped_over_window_with_key
    .select('id', 'result').collect()
  File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line 588, in collect
    sock_info = self._jdf.collectToPython()
  File "/__w/spark/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/__w/spark/spark/python/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.PythonException: 
  An exception was thrown from the Python worker. Please see the stack trace below.
Traceback (most recent call last):
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 601, in main
    process()
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 593, in process
    serializer.dump_stream(out_iter, outfile)
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 255, in dump_stream
    return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream)
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 81, in dump_stream
    for batch in iterator:
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 248, in init_stream_yield_batches
    for series in iterator:
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 426, in mapper
    return f(keys, vals)
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 170, in <lambda>
    return lambda k, v: [(wrapped(k, v), to_arrow_type(return_type))]
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 158, in wrapped
    result = f(key, pd.concat(value_series, axis=1))
  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/util.py", line 68, in wrapper
    return f(*args, **kwargs)
  File "/__w/spark/spark/python/pyspark/sql/tests/test_pandas_grouped_map.py", line 590, in f
    "{} != {}".format(expected_key[i][1], window_range)
AssertionError: {'start': datetime.datetime(2018, 3, 15, 0, 0), 'end': datetime.datetime(2018, 3, 20, 0, 0)} != {'start': datetime.datetime(2018, 3, 15, 0, 0, tzinfo=<StaticTzInfo 'Etc/UTC'>), 'end': datetime.datetime(2018, 3, 20, 0, 0, tzinfo=<StaticTzInfo 'Etc/UTC'>)}
{code}

We should verify and support PyArrow 2.0.0+

See also https://github.com/apache/spark/runs/1278918780
Bug: Fix PySpark test cases in YarnClusterSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 07:49
Affects Version/s: 3.1.0

See https://github.com/apache/spark/pull/29843#issuecomment-712586049.
The test is broken somewhere, and we should fix it to make the tests pass.

See also https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129928/testReport/
Bug: Using Delegation Tokens to access HBase through Spark (Java)
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 19/Oct/20 15:18
Affects Version/s: 2.3.0

I have a requirement to access a kerberized HBase through delegation tokens instead of keytab. I have generated the token using the token util API and also loaded it back into the UserGroupInformation. However, hasKerberosCredentials returns a false. I am looking for the  the right way to use a saved delegation token to access HBase . The code that I have currently looks like shown below. I see the error :   
{code:java}
client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4850 ms ago, cancelled=false, msg=Connection closed, details=row 'employee,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=myhost.com,16020,1602758016346, seqNum=-1{code}
 
{code:java}
UserGroupInformation ugi;
 ugi = UserGroupInformation.getLoginUser();
 Credentials creds = new Credentials ();
 creds = Credentials.readTokenStorageFile (new Path("file:///zen-volume-home/tokenFile"),conf);
 
 System.out.println("TOKEN *********" + creds.getToken(new org.apache.hadoop.io.Text("hbase")));
 ugi.addToken(creds.getToken(new org.apache.hadoop.io.Text("hbase")));
 ugi.addCredentials(creds);

/* I do see the token getting printed. However, I am looking for information on how to use this ugi which has the token added to it, to access data on HBase.
I have tried doing this :  */

SQLContext sqlC = ugi.getLoginUser().doAs((new PrivilegedExceptionAction<SQLContext>() {
		            public SQLContext run() throws Exception {
		     
		     sparkconf = new SparkConf().setAppName("Hbase With Spark");
		     jsc = new JavaSparkContext(sparkconf);
		     hbaseContext = new JavaHBaseContext(jsc, conf);
		 sqlContext = new org.apache.spark.sql.SQLContext(jsc);		
		        		
		 String sqlMapping  = "name STRING all:name" ;	
		 Dataset<Row> dfEmp = sqlContext.read().format("org.apache.hadoop.hbase.spark")
		        				.option("hbase.columns.mapping", sqlMapping)		        				.option("hbase.table", "employee").load();
		        		        
		        		dfEmp.registerTempTable("empdata");  
		        		dfEmp.show();
		        		return sqlContext;
		            }
		        }));
		        		{code}
Bug: SQL Reference: Run SQL on files directly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/20 07:33
Affects Version/s: 3.0.1

Currently, SQL reference ([https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select.html] ) does not show the feature "Run SQL on files directly", which is documented in [https://spark.apache.org/docs/3.0.1/sql-data-sources-load-save-functions.html#run-sql-on-files-directly] We should add it to SQL reference.
Bug: Allow saving/loading INT96 in parquet w/o rebasing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/20 11:56
Affects Version/s: 3.1.0

Currently, Spark always performs rebasing of INT96 columns in Parquet datasource but this is not required by parquet spec. This tickets aims to allow users to turn off rebasing via SQL config.
Bug: Check propagation of datasource options to underlying file system for built-in file-based datasources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/20 12:26
Affects Version/s: 3.1.0

Add a common trait with a test to check that datasource options are propagated to underlying file systems. Individual tests were already added by SPARK-33094 and SPARK-33089. The ticket aims to de-duplicate the tests.
Bug: spark REST API Unable to get JobDescription
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 16/Oct/20 09:28
Affects Version/s: 2.4.4

spark set job description ,use spark REST API (localhost:4040/api/v1/applications/xxx/jobs)unable to get job description,but it can be displayed at localhost:4040/jobs

spark.sparkContext.setJobDescription({color:#6a8759}"test_count"{color})
spark.range({color:#6897bb}100{color}).count()
Bug: Migration to NumPy documentation style in Core (pyspark.*, pyspark.resource.*, etc.)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 03:08
Affects Version/s: 3.1.0

 This JIRA targets to migrate to NumPy documentation style in Core (pyspark.\*, pyspark.resource.\*, etc.). Please also see the parent JIRA.
Bug: Why does ArrayType schema change between read/write for parquet files?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Oct/20 13:07
Affects Version/s: 3.0.0

I have parquet files that have been produced with org.apache.parquet Java library (not Spark). The schema has a list of authors that's reported by [https://github.com/apache/parquet-mr/tree/master/parquet-tools] like this:

repeated binary authors (STRING);

If I do spark.read.parquet(input_dir).write.parquet(output_dir) and do the same for the output files, the authors column has been changed into:

optional group authors (LIST) {
  repeated group list {
    optional binary element (STRING);
  }
 }

It seems to mean the same thing but from schema perspective this is different.

I have other set of tools that are reading the output from this step (with real logic) but I fail to match the schemas. The original data works fine. Also, df.printSchema() shows the same for both (except for possible nullability change, which we can ignore for this case)

Any thoughts if this is intentional and if I have any control for this from within Spark?