av9ash/GitBugs huggingface dataset:
Bug: Refactor second set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 18:57
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the second set of 20.
{code:java}
inputTypeUnsupportedError
invalidFractionOfSecondError
overflowInSumOfDecimalError
overflowInIntegralDivideError
mapSizeExceedArraySizeWhenZipMapError
copyNullFieldNotAllowedError
literalTypeUnsupportedError
noDefaultForDataTypeError
doGenCodeOfAliasShouldNotBeCalledError
orderedOperationUnsupportedByDataTypeError
regexGroupIndexLessThanZeroError
regexGroupIndexExceedGroupCountError
invalidUrlError
dataTypeOperationUnsupportedError
mergeUnsupportedByWindowFunctionError
dataTypeUnexpectedError
typeUnsupportedError
negativeValueUnexpectedError
addNewFunctionMismatchedWithFunctionError
cannotGenerateCodeForUncomparableTypeError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: RemoveNoopOperators should keep output schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/21 06:55
Affects Version/s: 3.2.0

!image-2021-07-30-17-46-59-196.png|width=539,height=220!

[https://github.com/apache/spark/pull/33587]

 

Only first level?
Bug: Group exception messages in core/scheduler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 18:15
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/scheduler'
|| Filename                ||   Count ||
| DAGScheduler.scala      |       4 |
| LiveListenerBus.scala   |       1 |
| TaskInfo.scala          |       1 |
| TaskSchedulerImpl.scala |       4 |
| TaskSetManager.scala    |       1 |


'core/src/main/scala/org/apache/spark/scheduler/cluster'
|| Filename                            ||   Count ||
| CoarseGrainedSchedulerBackend.scala |       2 |


'core/src/main/scala/org/apache/spark/scheduler/dynalloc'
|| Filename              ||   Count ||
| ExecutorMonitor.scala |       1 |
Bug: Support initial state for flatMapGroupsWithState in batch mode
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Jul/21 05:58
Affects Version/s: 3.2.0

SPARK-35897 added support for an initial state with flatMapGroupsWithState. However this lacked support for initial state in batch mode of flatMapGroupsWIthState. This Jira is for adding support for batch mode.
Bug: Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 06:30
Affects Version/s: 3.2.0

{code:java}
withTempPath { path =>
        spark.sql(
          s"""
             |SELECT
             |NAMED_STRUCT('ID', ID, 'IF(ID=1,ID,0)', IF(ID=1,ID,0), 'B', ABS(ID)) AS col1
             |FROM v
               """.stripMargin).write.mode(SaveMode.Overwrite).parquet(path.getCanonicalPath)
      }
{code}
!screenshot-1.png|width=1302,height=176!
Bug: extractJoinKeysWithColStats support EqualNullSafe
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jul/21 14:24
Affects Version/s: 3.3.0

sql("select * from date_dim join item on d_date_sk = i_item_sk").explain("cost")
{noformat}
== Optimized Logical Plan ==
Join Inner, (d_date_sk#0 = i_item_sk#28), Statistics(sizeInBytes=1.0 B, rowCount=0)
:- Relation default.date_dim[d_date_sk#0,d_date_id#1,d_date#2,d_month_seq#3,d_week_seq#4,d_quarter_seq#5,d_year#6,d_dow#7,d_moy#8,d_dom#9,d_qoy#10,d_fy_year#11,d_fy_quarter_seq#12,d_fy_week_seq#13,d_day_name#14,d_quarter_name#15,d_holiday#16,d_weekend#17,d_following_holiday#18,d_first_dom#19,d_last_dom#20,d_same_day_ly#21,d_same_day_lq#22,d_current_day#23,... 4 more fields] parquet, Statistics(sizeInBytes=17.6 MiB, rowCount=7.30E+4)
+- Relation default.item[i_item_sk#28,i_item_id#29,i_rec_start_date#30,i_rec_end_date#31,i_item_desc#32,i_current_price#33,i_wholesale_cost#34,i_brand_id#35,i_brand#36,i_class_id#37,i_class#38,i_category_id#39,i_category#40,i_manufact_id#41,i_manufact#42,i_size#43,i_formulation#44,i_color#45,i_units#46,i_container#47,i_manager_id#48,i_product_name#49] parquet, Statistics(sizeInBytes=85.2 MiB, rowCount=2.04E+5)
{noformat}

sql("select * from date_dim join item on d_date_sk <=> i_item_sk").explain("cost")
{noformat}
== Optimized Logical Plan ==
Join Inner, (d_date_sk#0 <=> i_item_sk#28), Statistics(sizeInBytes=9.2 TiB, rowCount=1.49E+10)
:- Relation default.date_dim[d_date_sk#0,d_date_id#1,d_date#2,d_month_seq#3,d_week_seq#4,d_quarter_seq#5,d_year#6,d_dow#7,d_moy#8,d_dom#9,d_qoy#10,d_fy_year#11,d_fy_quarter_seq#12,d_fy_week_seq#13,d_day_name#14,d_quarter_name#15,d_holiday#16,d_weekend#17,d_following_holiday#18,d_first_dom#19,d_last_dom#20,d_same_day_ly#21,d_same_day_lq#22,d_current_day#23,... 4 more fields] parquet, Statistics(sizeInBytes=17.6 MiB, rowCount=7.30E+4)
+- Relation default.item[i_item_sk#28,i_item_id#29,i_rec_start_date#30,i_rec_end_date#31,i_item_desc#32,i_current_price#33,i_wholesale_cost#34,i_brand_id#35,i_brand#36,i_class_id#37,i_class#38,i_category_id#39,i_category#40,i_manufact_id#41,i_manufact#42,i_size#43,i_formulation#44,i_color#45,i_units#46,i_container#47,i_manager_id#48,i_product_name#49] parquet, Statistics(sizeInBytes=85.2 MiB, rowCount=2.04E+5)
{noformat}

https://github.com/apache/spark/blob/d6a68e0b67ff7de58073c176dd097070e88ac831/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala#L329-L339
Bug: [CORE] Support getting CPU number in TaskContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 04:10
Affects Version/s: 3.1.2

In stage-level resource scheduling, the allocated 3rd party resources can be obtained in TaskContext using resources() interface, however there is no API to get how many cpus are allocated for the task. Will add a cpus() interface to TaskContext in addition to resources().
Bug: EventLogFileReaders should not complain in case of no event log files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/21 07:47
Affects Version/s: 3.1.2

{code}
21/07/30 07:38:26 WARN FsHistoryProvider: Error while reading new log s3a://.../eventlog_v2_spark-95b5c736c8e44037afcf152534d08771
java.lang.IllegalArgumentException: requirement failed: Log directory must contain at least one event log file!
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.deploy.history.RollingEventLogFilesFileReader.files$lzycompute(EventLogFileReaders.scala:216)
{code}

{code}
$ aws s3 ls s3://.../eventlog_v2_spark-95b5c736c8e44037afcf152534d08771/
2021-06-26 22:31:40          0 appstatus_spark-95b5c736c8e44037afcf152534d08771.inprogress
{code}
Bug: Use "void" as the type name of NullType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 08:09
Affects Version/s: 3.3.0

In PR: [https://github.com/apache/spark/pull/28833,] we support parsing "void" as NullType. But still use "null" as the type name. This leads some confusing and inconsistent issues. For example:

`org.apache.spark.sql.types.DataType.fromDDL(NullType.toDDL)` is not working
Bug: Support TimestampNTZ in Avro data source 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 06:10
Affects Version/s: 3.3.0

As per the Avro spec https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28microsecond+precision%29, Spark can convert TimestampNTZ type from/to Avro's Local timestamp type.
Bug: Refactor seventeenth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:32
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the seventeenth set of 20.
{code:java}
legacyCheckpointDirectoryExistsError
subprocessExitedError
outputDataTypeUnsupportedByNodeWithoutSerdeError
invalidStartIndexError
concurrentModificationOnExternalAppendOnlyUnsafeRowArrayError
doExecuteBroadcastNotImplementedError
databaseNameConflictWithSystemPreservedDatabaseError
commentOnTableUnsupportedError
unsupportedUpdateColumnNullabilityError
renameColumnUnsupportedForOlderMySQLError
failedToExecuteQueryError
nestedFieldUnsupportedError
transformationsAndActionsNotInvokedByDriverError
repeatedPivotsUnsupportedError
pivotNotAfterGroupByUnsupportedError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor sixteenth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:32
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the sixteenth set of 20.
{code:java}
cannotDropMultiPartitionsOnNonatomicPartitionTableError
truncateMultiPartitionUnsupportedError
overwriteTableByUnsupportedExpressionError
dynamicPartitionOverwriteUnsupportedByTableError
failedMergingSchemaError
cannotBroadcastTableOverMaxTableRowsError
cannotBroadcastTableOverMaxTableBytesError
notEnoughMemoryToBuildAndBroadcastTableError
executeCodePathUnsupportedError
cannotMergeClassWithOtherClassError
continuousProcessingUnsupportedByDataSourceError
failedToReadDataError
failedToGenerateEpochMarkerError
foreachWriterAbortedDueToTaskFailureError
integerOverflowError
failedToReadDeltaFileError
failedToReadSnapshotFileError
cannotPurgeAsBreakInternalStateError
cleanUpSourceFilesUnsupportedError
latestOffsetNotCalledError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Rename classes in shuffle RPC used for block push operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 00:12
Affects Version/s: 3.1.0

In the current implementation of push-based shuffle, we are reusing certain code between both block fetch and block push.

This is generally good except that certain classes that are meant to be used for both block fetch and block push now have names that indicate they are only for block fetches, which is confusing.

This ticket renames these classes to be more generic to be reused across both block fetch and block push.
Bug: Refactor twelfth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:28
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the twelfth set of 20.
{code:java}
cannotRewriteDomainJoinWithConditionsError
decorrelateInnerQueryThroughPlanUnsupportedError
methodCalledInAnalyzerNotAllowedError
cannotSafelyMergeSerdePropertiesError
pairUnsupportedAtFunctionError
onceStrategyIdempotenceIsBrokenForBatchError[TreeType
structuralIntegrityOfInputPlanIsBrokenInClassError
structuralIntegrityIsBrokenAfterApplyingRuleError
ruleIdNotFoundForRuleError
cannotCreateArrayWithElementsExceedLimitError
indexOutOfBoundsOfArrayDataError
malformedRecordsDetectedInRecordParsingError
remoteOperationsUnsupportedError
invalidKerberosConfigForHiveServer2Error
parentSparkUIToAttachTabNotFoundError
inferSchemaUnsupportedForHiveError
requestedPartitionsMismatchTablePartitionsError
dynamicPartitionKeyNotAmongWrittenPartitionPathsError
cannotRemovePartitionDirError
cannotCreateStagingDirError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor tenth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:26
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the tenth set of 20.
{code:java}
registeringStreamingQueryListenerError
concurrentQueryInstanceError
cannotParseJsonArraysAsStructsError
cannotParseStringAsDataTypeError
failToParseEmptyStringForDataTypeError
failToParseValueForDataTypeError
rootConverterReturnNullError
cannotHaveCircularReferencesInBeanClassError
cannotHaveCircularReferencesInClassError
cannotUseInvalidJavaIdentifierAsFieldNameError
cannotFindEncoderForTypeError
attributesForTypeUnsupportedError
schemaForTypeUnsupportedError
cannotFindConstructorForTypeError
paramExceedOneCharError
paramIsNotIntegerError
paramIsNotBooleanValueError
foundNullValueForNotNullableFieldError
malformedCSVRecordError
elementsOfTupleExceedLimitError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor ninth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:25
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the ninth set of 20.
{code:java}
unscaledValueTooLargeForPrecisionError
decimalPrecisionExceedsMaxPrecisionError
outOfDecimalTypeRangeError
unsupportedArrayTypeError
unsupportedJavaTypeError
failedParsingStructTypeError
failedMergingFieldsError
cannotMergeDecimalTypesWithIncompatiblePrecisionAndScaleError
cannotMergeDecimalTypesWithIncompatiblePrecisionError
cannotMergeDecimalTypesWithIncompatibleScaleError
cannotMergeIncompatibleDataTypesError
exceedMapSizeLimitError
duplicateMapKeyFoundError
mapDataKeyArrayLengthDiffersFromValueArrayLengthError
fieldDiffersFromDerivedLocalDateError
failToParseDateTimeInNewParserError
failToFormatDateTimeInNewFormatterError
failToRecognizePatternAfterUpgradeError
failToRecognizePatternError
cannotCastUTF8StringToDataTypeError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: date_trunc returns incorrect output
Status: Open | Priority: Major | Resolution: 
Created: 09/Jul/21 03:33
Affects Version/s: 3.2.0

Hi,

Running date_trunc on any hour of "1891-10-01" returns incorrect output for "Europe/Bratislava" timezone.
Use the following steps in order to reproduce the issue:
 * Run spark-shell using:

{code:java}
TZ="Europe/Bratislava" ./bin/spark-shell --conf spark.driver.extraJavaOptions='-Duser.timezone=Europe/Bratislava' --conf spark.executor.extraJavaOptions='-Duser.timezone=Europe/Bratislava' --conf spark.sql.session.timeZone="Europe/Bratislava"{code}

 * Generate test data:

{code:java}
((0 until 9).map(hour => s"1891-10-01 00:0$hour:00") ++ (10 until 24).map(hour => s"1891-10-01 00:$hour:00")).toDF("ts_string").createOrReplaceTempView("temp_ts")
{code}

 * Run query:

{code:java}
sql("select ts_string, cast(ts_string as TIMESTAMP) as ts, date_trunc('day', ts_string) from temp_ts").show(false)
{code}

 * Output:

{code:java}
+-------------------+-------------------+--------------------------+
|ts_string          |ts                 |date_trunc(day, ts_string)|
+-------------------+-------------------+--------------------------+
|1891-10-01 00:00:00|1891-10-01 00:02:16|1891-10-01 00:02:16       |
|1891-10-01 00:01:00|1891-10-01 00:03:16|1891-10-01 00:02:16       |
|1891-10-01 00:02:00|1891-10-01 00:04:16|1891-10-01 00:02:16       |
|1891-10-01 00:03:00|1891-10-01 00:03:00|1891-10-01 00:02:16       |
|1891-10-01 00:04:00|1891-10-01 00:04:00|1891-10-01 00:02:16       |
|1891-10-01 00:05:00|1891-10-01 00:05:00|1891-10-01 00:02:16       |
|1891-10-01 00:06:00|1891-10-01 00:06:00|1891-10-01 00:02:16       |
|1891-10-01 00:07:00|1891-10-01 00:07:00|1891-10-01 00:02:16       |
|1891-10-01 00:08:00|1891-10-01 00:08:00|1891-10-01 00:02:16       |
|1891-10-01 00:10:00|1891-10-01 00:10:00|1891-10-01 00:02:16       |
|1891-10-01 00:11:00|1891-10-01 00:11:00|1891-10-01 00:02:16       |
|1891-10-01 00:12:00|1891-10-01 00:12:00|1891-10-01 00:02:16       |
|1891-10-01 00:13:00|1891-10-01 00:13:00|1891-10-01 00:02:16       |
|1891-10-01 00:14:00|1891-10-01 00:14:00|1891-10-01 00:02:16       |
|1891-10-01 00:15:00|1891-10-01 00:15:00|1891-10-01 00:02:16       |
|1891-10-01 00:16:00|1891-10-01 00:16:00|1891-10-01 00:02:16       |
|1891-10-01 00:17:00|1891-10-01 00:17:00|1891-10-01 00:02:16       |
|1891-10-01 00:18:00|1891-10-01 00:18:00|1891-10-01 00:02:16       |
|1891-10-01 00:19:00|1891-10-01 00:19:00|1891-10-01 00:02:16       |
|1891-10-01 00:20:00|1891-10-01 00:20:00|1891-10-01 00:02:16       |
+-------------------+-------------------+--------------------------+
only showing top 20 rows
{code}
Bug: (RemoveRedundantAliases should keep output schema name) The result incorrect if the partition path case is inconsistent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 14:10
Affects Version/s: 3.2.0

Please reproduce this issue using HDFS. Local HDFS can not reproduce this issue.
{code:scala}
sql("create table t1(cal_dt date) using parquet")
sql("insert into t1 values (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
sql("create view t1_v as select * from t1")
sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'")
sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
sql("SELECT * FROM t2 ").show
{code}


{noformat}
// It should not empty.
scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
+----+------+
|FLAG|CAL_DT|
+----+------+
+----+------+


scala> sql("SELECT * FROM t2 ").show
+----+----------+
|FLAG|    CAL_DT|
+----+----------+
|   1|2021-06-27|
|   1|2021-06-28|
+----+----------+

scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
+----+----------+
|FLAG|    CAL_DT|
+----+----------+
|   2|2021-06-29|
|   2|2021-06-30|
+----+----------+
{noformat}
Bug: PruneFileSourcePartitionsSuite tests are failing after the fix to SPARK-36136
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Jul/21 17:13
Affects Version/s: 3.2.0

After the fix to [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package, couple of tests in PruneFileSourcePartitionsSuite are failing now.
Bug: Spark submit REST cluster/standalone mode - launching an s3a jar with STS
Status: Reopened | Priority: Major | Resolution: 
Created: 01/Jul/21 14:25
Affects Version/s: 3.1.2

{code:java}
/var/lib/spark-2.4.8-bin-hadoop2.7/bin/spark-submit --master spark://myhost:6066 --conf spark.hadoop.fs.s3a.access.key='redact1' --conf spark.executorEnv.AWS_ACCESS_KEY_ID='redact1' --conf spark.driverEnv.AWS_ACCESS_KEY_ID='redact1' --conf spark.hadoop.fs.s3a.secret.key='redact2' --conf spark.executorEnv.AWS_SECRET_ACCESS_KEY='redact2' --conf spark.driverEnv.AWS_SECRET_ACCESS_KEY='redact2' --conf spark.hadoop.fs.s3a.session.token='redact3' --conf spark.executorEnv.AWS_SESSION_TOKEN='redact3' --conf spark.driverEnv.AWS_SESSION_TOKEN='redact3' --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider --conf spark.driver.extraJavaOptions='-DAWS_ACCESS_KEY_ID=redact1 -DAWS_SECRET_ACCESS_KEY=redact2 -DAWS_SESSION_TOKEN=redact3' --conf spark.executor.extraJavaOptions='-DAWS_ACCESS_KEY_ID=redact1 -DAWS_SECRET_ACCESS_KEY=redact2 -DAWS_SESSION_TOKEN=redact3' --total-executor-cores 4 --executor-cores 2 --executor-memory 2g --driver-memory 1g --name lin1 --deploy-mode cluster --conf spark.eventLog.enabled=false --class com.yotpo.metorikku.Metorikku s3a://mybuc/metorikku_2.11.jar -c s3a://mybuc/spark_ingestion_job.yaml
{code}
running the above command give below stack trace:

 
{code:java}
 Exception from the cluster:\njava.nio.file.AccessDeniedException: s3a://mybuc/metorikku_2.11.jar: getFileStatus on s3a://mybuc/metorikku_2.11.jar: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: xx; S3 Extended Request ID: /1qj/yy=), S3 Extended Request ID: /1qj/yy=\n\
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:101)
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1542)
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1463)
org.apache.hadoop.fs.s3a.S3AFileSystem.isFile(S3AFileSystem.java:2030)
org.apache.spark.util.Utils$.fetchHcfsFile(Utils.scala:747)
org.apache.spark.util.Utils$.doFetchFile(Utils.scala:723)
org.apache.spark.util.Utils$.fetchFile(Utils.scala:509)
org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)
org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)
org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92){code}
all the ec2s in the spark cluster only have access to s3 via STS tokens. The jar itself reads csvs from s3 using the tokens, and everything works if either 1. i change the commandline to point to local jars on the ec2 OR 2. use port 7077/client mode instead of cluster mode. But it seems the jar itself can't be launched off s3, as if the tokens are not being picked up properly.
Bug: Refactor eighth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:24
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the eighth set of 20.
{code:java}
executeBroadcastTimeoutError
cannotCompareCostWithTargetCostError
unsupportedDataTypeError
notSupportTypeError
notSupportNonPrimitiveTypeError
unsupportedTypeError
useDictionaryEncodingWhenDictionaryOverflowError
endOfIteratorError
cannotAllocateMemoryToGrowBytesToBytesMapError
cannotAcquireMemoryToBuildLongHashedRelationError
cannotAcquireMemoryToBuildUnsafeHashedRelationError
rowLargerThan256MUnsupportedError
cannotBuildHashedRelationWithUniqueKeysExceededError
cannotBuildHashedRelationLargerThan8GError
failedToPushRowIntoRowQueueError
unexpectedWindowFunctionFrameError
cannotParseStatisticAsPercentileError
statisticNotRecognizedError
unknownColumnError
unexpectedAccumulableUpdateValueError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Upgrade the RocksDB version to 6.20.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 11:39
Affects Version/s: 3.2.0

As the discussion in [https://github.com/apache/spark/pull/32928/files#r654049392,] after confirming the compatibility, we can use a newer RocksDB version for the state store implementation.