av9ash/GitBugs huggingface dataset:
Bug: Support TimestampNTZ type in Parquet file source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 14:58
Affects Version/s: 3.3.0

As per https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp, Parquet supports both TIMESTAMP_NTZ and TIMESTAMP_LTZ (Spark's current default timestamp type):
* A TIMESTAMP with isAdjustedToUTC=true => TIMESTAMP_LTZ
* A TIMESTAMP with isAdjustedToUTC=false => TIMESTAMP_NTZ

In Spark 3.1 or prior,  the Parquet writer follows the definition and sets the field `isAdjustedToUTC` as `true`, while the Parquet reader doesn’t respect the `isAdjustedToUTC` flag and convert any Parquet Timestamp type as TIMESTAMP_LTZ.

Since 3.2, with the support of timestamp without time zone type:
* Parquet writer follows the definition and sets the field `isAdjustedToUTC` as `false` on writing TIMESTAMP_NTZ. 

* Parquet reader 
** For schema inference, Spark converts the Parquet timestamp type to the corresponding catalyst timestamp type according to the timestamp annotation flag `isAdjustedToUTC`.
** If merge schema is enabled in schema inference and some of the files are inferred as TIMESTAMP_NTZ while the others are TIMESTAMP_LTZ, the result type is TIMESTAMP_LTZ  which is considered as the “wider” type
** If a column of a user-provided schema is TIMESTAMP_LTZ and the column was written as  TIMESTAMP_NTZ type, Spark allows the read operation.
** If a column of a user-provided schema is TIMESTAMP_NTZ and the column was written as  TIMESTAMP_LTZ type, the read operation is not allowed since the TIMESTAMP_NTZ is considered as narrower than TIMESTAMP_LTZ.
Bug: Refactor fifteenth set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:31
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the fifteenth set of 20.
{code:java}
unsupportedOperationExceptionError
nullLiteralsCannotBeCastedError
notUserDefinedTypeError
cannotLoadUserDefinedTypeError
timeZoneIdNotSpecifiedForTimestampTypeError
notPublicClassError
primitiveTypesNotSupportedError
fieldIndexOnRowWithoutSchemaError
valueIsNullError
onlySupportDataSourcesProvidingFileFormatError
failToSetOriginalPermissionBackError
failToSetOriginalACLBackError
multiFailuresInStageMaterializationError
unrecognizedCompressionSchemaTypeIDError
getParentLoggerNotImplementedError
cannotCreateParquetConverterForTypeError
cannotCreateParquetConverterForDecimalTypeError
cannotCreateParquetConverterForDataTypeError
cannotAddMultiPartitionsOnNonatomicPartitionTableError
userSpecifiedSchemaUnsupportedByDataSourceError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor eleventh set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:26
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the eleventh set of 20.
{code:java}
expressionDecodingError
expressionEncodingError
classHasUnexpectedSerializerError
cannotGetOuterPointerForInnerClassError
userDefinedTypeNotAnnotatedAndRegisteredError
invalidInputSyntaxForBooleanError
unsupportedOperandTypeForSizeFunctionError
unexpectedValueForStartInFunctionError
unexpectedValueForLengthInFunctionError
sqlArrayIndexNotStartAtOneError
concatArraysWithElementsExceedLimitError
flattenArraysWithElementsExceedLimitError
createArrayWithElementsExceedLimitError
unionArrayWithElementsExceedLimitError
initialTypeNotTargetDataTypeError
initialTypeNotTargetDataTypesError
cannotConvertColumnToJSONError
malformedRecordsDetectedInSchemaInferenceError
malformedJSONError
malformedRecordsDetectedInSchemaInferenceError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Group exception messages in core/resource
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 18:14
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/resource'
|| Filename                            ||   Count ||
| ResourceAllocator.scala             |       4 |
| ResourceDiscoveryScriptPlugin.scala |       3 |
| ResourceInformation.scala           |       2 |
| ResourceProfile.scala               |       4 |
| ResourceProfileManager.scala        |       2 |
| ResourceUtils.scala                 |      11 |
Bug: Group exception messages in core/status
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 18:18
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/status'
|| Filename             ||   Count ||
| AppStatusStore.scala |       3 |
| KVUtils.scala        |       1 |
| LiveEntity.scala     |       2 |


'core/src/main/scala/org/apache/spark/status/api/v1'
|| Filename                     ||   Count ||
| ApiRootResource.scala        |       4 |
| OneApplicationResource.scala |      11 |
| SimpleDateParam.scala        |       1 |
| StagesResource.scala         |       4 |
Bug: Refactor fourth set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:12
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the fourth set of 20.
{code:java}
unableToCreateDatabaseAsFailedToCreateDirectoryError
unableToDropDatabaseAsFailedToDeleteDirectoryError
unableToCreateTableAsFailedToCreateDirectoryError
unableToDeletePartitionPathError
unableToDropTableAsFailedToDeleteDirectoryError
unableToRenameTableAsFailedToRenameDirectoryError
unableToCreatePartitionPathError
unableToRenamePartitionPathError
methodNotImplementedError
tableStatsNotSpecifiedError
unaryMinusCauseOverflowError
binaryArithmeticCauseOverflowError
failedSplitSubExpressionMsg
failedSplitSubExpressionError
failedToCompileMsg
internalCompilerError
compilerError
unsupportedTableChangeError
notADatasourceRDDPartitionError
dataPathNotSpecifiedError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Group exception messages in core/deploy
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 18:20
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/deploy'
|| Filename                      ||   Count ||
| FaultToleranceTest.scala      |       1 |
| PythonRunner.scala            |       1 |
| RRunner.scala                 |       2 |
| SparkHadoopUtil.scala         |       2 |
| SparkSubmit.scala             |       7 |
| SparkSubmitArguments.scala    |       3 |
| StandaloneResourceUtils.scala |       1 |


'core/src/main/scala/org/apache/spark/deploy/history'
|| Filename                         ||   Count ||
| ApplicationCache.scala           |       2 |
| EventLogFileWriters.scala        |       2 |
| FsHistoryProvider.scala          |       5 |
| HistoryServer.scala              |       2 |
| HistoryServerMemoryManager.scala |       1 |


'core/src/main/scala/org/apache/spark/deploy/master'
|| Filename     ||   Count ||
| Master.scala |       2 |


'core/src/main/scala/org/apache/spark/deploy/rest'
|| Filename                        ||   Count ||
| RestSubmissionClient.scala      |      11 |
| StandaloneRestServer.scala      |       2 |
| SubmitRestProtocolMessage.scala |       5 |
| SubmitRestProtocolRequest.scala |       1 |


'core/src/main/scala/org/apache/spark/deploy/security'
|| Filename                              ||   Count ||
| HadoopFSDelegationTokenProvider.scala |       1 |


'core/src/main/scala/org/apache/spark/deploy/worker'
|| Filename           ||   Count ||
| DriverRunner.scala |       2 |
| Worker.scala       |       3 |


'core/src/main/scala/org/apache/spark/deploy/worker/ui'
|| Filename      ||   Count ||
| LogPage.scala |       2 |
Bug: Group exception messages in core/util
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 18:17
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/util'
|| Filename                 ||   Count ||
| AccumulatorV2.scala      |       4 |
| ClosureCleaner.scala     |       1 |
| DependencyUtils.scala    |       1 |
| KeyLock.scala            |       1 |
| ListenerBus.scala        |       1 |
| NextIterator.scala       |       1 |
| SerializableBuffer.scala |       2 |
| ThreadUtils.scala        |       4 |
| Utils.scala              |      16 |


'core/src/main/scala/org/apache/spark/util/collection'
|| Filename              ||   Count ||
| AppendOnlyMap.scala   |       1 |
| CompactBuffer.scala   |       1 |
| ImmutableBitSet.scala |       6 |
| MedianHeap.scala      |       1 |
| OpenHashSet.scala     |       2 |


'core/src/main/scala/org/apache/spark/util/io'
|| Filename                ||   Count ||
| ChunkedByteBuffer.scala |       1 |


'core/src/main/scala/org/apache/spark/util/logging'
|| Filename           ||   Count ||
| DriverLogger.scala |       1 |


'core/src/main/scala/org/apache/spark/util/random'
|| Filename            ||   Count ||
| RandomSampler.scala |       1 |
Bug: Refactor third set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:07
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the third set of 20.
{code:java}
cannotGenerateCodeForUnsupportedTypeError
cannotInterpolateClassIntoCodeBlockError
customCollectionClsNotResolvedError
classUnsupportedByMapObjectsError
nullAsMapKeyNotAllowedError
methodNotDeclaredError
constructorNotFoundError
primaryConstructorNotFoundError
unsupportedNaturalJoinTypeError
notExpectedUnresolvedEncoderError
unsupportedEncoderError
notOverrideExpectedMethodsError
failToConvertValueToJsonError
unexpectedOperatorInCorrelatedSubquery
unreachableError
unsupportedRoundingMode
resolveCannotHandleNestedSchema
inputExternalRowCannotBeNullError
fieldCannotBeNullMsg
fieldCannotBeNullError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor seventh set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:23
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the seventh set of 20.
{code:java}
missingJdbcTableNameAndQueryError
emptyOptionError
invalidJdbcTxnIsolationLevelError
cannotGetJdbcTypeError
unrecognizedSqlTypeError
unsupportedJdbcTypeError
unsupportedArrayElementTypeBasedOnBinaryError
nestedArraysUnsupportedError
cannotTranslateNonNullValueForFieldError
invalidJdbcNumPartitionsError
transactionUnsupportedByJdbcServerError
dataTypeUnsupportedYetError
unsupportedOperationForDataTypeError
inputFilterNotFullyConvertibleError
cannotReadFooterForFileError
cannotReadFooterForFileError
foundDuplicateFieldInCaseInsensitiveModeError
failedToMergeIncompatibleSchemasError
ddlUnsupportedTemporarilyError
operatingOnCanonicalizationPlanError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor thirteenth set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:29
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the thirteenth set of 20.
{code:java}
serDeInterfaceNotFoundError
convertHiveTableToCatalogTableError
cannotRecognizeHiveTypeError
getTablesByTypeUnsupportedByHiveVersionError
dropTableWithPurgeUnsupportedError
alterTableWithDropPartitionAndPurgeUnsupportedError
invalidPartitionFilterError
getPartitionMetadataByFilterError
unsupportedHiveMetastoreVersionError
loadHiveClientCausesNoClassDefFoundError
cannotFetchTablesOfDatabaseError
illegalLocationClauseForViewPartitionError
renamePathAsExistsPathError
renameAsExistsPathError
renameSrcPathNotFoundError
failedRenameTempFileError
legacyMetadataPathExistsError
partitionColumnNotFoundInSchemaError
stateNotDefinedOrAlreadyRemovedError
cannotSetTimeoutDurationError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Refactor fifth set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:15
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the fifth set of 20.
{code:java}
createStreamingSourceNotSpecifySchemaError
streamedOperatorUnsupportedByDataSourceError
multiplePathsSpecifiedError
failedToFindDataSourceError
removedClassInSpark2Error
incompatibleDataSourceRegisterError
unrecognizedFileFormatError
sparkUpgradeInReadingDatesError
sparkUpgradeInWritingDatesError
buildReaderUnsupportedForFileFormatError
jobAbortedError
taskFailedWhileWritingRowsError
readCurrentFileNotFoundError
unsupportedSaveModeError
cannotClearOutputDirectoryError
cannotClearPartitionDirectoryError
failedToCastValueToDataTypeForPartitionColumnError
endOfStreamError
fallbackV1RelationReportsInconsistentSchemaError
cannotDropNonemptyNamespaceError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Group exception messages in core/api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 18:19
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/api/java'
|| Filename        ||   Count ||
| JavaUtils.scala |       1 |


'core/src/main/scala/org/apache/spark/api/python'
|| Filename                                ||   Count ||
| Py4JServer.scala                        |       3 |
| PythonHadoopUtil.scala                  |       1 |
| PythonRDD.scala                         |       3 |
| PythonRunner.scala                      |       4 |
| PythonWorkerFactory.scala               |       4 |
| SerDeUtil.scala                         |       1 |
| WriteInputFormatTestDataGenerator.scala |       2 |


'core/src/main/scala/org/apache/spark/api/r'
|| Filename               ||   Count ||
| BaseRRunner.scala      |       1 |
| JVMObjectTracker.scala |       1 |
| RBackendHandler.scala  |       2 |
| RUtils.scala           |       1 |
| SerDe.scala            |       4 |
Bug: Hadoop config map not mount
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Jul/21 04:36
Affects Version/s: 3.1.2

When i spec --conf spark.kubernetes.hadoop.configMapName in submit args, i see driver pod success mount this config, but exectuor not, **I looked at the source code and found that it was caused by no hadoop conf steps added.
Bug: Delete appName from StreamingSource 
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Jul/21 13:03
Affects Version/s: 3.1.2

The StreamingSource includes the appName in its sourceName. However, the appName should not be handled by the StreamingSource. It is already handled by the MetricsSystem. See all other MetricSources, e.g. ExecutorMetricsSource.

Why is this important? See this part from the [documentation|https://spark.apache.org/docs/latest/monitoring.html#metrics]:

??Often times, users want to be able to track the metrics across apps for driver and executors, which is hard to do with application ID (i.e. {{spark.app.id}}) since it changes with every invocation of the app. For such use cases, a custom namespace can be specified for metrics reporting using {{spark.metrics.namespace}} configuration property. If, say, users wanted to set the metrics namespace to the name of the application, they can set the {{spark.metrics.namespace}} property to a value like {{$\{spark.app.name}}}. This value is then expanded appropriately by Spark and is used as the root namespace of the metrics system.??

This is only possible if the MetricsSystem handles the namespace which it does. But the StreamingSource additionally adds the appName in its sourceName, thus there is no way to configure a namespace that does not include the appName.
Bug: Add a new conf to allow K8s API server-side cache for pod listing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/21 23:31
Affects Version/s: 3.3.0

- https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter

- resourceVersion unset
{code}
Most Recent: Return data at the most recent resource version. The returned data must be consistent (i.e. served from etcd via a quorum read).
{code}

- resourceVersion="0"
{code}
Any: Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served.
{code}
Bug: aggsBuffer should collect AggregateExpression in the map range
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 04:51
Affects Version/s: 2.4.8

show demo for this ISSUE:
{code:java}
// SQL without error

SELECT name, count(name) c
FROM VALUES ('Alice'), ('Bob') people(name)
GROUP BY name GROUPING SETS(name);

// An error is reported after exchanging the order of the query columns:

SELECT count(name) c, name
FROM VALUES ('Alice'), ('Bob') people(name)
GROUP BY name GROUPING SETS(name);

{code}
The error message is：
{code:java}
Error in query: expression 'people.`name`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;
Aggregate [name#5, spark_grouping_id#3], [count(name#1) AS c#0L, name#1]
+- Expand [List(name#1, name#4, 0)], [name#1, name#5, spark_grouping_id#3]
   +- Project [name#1, name#1 AS name#4]
      +- SubqueryAlias `people`
         +- LocalRelation [name#1]

{code}
So far, I have checked that there is no problem before version 2.3.

 

During debugging, I found that the behavior of constructAggregateExprs in ResolveGroupingAnalytics has changed.
{code:java}
    /*
     * Construct new aggregate expressions by replacing grouping functions.
     */
    private def constructAggregateExprs(
        groupByExprs: Seq[Expression],
        aggregations: Seq[NamedExpression],
        groupByAliases: Seq[Alias],
        groupingAttrs: Seq[Expression],
        gid: Attribute): Seq[NamedExpression] = aggregations.map {
      // collect all the found AggregateExpression, so we can check an expression is part of
      // any AggregateExpression or not.
      val aggsBuffer = ArrayBuffer[Expression]()
      // Returns whether the expression belongs to any expressions in `aggsBuffer` or not.
      def isPartOfAggregation(e: Expression): Boolean = {
        aggsBuffer.exists(a => a.find(_ eq e).isDefined)
      }
      replaceGroupingFunc(_, groupByExprs, gid).transformDown {
        // AggregateExpression should be computed on the unmodified value of its argument
        // expressions, so we should not replace any references to grouping expression
        // inside it.
        case e: AggregateExpression =>
          aggsBuffer += e
          e
        case e if isPartOfAggregation(e) => e
        case e =>
          // Replace expression by expand output attribute.
          val index = groupByAliases.indexWhere(_.child.semanticEquals(e))
          if (index == -1) {
            e
          } else {
            groupingAttrs(index)
          }
      }.asInstanceOf[NamedExpression]
    }

{code}
When performing aggregations.map, the aggsBuffer here seems to be outside the scope of the map. It can store the AggregateExpression of all the elements processed by the map function, but this is not before 2.3.
Bug: Refactor fourteenth set of 20 query execution errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 19:31
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the fourteenth set of 20.
{code:java}
cannotGetEventTimeWatermarkError
cannotSetTimeoutTimestampError
batchMetadataFileNotFoundError
multiStreamingQueriesUsingPathConcurrentlyError
addFilesWithAbsolutePathUnsupportedError
microBatchUnsupportedByDataSourceError
cannotExecuteStreamingRelationExecError
invalidStreamingOutputModeError
catalogPluginClassNotFoundError
catalogPluginClassNotImplementedError
catalogPluginClassNotFoundForCatalogError
catalogFailToFindPublicNoArgConstructorError
catalogFailToCallPublicNoArgConstructorError
cannotInstantiateAbstractCatalogPluginClassError
failedToInstantiateConstructorForCatalogError
noSuchElementExceptionError
noSuchElementExceptionError
cannotMutateReadOnlySQLConfError
cannotCloneOrCopyReadOnlySQLConfError
cannotGetSQLConfInSchedulerEventLoopThreadError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Remove TimestampNTZ type support in Spark 3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 16:20
Affects Version/s: 3.3.0

As of now, there are some blockers for delivering the TimestampNTZ project in Spark 3.2:
# In the Hive Thrift server, both TimestampType and TimestampNTZType are mapped to the same timestamp type, which can cause confusion for users. 
# For the Parquet data source, the new written TimestampNTZType Parquet columns will be read as TimestampType in old Spark releases. Also, we need to decide the merge schema for files mixed with TimestampType and TimestampNTZ type.
# The type coercion rules for TimestampNTZType are incomplete. For example, what should the data type of the in clause "IN(Timestamp'2020-01-01 00:00:00', TimestampNtz'2020-01-01 00:00:00') be.
# It is tricky to support TimestampNTZType in JSON/CSV data readers. We need to avoid regressions as possible as we can.

There are 10 days left for the expected 3.2 RC date. So, I propose to release the TimestampNTZ type in Spark 3.3 instead of Spark 3.2. So that we have enough time to make considerate designs for the issues.