Repository: apache/hudi
Pull request #12888: [HUDI-9030] Compatibility fixes for table version 6 with 1.0
Author: login: lokeshj1703
Labels: name: size:L
description: PR with lines of changes in (300, 1000]
name: release-1.0.2
Created: 2025-02-27T11:32:11Z | Merged: 2025-03-05T11:45:40Z

### Change Logs

The PR fixes the compatibility issues related to table version 6 with 1.0 binary. Below are the issues which were found by running the tests below with table version 6:

```
spark-client module tests:
TestHoodieSparkCopyOnWriteTableArchiveWithReplace
TestHoodieSparkCopyOnWriteTableRollback
TestHoodieSparkMergeOnReadTableCompaction
TestHoodieSparkMergeOnReadTableIncrementalRead
TestHoodieSparkMergeOnReadTableInsertUpdateDelete
TestHoodieSparkMergeOnReadTableRollback
TestHoodieSparkRollback
TestHoodieMergeOnReadTable
TestCleanerInsertAndCleanByCommits

spark scala tests:
TestRecordLevelIndex
TestHoodieSparkSqlWriter
TestCOWDataSource
TestCOWDataSourceStorage
TestMORDataSource
TestMORDataSourceStorage
TestMORDataSourceWithBucketIndex
TestSparkDataSource
TestSparkSqlCoreFlow
TestStreamingSource
TestStructuredStreaming
TestTimeTravelQuery
```

Issues
1.  For table version 6, AbstractHoodieLogRecordScanner ignores the log blocks which belong to inflight instants while scanning. PR fixes the logic so that such log blocks are not ignored. This is required for updating RLI which reads the deleted records from data table
2. hoodie.file.group.reader.enabled needs to be disabled in tbl version 6
3. The new rollback logic filters log files using the deltacommit timestamp and then marks them for deletion. This does not work for tbl version 6 since log files do not have deltacommit timestamp in the name. Therefore older rollback logic was brought back here.
4. PR removes the validation while scheduling compaction which validates that compaction instant should be greater than all completed deltacommit instants. This validation was added for table version 6 but is not really required.
5. KEY_GENERATOR_CLASS_NAME and KEY_GENERATOR_TYPE are new configs which are required in tbl version 6 as well. PR makes a change so that these configs are not ignored.
6. MarkerBasedRollbackStrategy#createRollbackRequestForCreateAndMerge removes validation that log file should not have IOType as CREATE in table version 6. CREATE is still used for log files with tbl version 6 in LogFileCreationCallback#preFileCreation
7. hoodie.datasource.read.incr.fallback.fulltablescan.enable needs to be disabled for tbl version 6

### Impact

Adds changes to fix issues related to compatibility with table version 6

### Risk level (write none, low medium or high below)

low

### Documentation Update

NA

### Contributor's checklist

- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed



Diff:
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java
index bca2dfe26c5d3..1dc9a3742ab0b 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java
@@ -146,7 +146,7 @@ public HoodieData<WriteStatus> compact(
 
     boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()   // the engine needs to support fg reader first
         && !metaClient.isMetadataTable()
-        && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
+        && HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), config)
         && operationType == WriteOperationType.COMPACT
         && !hasBootstrapFile(operations)                                            // bootstrap file read for fg reader is not ready
         && config.populateMetaFields();                                             // Virtual key support by fg reader is not ready
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
index 323be08b4ef96..21632118bed05 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
@@ -47,12 +47,9 @@
 import javax.annotation.Nullable;
 
 import java.io.IOException;
-import java.util.List;
 import java.util.Map;
-import java.util.stream.Collectors;
 
 import static org.apache.hudi.common.table.timeline.InstantComparison.GREATER_THAN;
-import static org.apache.hudi.common.table.timeline.InstantComparison.GREATER_THAN_OR_EQUALS;
 import static org.apache.hudi.common.table.timeline.InstantComparison.compareTimestamps;
 import static org.apache.hudi.common.util.CollectionUtils.nonEmpty;
 import static org.apache.hudi.common.util.ValidationUtils.checkArgument;
@@ -104,14 +101,6 @@ public Option<HoodieCompactionPlan> execute() {
           return Option.empty();
         }
       }
-      // Committed and pending compaction instants should have strictly lower timestamps
-      List<HoodieInstant> conflictingInstants = table.getActiveTimeline()
-          .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
-          .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
-          .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
-          "Following instants have timestamps >= compactionInstant (" + instantTime + ") Instants :"
-              + conflictingInstants);
     }
 
     HoodieCompactionPlan plan = scheduleCompaction();
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java
index 1d62a0911e94f..161beb4720b11 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java
@@ -21,23 +21,29 @@
 import org.apache.hudi.avro.model.HoodieRollbackRequest;
 import org.apache.hudi.common.engine.HoodieEngineContext;
 import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
 import org.apache.hudi.common.model.HoodieCommitMetadata;
 import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
 import org.apache.hudi.common.model.WriteOperationType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
 import org.apache.hudi.common.table.timeline.CompletionTimeQueryView;
 import org.apache.hudi.common.table.timeline.HoodieInstant;
 import org.apache.hudi.common.table.timeline.HoodieTimeline;
 import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.exception.HoodieIOException;
 import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.hadoop.fs.HadoopFSUtils;
 import org.apache.hudi.storage.HoodieStorage;
 import org.apache.hudi.storage.StoragePath;
 import org.apache.hudi.storage.StoragePathFilter;
 import org.apache.hudi.storage.StoragePathInfo;
 import org.apache.hudi.table.HoodieTable;
 
+import org.apache.hadoop.fs.Path;
 import org.jetbrains.annotations.NotNull;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -46,14 +52,18 @@
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
 import java.util.function.Supplier;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
+import static org.apache.hudi.common.table.timeline.InstantComparison.LESSER_THAN;
 import static org.apache.hudi.common.table.timeline.InstantComparison.LESSER_THAN_OR_EQUALS;
 import static org.apache.hudi.common.table.timeline.InstantComparison.compareTimestamps;
 import static org.apache.hudi.common.table.timeline.MetadataConversionUtils.getHoodieCommitMetadata;
+import static org.apache.hudi.common.util.ValidationUtils.checkArgument;
 import static org.apache.hudi.table.action.rollback.BaseRollbackHelper.EMPTY_STRING;
 
 /**
@@ -164,7 +174,9 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
               // We do not know fileIds for inserts (first inserts are either log files or base files),
               // delete all files for the corresponding failed commit, if present (same as COW)
               hoodieRollbackRequests.addAll(getHoodieRollbackRequests(partitionPath, filesToDelete.get()));
-
+              if (metaClient.getTableConfig().getTableVersion().lesserThan(HoodieTableVersion.EIGHT)) {
+                hoodieRollbackRequests.addAll(getRollbackRequestToAppendForVersionSix(partitionPath, instantToRollback, commitMetadataOptional.get(), table));
+              }
               break;
             default:
               throw new HoodieRollbackException("Unknown listing type, during rollback of " + instantToRollback);
@@ -181,6 +193,62 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
     }
   }
 
+  public static List<HoodieRollbackRequest> getRollbackRequestToAppendForVersionSix(String partitionPath, HoodieInstant rollbackInstant,
+                                                                                    HoodieCommitMetadata commitMetadata, HoodieTable<?, ?, ?, ?> table) {
+    List<HoodieRollbackRequest> hoodieRollbackRequests =  new ArrayList<>();
+    checkArgument(table.version().lesserThan(HoodieTableVersion.EIGHT));
+    checkArgument(rollbackInstant.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION));
+
+    // wStat.getPrevCommit() might not give the right commit time in the following
+    // scenario : If a compaction was scheduled, the new commitTime associated with the requested compaction will be
+    // used to write the new log files. In this case, the commit time for the log file is the compaction requested time.
+    // But the index (global) might store the baseCommit of the base and not the requested, hence get the
+    // baseCommit always by listing the file slice
+    // With multi writers, rollbacks could be lazy. and so we need to use getLatestFileSlicesBeforeOrOn() instead of getLatestFileSlices()
+    Map<String, FileSlice> latestFileSlices = table.getSliceView()
+        .getLatestFileSlicesBeforeOrOn(partitionPath, rollbackInstant.requestedTime(), true)
+        .collect(Collectors.toMap(FileSlice::getFileId, Function.identity()));
+
+    List<HoodieWriteStat> hoodieWriteStats = Option.ofNullable(commitMetadata.getPartitionToWriteStats().get(partitionPath)).orElse(Collections.emptyList());
+    hoodieWriteStats = hoodieWriteStats.stream()
+        .filter(writeStat -> {
+          // Filter out stats without prevCommit since they are all inserts
+          boolean validForRollback = (writeStat != null) && (!writeStat.getPrevCommit().equals(HoodieWriteStat.NULL_COMMIT))
+              && (writeStat.getPrevCommit() != null) && latestFileSlices.containsKey(writeStat.getFileId());
+
+          if (!validForRollback) {
+            return false;
+          }
+
+          FileSlice latestFileSlice = latestFileSlices.get(writeStat.getFileId());
+
+          // For sanity, log-file base-instant time can never be less than base-commit on which we are rolling back
+          checkArgument(
+              compareTimestamps(latestFileSlice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, rollbackInstant.requestedTime()),
+              "Log-file base-instant could not be less than the instant being rolled back");
+
+          // Command block "rolling back" the preceding block {@link HoodieCommandBlockTypeEnum#ROLLBACK_PREVIOUS_BLOCK}
+          // w/in the latest file-slice is appended iff base-instant of the log-file is _strictly_ less
+          // than the instant of the Delta Commit being rolled back. Otherwise, log-file will be cleaned up
+          // in a different branch of the flow.
+          return compareTimestamps(latestFileSlice.getBaseInstantTime(), LESSER_THAN, rollbackInstant.requestedTime());
+        })
+        .collect(Collectors.toList());
+
+    for (HoodieWriteStat writeStat : hoodieWriteStats.stream().filter(
+        hoodieWriteStat -> !StringUtils.isNullOrEmpty(hoodieWriteStat.getFileId())).collect(Collectors.toList())) {
+      FileSlice latestFileSlice = latestFileSlices.get(writeStat.getFileId());
+      String fileId = writeStat.getFileId();
+      String latestBaseInstant = latestFileSlice.getBaseInstantTime();
+      Path fullLogFilePath = HadoopFSUtils.constructAbsolutePathInHadoopPath(table.getConfig().getBasePath(), writeStat.getPath());
+      Map<String, Long> logFilesWithBlocksToRollback = Collections.singletonMap(
+          fullLogFilePath.toString(), writeStat.getTotalWriteBytes() > 0 ? writeStat.getTotalWriteBytes() : 1L);
+      hoodieRollbackRequests.add(new HoodieRollbackRequest(partitionPath, fileId, latestBaseInstant,
+          Collections.emptyList(), logFilesWithBlocksToRollback));
+    }
+    return hoodieRollbackRequests;
+  }
+
   private List<StoragePathInfo> listAllFilesSinceCommit(String commit,
                                                         String baseFileExtension,
                                                         String partitionPath,
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
index 67882b47e00a3..53a814e8f2f06 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
@@ -90,7 +90,7 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
         switch (type) {
           case MERGE:
           case CREATE:
-            return createRollbackRequestForCreateAndMerge(fileId, partitionPath, filePath, instantToRollback);
+            return createRollbackRequestForCreateAndMerge(fileId, partitionPath, filePath, instantToRollback, filePathStr);
           case APPEND:
             return createRollbackRequestForAppend(fileId, partitionPath, filePath, instantToRollback, filePathStr);
           default:
@@ -106,10 +106,8 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
     }
   }
 
-  protected HoodieRollbackRequest createRollbackRequestForCreateAndMerge(String fileId,
-                                                                         String partitionPath,
-                                                                         StoragePath filePath,
-                                                                         HoodieInstant instantToRollback) {
+  protected HoodieRollbackRequest createRollbackRequestForCreateAndMerge(String fileId, String partitionPath, StoragePath filePath,
+                                                                         HoodieInstant instantToRollback, String filePathToRollback) {
     if (table.version().greaterThanOrEquals(HoodieTableVersion.EIGHT)) {
       return new HoodieRollbackRequest(partitionPath, fileId, instantToRollback.requestedTime(),
           Collections.singletonList(filePath.toString()), Collections.emptyMap());
@@ -120,7 +118,7 @@ protected HoodieRollbackRequest createRollbackRequestForCreateAndMerge(String fi
         fileId = baseFileToDelete.getFileId();
         baseInstantTime = baseFileToDelete.getCommitTime();
       } else if (FSUtils.isLogFile(filePath)) {
-        throw new HoodieRollbackException("Log files should have only APPEND as IOTypes " + filePath);
+        return createRollbackRequestForAppend(fileId, partitionPath, filePath, instantToRollback, filePathToRollback);
       }
       Objects.requireNonNull(fileId, "Cannot find valid fileId from path: " + filePath);
       Objects.requireNonNull(baseInstantTime, "Cannot find valid base instant from path: " + filePath);
diff --git a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
index 747c47b987b4d..c5b8957556d88 100644
--- a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
+++ b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
@@ -371,7 +371,7 @@ private Dataset<Row> readRecordsForGroupAsRow(JavaSparkContext jsc,
                                                 Schema tableSchemaWithMetaFields) {
     List<ClusteringOperation> clusteringOps = clusteringGroup.getSlices().stream()
         .map(ClusteringOperation::create).collect(Collectors.toList());
-    boolean canUseFileGroupReaderBasedClustering = getWriteConfig().getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
+    boolean canUseFileGroupReaderBasedClustering = HoodieReaderConfig.isFileGroupReaderEnabled(getHoodieTable().getMetaClient().getTableConfig().getTableVersion(), getWriteConfig())
         && getWriteConfig().getBooleanOrDefault(HoodieTableConfig.POPULATE_META_FIELDS)
         && clusteringOps.stream().allMatch(slice -> StringUtils.isNullOrEmpty(slice.getBootstrapFilePath()));
 
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java b/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
index 56935fc5d8170..29334c020fbf6 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
@@ -19,8 +19,14 @@
 
 package org.apache.hudi.common.config;
 
+import org.apache.hudi.common.table.HoodieTableVersion;
+
+import org.apache.hadoop.conf.Configuration;
+
 import javax.annotation.concurrent.Immutable;
 
+import java.util.Map;
+
 /**
  * Configurations for reading a file group
  */
@@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED);
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Map<String, String> parameters) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && Boolean.parseBoolean(parameters.getOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue().toString()));
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Configuration conf) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && conf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue());
+  }
 }
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java b/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
index c7f948e5d2472..1ed8b68d80b20 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
@@ -318,6 +318,11 @@ public class HoodieTableConfig extends HoodieConfig {
       DATE_TIME_PARSER
   );
 
+  private static final Set<String> CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES = new HashSet<>(Arrays.asList(
+      KEY_GENERATOR_CLASS_NAME.key(),
+      KEY_GENERATOR_TYPE.key()
+  ));
+
   public static final ConfigProperty<String> TABLE_CHECKSUM = ConfigProperty
       .key("hoodie.table.checksum")
       .noDefaultValue()
@@ -632,6 +637,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES.contains(configProperty.key());
     if (!valid) {
       LOG.warn("Table version {} is lower than or equal to config's first version {}. Config {} will be ignored.",
           tableVersion, firstVersion, configProperty.key());
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java b/hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
index 10aa50efe3e16..b0140a302e061 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
@@ -149,6 +149,8 @@ public abstract class AbstractHoodieLogRecordScanner {
   private final boolean enableOptimizedLogBlocksScan;
   // table version for compatibility
   private final HoodieTableVersion tableVersion;
+  // Allows to consider inflight instants while merging log records
+  protected boolean allowInflightInstants = false;
   // for pending log block check with table version before 8
   private HoodieTimeline commitsTimeline = null;
   private HoodieTimeline completedInstantsTimeline = null;
@@ -281,7 +283,7 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
         final String instantTime = logBlock.getLogBlockHeader().get(INSTANT_TIME);
         totalLogBlocks.incrementAndGet();
         if (logBlock.isDataOrDeleteBlock()) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
             if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
                 || getOrCreateInflightInstantsTimeline().containsInstant(instantTime)) {
               // hit an uncommitted block possibly from a failed write, move to the next one and skip processing this one
@@ -479,7 +481,7 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
             if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
                 || getOrCreateInflightInstantsTimeline().containsInstant(instantTime)) {
               // hit an uncommitted block possibly from a failed write, move to the next one and skip processing this one
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java b/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
index 4f5d0ac46399d..413942b8fd90a 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
@@ -103,7 +103,8 @@ protected HoodieMergedLogRecordScanner(HoodieStorage storage, String basePath, L
                                          InternalSchema internalSchema,
                                          Option<String> keyFieldOverride,
                                          boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger,
-                                         Option<HoodieTableMetaClient> hoodieTableMetaClientOption) {
+                                         Option<HoodieTableMetaClient> hoodieTableMetaClientOption,
+                                         boolean allowInflightInstants) {
     super(storage, basePath, logFilePaths, readerSchema, latestInstantTime, reverseReader, bufferSize,
         instantRange, withOperationField, forceFullScan, partitionName, internalSchema, keyFieldOverride, enableOptimizedLogBlocksScan, recordMerger,
         hoodieTableMetaClientOption);
@@ -113,6 +114,7 @@ protected HoodieMergedLogRecordScanner(HoodieStorage storage, String basePath, L
       this.records = new ExternalSpillableMap<>(maxMemorySizeInBytes, spillableMapBasePath, new DefaultSizeEstimator(),
           new HoodieRecordSizeEstimator(readerSchema), diskMapType, new DefaultSerializer<>(), isBitCaskDiskMapCompressionEnabled);
       this.scannedPrefixes = new HashSet<>();
+      this.allowInflightInstants = allowInflightInstants;
     } catch (IOException e) {
       throw new HoodieIOException("IOException when creating ExternalSpillableMap at " + spillableMapBasePath, e);
     }
@@ -342,6 +344,7 @@ public static class Builder extends AbstractHoodieLogRecordScanner.Builder {
     // By default, we're doing a full-scan
     private boolean forceFullScan = true;
     private boolean enableOptimizedLogBlocksScan = false;
+    protected boolean allowInflightInstants = false;
     private HoodieRecordMerger recordMerger = HoodiePreCombineAvroRecordMerger.INSTANCE;
     protected HoodieTableMetaClient hoodieTableMetaClient;
 
@@ -466,6 +469,11 @@ public Builder withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient)
       return this;
     }
 
+    public Builder withAllowInflightInstants(boolean allowInflightInstants) {
+      this.allowInflightInstants = allowInflightInstants;
+      return this;
+    }
+
     @Override
     public HoodieMergedLogRecordScanner build() {
       if (this.partitionName == null && CollectionUtils.nonEmpty(this.logFilePaths)) {
@@ -479,7 +487,7 @@ public HoodieMergedLogRecordScanner build() {
           bufferSize, spillableMapBasePath, instantRange,
           diskMapType, isBitCaskDiskMapCompressionEnabled, withOperationField, forceFullScan,
           Option.ofNullable(partitionName), internalSchema, Option.ofNullable(keyFieldOverride), enableOptimizedLogBlocksScan, recordMerger,
-          Option.ofNullable(hoodieTableMetaClient));
+          Option.ofNullable(hoodieTableMetaClient), allowInflightInstants);
     }
   }
 }
diff --git a/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java b/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
index 695b011e87557..36436e6a56d10 100644
--- a/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
+++ b/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
@@ -1008,6 +1008,7 @@ private static Map<String, HoodieRecord> getLogRecords(List<String> logFilePaths
           .withBitCaskDiskMapCompressionEnabled(storageConf.getBoolean(DISK_MAP_BITCASK_COMPRESSION_ENABLED.key(), DISK_MAP_BITCASK_COMPRESSION_ENABLED.defaultValue()))
           .withRecordMerger(recordMerger)
           .withTableMetaClient(datasetMetaClient)
+          .withAllowInflightInstants(true)
           .build();
       return mergedLogRecordScanner.getRecords();
     }
diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java
index 73e76b8189bdd..4e3ed585188cb 100644
--- a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java
@@ -40,10 +40,10 @@
 
 import static org.hamcrest.CoreMatchers.is;
 import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
 import static org.junit.jupiter.api.Assertions.assertArrayEquals;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertNull;
 import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
@@ -67,7 +67,7 @@ void testInitTableWithSpecificVersion() throws IOException {
     // Validate the partition fields & preCombineField in hoodie.properties.
     HoodieTableMetaClient metaClient1 = HoodieTestUtils.createMetaClient(tempFile.getAbsolutePath());
     assertArrayEquals(metaClient1.getTableConfig().getPartitionFields().get(), new String[] {"p0", "p1"});
-    assertNull(metaClient1.getTableConfig().getKeyGeneratorClassName());
+    assertNotNull(metaClient1.getTableConfig().getKeyGeneratorClassName());
     assertEquals(HoodieTableVersion.SIX, metaClient1.getTableConfig().getTableVersion());
   }
 
diff --git a/hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestHoodieReaderConfig.java b/hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestHoodieReaderConfig.java
new file mode 100644
index 0000000000000..5871b51cb871d
--- /dev/null
+++ b/hudi-hadoop-common/src/test/java/org/apache/hudi/common/table/TestHoodieReaderConfig.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.HoodieReaderConfig;
+
+import org.apache.hadoop.conf.Configuration;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.FILE_GROUP_READER_ENABLED;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class TestHoodieReaderConfig {
+
+  @Test
+  public void testIsFileGroupReaderEnabled() {
+    HoodieConfig config = new HoodieConfig();
+    // tbl version 8 by default allows file group reader
+    assertTrue(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, config));
+    // tbl version 6 does not allow file group reader
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.SIX, config));
+    config.setValue(FILE_GROUP_READER_ENABLED, "false");
+    // tbl version 8 does not allow file group reader when config is set to false
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, config));
+
+    Map<String, String> map = new HashMap<>();
+    // tbl version 8 by default allows file group reader
+    assertTrue(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, map));
+    // tbl version 6 does not allow file group reader
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.SIX, map));
+    map.put(FILE_GROUP_READER_ENABLED.key(), "false");
+    // tbl version 8 does not allow file group reader when config is set to false
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, map));
+
+    Configuration conf = new Configuration();
+    // tbl version 8 by default allows file group reader
+    assertTrue(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, conf));
+    // tbl version 6 does not allow file group reader
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.SIX, conf));
+    conf.set(FILE_GROUP_READER_ENABLED.key(), "false");
+    // tbl version 8 does not allow file group reader when config is set to false
+    assertFalse(HoodieReaderConfig.isFileGroupReaderEnabled(HoodieTableVersion.EIGHT, conf));
+  }
+}
diff --git a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java
index a0df73c34e218..4739a46a32505 100644
--- a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java
+++ b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java
@@ -22,6 +22,7 @@
 import org.apache.hudi.common.config.TypedProperties;
 import org.apache.hudi.common.model.HoodiePayloadProps;
 import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
 import org.apache.hudi.common.table.TableSchemaResolver;
 import org.apache.hudi.common.util.Option;
@@ -111,8 +112,13 @@ public AbstractRealtimeRecordReader(RealtimeSplit split, JobConf job) {
   }
 
   private boolean usesCustomPayload(HoodieTableMetaClient metaClient) {
-    return metaClient.getTableConfig().getRecordMergeMode().equals(RecordMergeMode.CUSTOM)
-        && metaClient.getTableConfig().getRecordMergeStrategyId().equals(HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID);
+    HoodieTableConfig tableConfig = metaClient.getTableConfig();
+    if (tableConfig.contains(HoodieTableConfig.RECORD_MERGE_MODE)
+        && tableConfig.contains(HoodieTableConfig.RECORD_MERGE_STRATEGY_ID)) {
+      return tableConfig.getRecordMergeMode().equals(RecordMergeMode.CUSTOM)
+          && tableConfig.getRecordMergeStrategyId().equals(HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID);
+    }
+    return false;
   }
 
   private void prepareHiveAvroSerializer() {
diff --git a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieCombineRealtimeRecordReader.java b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieCombineRealtimeRecordReader.java
index a663bbc6ba73e..d25dbbfaa1f55 100644
--- a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieCombineRealtimeRecordReader.java
+++ b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieCombineRealtimeRecordReader.java
@@ -54,8 +54,8 @@ public class HoodieCombineRealtimeRecordReader implements RecordReader<NullWrita
 
   public HoodieCombineRealtimeRecordReader(JobConf jobConf, CombineFileSplit split,
       List<RecordReader> readers) {
-    useFileGroupReader = shouldUseFilegroupReader(jobConf, split);
     try {
+      useFileGroupReader = shouldUseFilegroupReader(jobConf, split);
       ValidationUtils.checkArgument(((HoodieCombineRealtimeFileSplit) split).getRealtimeFileSplits().size() == readers
           .size(), "Num Splits does not match number of unique RecordReaders!");
       for (InputSplit rtSplit : ((HoodieCombineRealtimeFileSplit) split).getRealtimeFileSplits()) {
diff --git a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
index aa24bd36fd92b..3797b052599dc 100644
--- a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
+++ b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
@@ -69,6 +69,7 @@
 import org.apache.hadoop.mapred.FileSplit;
 import org.apache.hadoop.mapred.InputSplit;
 import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.lib.CombineFileSplit;
 import org.apache.hadoop.mapreduce.Job;
 import org.apache.hadoop.mapreduce.JobContext;
 import org.slf4j.Logger;
@@ -92,6 +93,7 @@
 import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
 import static org.apache.hudi.common.table.timeline.TimelineUtils.handleHollowCommitIfNeeded;
 import static org.apache.hudi.hadoop.fs.HadoopFSUtils.convertToStoragePath;
+import static org.apache.hudi.hadoop.fs.HadoopFSUtils.getStorageConf;
 
 public class HoodieInputFormatUtils {
 
@@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
+      return HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), jobConf)
+          && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
+          && !(split instanceof BootstrapBaseFileSplit);
+    } else if (split instanceof CombineFileSplit) {
+      for (Path path : ((CombineFileSplit) split).getPaths()) {
+        HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTablePath(jobConf, path)).build();
+        boolean isFileGroupReaderEnabled = HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), jobConf)
+            && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
+            && !(split instanceof BootstrapBaseFileSplit);
+        if (!isFileGroupReaderEnabled) {
+          return false;
+        }
+      }
+      return true;
+    } else {
+      return false;
+    }
   }
 }
diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala
index 8070ba1619c93..21f212e477763 100644
--- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala
+++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala
@@ -304,9 +304,8 @@ object DefaultSource {
         Option(schema)
       }
 
-      val useNewParquetFileFormat = parameters.getOrElse(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(),
-        HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue().toString).toBoolean &&
-        !metaClient.isMetadataTable && (globPaths == null || globPaths.isEmpty)
+      val useNewParquetFileFormat = !metaClient.isMetadataTable && (globPaths == null || globPaths.isEmpty) &&
+        HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig.getTableVersion, parameters.asJava)
       if (metaClient.getCommitsTimeline.filterCompletedInstants.countInstants() == 0) {
         new EmptyRelation(sqlContext, resolveSchema(metaClient, parameters, Some(schema)))
       } else if (isCdcQuery) {
diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV1.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV1.scala
index bc0869840d0db..df5b2cdd87041 100644
--- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV1.scala
+++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV1.scala
@@ -218,9 +218,7 @@ class IncrementalRelationV1(val sqlContext: SQLContext,
       //   1. the start commit is archived
       //   2. the end commit is archived
       //   3. there are files in metadata be deleted
-      val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
-        DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
-
+      val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key, "false").toBoolean
       val sOpts = optParams.filter(p => !p._1.equalsIgnoreCase("path"))
 
       val startInstantTime = optParams(DataSourceReadOptions.START_COMMIT.key)
diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV1.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV1.scala
index 62030fed870bd..4538d7c318217 100644
--- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV1.scala
+++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV1.scala
@@ -199,9 +199,7 @@ trait HoodieIncrementalRelationV1Trait extends HoodieBaseRelation {
   //   2. the end commit is archived
   //   3. there are files in metadata be deleted
   protected lazy val fullTableScan: Boolean = {
-    val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
-      DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
-
+    val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key, "false").toBoolean
     fallbackToFullTableScan && (startInstantArchived || endInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
diff --git a/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java
index f8686dff0e2e5..44d08f6fb4d5a 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java
+++ b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java
@@ -62,6 +62,7 @@
 import org.junit.jupiter.api.Tag;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
 import org.junit.jupiter.params.provider.ValueSource;
 
 import java.io.File;
@@ -153,14 +154,16 @@ void testCOWToMORConvertedTableRollback(boolean rollbackUsingMarkers) throws Exc
   }
 
   @ParameterizedTest
-  @ValueSource(booleans = {true, false})
-  void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) throws Exception {
+  @CsvSource(value = {"true,6", "false,6", "true,8", "false,8"})
+  void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers, int tableVersion) throws Exception {
     // NOTE: First writer will have Metadata table DISABLED
     HoodieWriteConfig.Builder cfgBuilder =
         getConfigBuilder(false, rollbackUsingMarkers, HoodieIndex.IndexType.SIMPLE);
 
     addConfigsForPopulateMetaFields(cfgBuilder, true);
     HoodieWriteConfig cfg = cfgBuilder.build();
+    cfg.setValue(HoodieTableConfig.VERSION, String.valueOf(tableVersion));
+    cfg.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(tableVersion));
 
     Properties properties = CollectionUtils.copy(cfg.getProps());
     properties.setProperty(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieTableConfig.BASE_FILE_FORMAT.defaultValue().toString());
@@ -207,13 +210,16 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
       assertTrue(dataFilesToRead.findAny().isPresent(),
           "should list the base files we wrote in the delta commit");
 
+      HoodieWriteConfig secondCfg = getHoodieWriteConfigWithSmallFileHandlingOff(true);
+      secondCfg.setValue(HoodieTableConfig.VERSION, String.valueOf(tableVersion));
+      secondCfg.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(tableVersion));
       /*
        * Write 2 (inserts + updates - testing failed delta commit)
        */
       final String commitTime1 = "000000002";
       // WriteClient with custom config (disable small file handling)
       // NOTE: Second writer will have Metadata table ENABLED
-      try (SparkRDDWriteClient secondClient = getHoodieWriteClient(getHoodieWriteConfigWithSmallFileHandlingOff(true));) {
+      try (SparkRDDWriteClient secondClient = getHoodieWriteClient(secondCfg);) {
         secondClient.startCommitWithTime(commitTime1);
 
         List<HoodieRecord> copyOfRecords = new ArrayList<>(records);
@@ -254,7 +260,7 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
        * Write 3 (inserts + updates - testing successful delta commit)
        */
       final String commitTime2 = "000000003";
-      try (SparkRDDWriteClient thirdClient = getHoodieWriteClient(getHoodieWriteConfigWithSmallFileHandlingOff(true));) {
+      try (SparkRDDWriteClient thirdClient = getHoodieWriteClient(secondCfg);) {
         thirdClient.startCommitWithTime(commitTime2);
 
         List<HoodieRecord> copyOfRecords = new ArrayList<>(records);
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala
index 51d8ad607e137..7d84c8088a384 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala
@@ -30,7 +30,7 @@ import org.apache.hudi.common.config.TimestampKeyGeneratorConfig.{TIMESTAMP_INPU
 import org.apache.hudi.common.fs.FSUtils
 import org.apache.hudi.common.model.{HoodieRecord, WriteOperationType}
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType
-import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient, TableSchemaResolver}
 import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline, TimelineUtils}
 import org.apache.hudi.common.testutils.{HoodieTestDataGenerator, HoodieTestUtils}
 import org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_FILE_NAME_GENERATOR
@@ -1158,9 +1158,12 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
       concat(col("driver"), lit("/"), col("rider"), lit("/"), udf_date_format(col("current_ts")))).count() == 0)
   }
 
-  @Test
-  def testPartitionPruningForTimestampBasedKeyGenerator(): Unit = {
-    val (writeOpts, readOpts) = getWriterReaderOptsLessPartitionPath(HoodieRecordType.AVRO, enableFileIndex = true)
+  @ParameterizedTest
+  @CsvSource(value = Array("6", "8"))
+  def testPartitionPruningForTimestampBasedKeyGenerator(tableVersion: Int): Unit = {
+    var (writeOpts, readOpts) = getWriterReaderOptsLessPartitionPath(HoodieRecordType.AVRO, enableFileIndex = true)
+    writeOpts = writeOpts + (HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
     val writer = getDataFrameWriter(classOf[TimestampBasedKeyGenerator].getName, writeOpts)
     writer.partitionBy("current_ts")
       .option(TIMESTAMP_TYPE_FIELD.key, "EPOCHMILLISECONDS")
@@ -1832,9 +1835,11 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
   }
 
   @ParameterizedTest
-  @EnumSource(value = classOf[HoodieInstant.State], names = Array("REQUESTED", "INFLIGHT", "COMPLETED"))
-  def testInsertOverwriteCluster(firstClusteringState: HoodieInstant.State): Unit = {
-    val (writeOpts, _) = getWriterReaderOpts()
+  @CsvSource(value = Array("REQUESTED,6", "REQUESTED,8", "INFLIGHT,6", "INFLIGHT,8", "COMPLETED,6", "COMPLETED,8"))
+  def testInsertOverwriteCluster(firstClusteringState: String, tableVersion: Int): Unit = {
+    var (writeOpts, _) = getWriterReaderOpts()
+    writeOpts = writeOpts + (HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
 
     // Insert Operation
     val records = recordsToStrings(dataGen.generateInserts("000", 100)).asScala.toList
@@ -1896,8 +1901,8 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
           .withPath(basePath)
           .withProps(optsWithCluster.asJava)
           .build()
-        if (firstClusteringState == HoodieInstant.State.INFLIGHT
-          || firstClusteringState == HoodieInstant.State.REQUESTED) {
+        if (firstClusteringState == HoodieInstant.State.INFLIGHT.name()
+          || firstClusteringState == HoodieInstant.State.REQUESTED.name()) {
           // Move the clustering to inflight for testing
           storage.deleteFile(new StoragePath(metaClient.getTimelinePath, INSTANT_FILE_NAME_GENERATOR.getFileName(lastInstant)))
           val inflightClustering = metaClient.reloadActiveTimeline.lastInstant.get
@@ -1906,7 +1911,7 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
             inflightClustering,
             metaClient.getActiveTimeline.getLastClusteringInstant.get)
         }
-        if (firstClusteringState == HoodieInstant.State.REQUESTED) {
+        if (firstClusteringState == HoodieInstant.State.REQUESTED.name()) {
           val table = HoodieSparkTable.create(writeConfig, context)
           table.rollbackInflightClustering(
             metaClient.getActiveTimeline.getLastClusteringInstant.get,
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
index 14ae787a8256d..a4e240c57fc47 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
@@ -34,7 +34,7 @@ import org.apache.hudi.config.{HoodieCompactionConfig, HoodieIndexConfig, Hoodie
 import org.apache.hudi.functional.TestCOWDataSource.convertColumnsToNullable
 import org.apache.hudi.index.HoodieIndex.IndexType
 import org.apache.hudi.metadata.HoodieTableMetadataUtil.{metadataPartitionExists, PARTITION_NAME_SECONDARY_INDEX_PREFIX}
-import org.apache.hudi.storage.StoragePath
+import org.apache.hudi.storage.{StoragePath, StoragePathInfo}
 import org.apache.hudi.table.action.compact.CompactionTriggerStrategy
 import org.apache.hudi.testutils.{DataSourceTestUtils, HoodieSparkClientTestBase}
 import org.apache.hudi.util.JFunction
@@ -50,6 +50,7 @@ import org.junit.jupiter.params.ParameterizedTest
 import org.junit.jupiter.params.provider.{CsvSource, EnumSource, ValueSource}
 
 import java.util.function.Consumer
+import java.util.stream.Collectors
 
 import scala.collection.JavaConverters._
 
@@ -462,9 +463,11 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
   }
 
   @ParameterizedTest
-  @EnumSource(value = classOf[HoodieRecordType], names = Array("AVRO", "SPARK"))
-  def testPayloadDelete(recordType: HoodieRecordType) {
-    val (writeOpts, readOpts) = getWriterReaderOpts(recordType)
+  @CsvSource(value = Array("AVRO,6", "AVRO,8", "SPARK,6", "SPARK,8"))
+  def testPayloadDelete(recordType: HoodieRecordType, tableVersion: Int) {
+    var (writeOpts, readOpts) = getWriterReaderOpts(recordType)
+    writeOpts = writeOpts + (HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
 
     // First Operation:
     // Producing parquet files to three default partitions.
@@ -539,10 +542,11 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
   }
 
   @ParameterizedTest
-  @EnumSource(value = classOf[HoodieRecordType], names = Array("AVRO", "SPARK"))
-  def testPrunedFiltered(recordType: HoodieRecordType) {
-
-    val (writeOpts, readOpts) = getWriterReaderOpts(recordType)
+  @CsvSource(value = Array("AVRO,6", "AVRO,8", "SPARK,6", "SPARK,8"))
+  def testPrunedFiltered(recordType: HoodieRecordType, tableVersion: Int) {
+    var (writeOpts, readOpts) = getWriterReaderOpts(recordType)
+    writeOpts = writeOpts + (HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
 
     // First Operation:
     // Producing parquet files to three default partitions.
@@ -604,7 +608,7 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
     val hudiIncDF2 = spark.read.format("org.apache.hudi")
       .options(readOpts)
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
-      .option(DataSourceReadOptions.START_COMMIT.key, commit2CompletionTime)
+      .option(DataSourceReadOptions.START_COMMIT.key, if (tableVersion == 6) commit1Time else commit2CompletionTime)
       .load(basePath)
 
     // filter first commit and only read log records
@@ -1429,6 +1433,69 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
       roDf.where(col(recordKeyField) === 0).select(dataField).collect()(0).getLong(0))
   }
 
+  @ParameterizedTest
+  @CsvSource(value = Array("true,6", "true,8", "false,6", "false,8"))
+//  @ValueSource(booleans = Array(true, false))
+  def testSnapshotQueryAfterInflightDeltaCommit(enableFileIndex: Boolean, tableVersion: Int): Unit = {
+    val (tableName, tablePath) = ("hoodie_mor_snapshot_read_test_table", s"${basePath}_mor_test_table")
+    val precombineField = "col3"
+    val recordKeyField = "key"
+    val dataField = "age"
+
+    val options = Map[String, String](
+      DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.MERGE_ON_READ.name,
+      DataSourceWriteOptions.OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+      DataSourceWriteOptions.PRECOMBINE_FIELD.key -> precombineField,
+      DataSourceWriteOptions.RECORDKEY_FIELD.key -> recordKeyField,
+      DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "",
+      DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key -> "org.apache.hudi.keygen.NonpartitionedKeyGenerator",
+      HoodieWriteConfig.TBL_NAME.key -> tableName,
+      "hoodie.insert.shuffle.parallelism" -> "1",
+      "hoodie.upsert.shuffle.parallelism" -> "1")
+    val pathForQuery = getPathForROQuery(tablePath, !enableFileIndex, 0)
+
+    var (writeOpts, readOpts) = getWriterReaderOpts(HoodieRecordType.AVRO, options, enableFileIndex)
+    writeOpts = writeOpts ++ Map(
+      HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
+
+    val firstDf = spark.range(0, 10).toDF(recordKeyField)
+      .withColumn(precombineField, expr(recordKeyField))
+      .withColumn(dataField, expr(recordKeyField + " + 1000"))
+    firstDf.write.format("hudi")
+      .options(writeOpts)
+      .mode(SaveMode.Overwrite)
+      .save(tablePath)
+
+    val secondDf = spark.range(0, 10).toDF(recordKeyField)
+      .withColumn(precombineField, expr(recordKeyField))
+      .withColumn(dataField, expr(recordKeyField + " + 2000"))
+    secondDf.write.format("hudi")
+      .options(writeOpts)
+      .mode(SaveMode.Append).save(tablePath)
+
+    // Snapshot query on MOR
+    val snapshotDf = spark.read.format("org.apache.hudi")
+      .options(readOpts)
+      .load(pathForQuery)
+
+    // Delete last completed instant
+    metaClient = createMetaClient(spark, tablePath)
+    val files = storage.listDirectEntries(metaClient.getTimelinePath).stream()
+      .filter(f => f.getPath.getName.contains(metaClient.getActiveTimeline.lastInstant().get().requestedTime())
+        && !f.getPath.getName.contains("inflight")
+        && !f.getPath.getName.contains("requested"))
+      .collect(Collectors.toList[StoragePathInfo]).asScala
+    assertEquals(1, files.size)
+    storage.deleteFile(files.head.getPath)
+
+    // verify snapshot query returns data written using firstDf
+    assertEquals(10, snapshotDf.count())
+    assertEquals(
+      1000L,
+      snapshotDf.where(col(recordKeyField) === 0).select(dataField).collect()(0).getLong(0))
+  }
+
   @ParameterizedTest
   @CsvSource(Array(
     "AVRO, AVRO, END_MAP",
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala
index 5a11cc71b4f92..56c2f01e1f32e 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndex.scala
@@ -23,7 +23,7 @@ import org.apache.hudi.DataSourceWriteOptions._
 import org.apache.hudi.client.transaction.PreferWriterConflictResolutionStrategy
 import org.apache.hudi.common.config.HoodieMetadataConfig
 import org.apache.hudi.common.model._
-import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
 import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
 import org.apache.hudi.common.testutils.{HoodieTestDataGenerator, InProcessTimeGenerator}
 import org.apache.hudi.common.testutils.RawTripTestPayload.recordsToStrings
@@ -40,7 +40,7 @@ import org.junit.jupiter.params.ParameterizedTest
 import org.junit.jupiter.params.provider.{Arguments, CsvSource, EnumSource, MethodSource}
 import org.junit.jupiter.params.provider.Arguments.arguments
 
-import java.util.Collections
+import java.util.{Collections, Properties}
 import java.util.concurrent.Executors
 
 import scala.collection.JavaConverters._
@@ -228,9 +228,20 @@ class TestRecordLevelIndex extends RecordLevelIndexTestBase {
   }
 
   @ParameterizedTest
-  @EnumSource(classOf[HoodieTableType])
-  def testRLIWithDelete(tableType: HoodieTableType): Unit = {
-    val hudiOpts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name())
+  @CsvSource(value = Array(
+    "COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"
+  ))
+  def testRLIWithDelete(tableType: String, tableVersion: Int): Unit = {
+    val hudiOpts = commonOpts ++ Map(
+      DataSourceWriteOptions.TABLE_TYPE.key -> tableType,
+      HoodieTableConfig.VERSION.key() -> tableVersion.toString,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString)
+    val props = new Properties()
+    for ((k, v) <- hudiOpts) {
+      props.put(k, v)
+    }
+    initMetaClient(HoodieTableType.valueOf(tableType), props)
+
     val insertDf = doWriteAndValidateDataAndRecordIndex(hudiOpts,
       operation = DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL,
       saveMode = SaveMode.Overwrite)
@@ -408,13 +419,22 @@ class TestRecordLevelIndex extends RecordLevelIndexTestBase {
   }
 
   @ParameterizedTest
-  @EnumSource(classOf[HoodieTableType])
-  def testRLIWithDTClustering(tableType: HoodieTableType): Unit = {
+  @CsvSource(value = Array(
+    "COPY_ON_WRITE,6", "COPY_ON_WRITE,8", "MERGE_ON_READ,6", "MERGE_ON_READ,8"
+  ))
+  def testRLIWithDTClustering(tableType: String, tableVersion: Int): Unit = {
     val hudiOpts = commonOpts ++ Map(
-      DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name(),
+      DataSourceWriteOptions.TABLE_TYPE.key -> tableType,
+      HoodieWriteConfig.WRITE_TABLE_VERSION.key() -> tableVersion.toString,
+      HoodieTableConfig.VERSION.key() -> tableVersion.toString,
       HoodieClusteringConfig.INLINE_CLUSTERING.key() -> "true",
       HoodieClusteringConfig.INLINE_CLUSTERING_MAX_COMMITS.key() -> "2"
     )
+    val props = new Properties()
+    for ((k, v) <- hudiOpts) {
+      props.put(k, v)
+    }
+    initMetaClient(HoodieTableType.valueOf(tableType), props)
 
     doWriteAndValidateDataAndRecordIndex(hudiOpts,
       operation = DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL,
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
index ce1a407eaa9d5..6ecc3af6bde68 100644
--- a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
@@ -380,8 +380,7 @@ private Pair<Option<Dataset<Row>>, Checkpoint> fetchNextBatchBasedOnRequestedTim
           .option(START_COMMIT().key(), queryInfo.getStartInstant())
           .option(END_COMMIT().key(), queryInfo.getEndInstant())
           .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
+              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(), "false"))
           .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
           .load(srcPath);
     } else {
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/QueryRunner.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/QueryRunner.java
index 2bb1ec1534538..3a3f65f0b5ddb 100644
--- a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/QueryRunner.java
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/QueryRunner.java
@@ -21,6 +21,8 @@
 import org.apache.hudi.DataSourceReadOptions;
 import org.apache.hudi.common.config.TypedProperties;
 import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.collection.Pair;
 import org.apache.hudi.exception.HoodieException;
@@ -38,8 +40,10 @@
 import java.util.Collections;
 import java.util.List;
 
+import static org.apache.hudi.DataSourceReadOptions.INCREMENTAL_READ_TABLE_VERSION;
 import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
 import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.hadoop.fs.HadoopFSUtils.getStorageConf;
 
 /**
  * This class is currently used only by s3 and gcs incr sources that supports size based batching
@@ -85,13 +89,16 @@ public static Dataset<Row> applyOrdering(Dataset<Row> dataset, List<String> orde
 
   public Pair<QueryInfo, Dataset<Row>> runIncrementalQuery(QueryInfo queryInfo) {
     LOG.info("Running incremental query");
+
+    HoodieTableVersion tableVersion = HoodieTableMetaClient.builder().setConf(getStorageConf()).setBasePath(sourcePath).build().getTableConfig().getTableVersion();
     return Pair.of(queryInfo, sparkSession.read().format("org.apache.hudi")
         .option(DataSourceReadOptions.QUERY_TYPE().key(), queryInfo.getQueryType())
+        .option(INCREMENTAL_READ_TABLE_VERSION().key(), tableVersion.versionCode())
         .option(DataSourceReadOptions.START_COMMIT().key(), queryInfo.getStartInstant())
         .option(DataSourceReadOptions.END_COMMIT().key(), queryInfo.getEndInstant())
         .option(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
             props.getString(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
+                tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) ? DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue() : "false"))
         .load(sourcePath));
   }
 


Reviews:
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-27T11:37:27Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-27T11:38:22Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T13:25:44Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T04:20:14Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T08:36:42Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T08:39:53Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T08:41:16Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T08:59:22Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T09:00:51Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T11:12:07Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T11:12:50Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T11:13:00Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T11:13:48Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T02:11:17Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T04:26:28Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T09:58:28Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T04:31:24Z
user: login: nsivabalan
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T04:31:37Z
user: login: yihua
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:20:42Z
user: login: yihua
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:25:22Z
user: login: yihua
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:34:35Z
user: login: yihua
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:43:43Z
user: login: yihua
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:51:27Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T22:51:04Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T22:54:36Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T23:57:54Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T23:58:19Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T00:00:30Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T00:02:39Z
user: login: nsivabalan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T00:04:07Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T06:18:04Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T06:19:31Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T06:21:56Z
user: login: lokeshj1703
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T06:27:06Z
user: login: linliu-code
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T18:17:36Z

Review comments:
diff_hunk: @@ -106,10 +106,8 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
     }
   }
 
-  protected HoodieRollbackRequest createRollbackRequestForCreateAndMerge(String fileId,
-                                                                         String partitionPath,
-                                                                         StoragePath filePath,
-                                                                         HoodieInstant instantToRollback) {
+  protected HoodieRollbackRequest createRollbackRequestForCreateAndMerge(String fileId, String partitionPath, StoragePath filePath,
+                                                                         HoodieInstant instantToRollback, String filePathToRollback) {
     if (table.version().greaterThanOrEquals(HoodieTableVersion.EIGHT)) {
       return new HoodieRollbackRequest(partitionPath, fileId, instantToRollback.requestedTime(),
           Collections.singletonList(filePath.toString()), Collections.emptyMap());
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
user: login: lokeshj1703
body: The change is only applicable for table version less than 8.
created_at: 2025-02-27T11:37:26Z
author_association: COLLABORATOR
diff_hunk: @@ -61,7 +61,7 @@ public class HoodieReaderConfig extends HoodieConfig {
 
   public static final ConfigProperty<Boolean> FILE_GROUP_READER_ENABLED = ConfigProperty
       .key("hoodie.file.group.reader.enabled")
-      .defaultValue(true)
+      .defaultValue(false)
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: lokeshj1703
body: @nsivabalan How do I fetch the table version here in the infer function? I think it only allows hoodie config as parameter.
created_at: 2025-02-27T11:38:22Z
author_association: COLLABORATOR
diff_hunk: @@ -61,7 +61,7 @@ public class HoodieReaderConfig extends HoodieConfig {
 
   public static final ConfigProperty<Boolean> FILE_GROUP_READER_ENABLED = ConfigProperty
       .key("hoodie.file.group.reader.enabled")
-      .defaultValue(true)
+      .defaultValue(false)
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: lokeshj1703
body: Fixed the usage of these configs
created_at: 2025-02-28T13:25:44Z
author_association: COLLABORATOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: nsivabalan
body: why are we removing it? 
and even if required, is it possible to enable it only for tbl version 8 or above?
created_at: 2025-03-01T04:09:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: nsivabalan
body: do we have UTs for this?
created_at: 2025-03-01T04:12:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -632,6 +632,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || configProperty.key().equals(KEY_GENERATOR_CLASS_NAME.key()) || configProperty.key().equals(KEY_GENERATOR_TYPE.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: nsivabalan
body: shouldn't this be &&? 
created_at: 2025-03-01T04:13:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,13 +281,6 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
         final String instantTime = logBlock.getLogBlockHeader().get(INSTANT_TIME);
         totalLogBlocks.incrementAndGet();
         if (logBlock.isDataOrDeleteBlock()) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
-            if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
-                || getOrCreateInflightInstantsTimeline().containsInstant(instantTime)) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: nsivabalan
body: we can't just remove this constraints. 
We needed this specifically for tbl6 and while preparing RLI records. 
But for a regular MOR table reader, we can't expose uncommitted data. 

So, we might have to add additional argument to log record reader and then set it to true only when we are preparing RLI records. If not, it should be false on which case the condition check and continue logic should still be there for tbl v 6. 


can we add docs here as to why we are allowing here. 
created_at: 2025-03-01T04:16:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -479,13 +472,6 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
-            if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: nsivabalan
body: same here. 
created_at: 2025-03-01T04:16:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -537,9 +538,13 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (!(split instanceof FileSplit)) {
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: nsivabalan
body: why is this fix required? can you help me understand
created_at: 2025-03-01T04:17:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -380,8 +380,7 @@ private Pair<Option<Dataset<Row>>, Checkpoint> fetchNextBatchBasedOnRequestedTim
           .option(START_COMMIT().key(), queryInfo.getStartInstant())
           .option(END_COMMIT().key(), queryInfo.getEndInstant())
           .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
+              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(), "false"))
path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
user: login: nsivabalan
body: why flipping the def value even for tbl version 8? 
created_at: 2025-03-01T04:19:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -380,8 +380,7 @@ private Pair<Option<Dataset<Row>>, Checkpoint> fetchNextBatchBasedOnRequestedTim
           .option(START_COMMIT().key(), queryInfo.getStartInstant())
           .option(END_COMMIT().key(), queryInfo.getEndInstant())
           .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
+              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(), "false"))
path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
user: login: lokeshj1703
body: The change is made for function `fetchNextBatchBasedOnRequestedTime` which is used for table version 6.
created_at: 2025-03-01T08:36:42Z
author_association: COLLABORATOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: lokeshj1703
body: This was not part of table version 8 code. This was added specifically for table version 6 in https://github.com/apache/hudi/pull/12805/files
created_at: 2025-03-01T08:39:53Z
author_association: COLLABORATOR
diff_hunk: @@ -632,6 +632,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || configProperty.key().equals(KEY_GENERATOR_CLASS_NAME.key()) || configProperty.key().equals(KEY_GENERATOR_TYPE.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: lokeshj1703
body: KEY_GENERATOR_CLASS_NAME  and KEY_GENERATOR_TYPE were added in table version 8 but are also used in table version 6 code. Valid flag fails for them when table version is 6 so need to check for these keys.
created_at: 2025-03-01T08:41:16Z
author_association: COLLABORATOR
diff_hunk: @@ -537,9 +538,13 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (!(split instanceof FileSplit)) {
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: lokeshj1703
body: https://github.com/apache/hudi/blob/d2a617f7b6c78da7614cce8db1bac7ea4e5bf7b4/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java#L120-L124

The file group reader is only created when input split is of type FileSplit. Therefore added this constraint.
created_at: 2025-03-01T08:59:22Z
author_association: COLLABORATOR
diff_hunk: @@ -537,9 +538,13 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (!(split instanceof FileSplit)) {
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: lokeshj1703
body: @jonvex Can you please take a look and check if the change here looks ok?
created_at: 2025-03-01T09:00:51Z
author_association: COLLABORATOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: lokeshj1703
body: Added UTs
created_at: 2025-03-01T11:12:07Z
author_association: COLLABORATOR
diff_hunk: @@ -281,13 +281,6 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
         final String instantTime = logBlock.getLogBlockHeader().get(INSTANT_TIME);
         totalLogBlocks.incrementAndGet();
         if (logBlock.isDataOrDeleteBlock()) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
-            if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
-                || getOrCreateInflightInstantsTimeline().containsInstant(instantTime)) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: lokeshj1703
body: Fixed the usage. Now these checks are not made for RLI.
created_at: 2025-03-01T11:12:50Z
author_association: COLLABORATOR
diff_hunk: @@ -479,13 +472,6 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
-            if (!getOrCreateCompletedInstantsTimeline().containsOrBeforeTimelineStarts(instantTime)
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: lokeshj1703
body: Addressed
created_at: 2025-03-01T11:13:00Z
author_association: COLLABORATOR
diff_hunk: @@ -537,9 +538,13 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (!(split instanceof FileSplit)) {
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: lokeshj1703
body: `shouldUseFilegroupReader` API is used for FileSplit, RealtimeSplit and CombineFileSplit. Addressing all three now. PTAL
created_at: 2025-03-01T11:13:48Z
author_association: COLLABORATOR
diff_hunk: @@ -632,6 +632,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || configProperty.key().equals(KEY_GENERATOR_CLASS_NAME.key()) || configProperty.key().equals(KEY_GENERATOR_TYPE.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: nsivabalan
body: lets discuss the right fix for this 
created_at: 2025-03-03T02:06:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: nsivabalan
body: lets confirm to get this reviewed from @jonvex 
created_at: 2025-03-03T02:09:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,7 +283,7 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
         final String instantTime = logBlock.getLogBlockHeader().get(INSTANT_TIME);
         totalLogBlocks.incrementAndGet();
         if (logBlock.isDataOrDeleteBlock()) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: nsivabalan
body: can we ensure we have a tbl v6 test for the following:
MOR w/ log files from pending ingestion commit. when we do snapshot read, the log file from pending ingestion commit should be skipped by the log record reader 
created_at: 2025-03-03T04:26:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,7 +283,7 @@ private void scanInternalV1(Option<KeySpec> keySpecOpt) {
         final String instantTime = logBlock.getLogBlockHeader().get(INSTANT_TIME);
         totalLogBlocks.incrementAndGet();
         if (logBlock.isDataOrDeleteBlock()) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: lokeshj1703
body: Addressed
created_at: 2025-03-03T09:58:28Z
author_association: COLLABORATOR
diff_hunk: @@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: nsivabalan
body: can you file a follow up jira to get this reviewed by ethan. for now, we can go ahead and land this patch. 
created_at: 2025-03-04T04:31:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,7 +146,7 @@ public HoodieData<WriteStatus> compact(
 
     boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()   // the engine needs to support fg reader first
         && !metaClient.isMetadataTable()
-        && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
+        && HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), config)
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java
user: login: yihua
body: The file group reader `HoodieFileGroupReader` should be the only way of reading records from a file group eventually.  Let's keep that goal in mind and do not introduce any new fallback.
created_at: 2025-03-05T18:20:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: yihua
body: Do we need this for backwards compatible compaction for table version 6?
created_at: 2025-03-05T18:21:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,7 +174,9 @@ public List<HoodieRollbackRequest> getRollbackRequests(HoodieInstant instantToRo
               // We do not know fileIds for inserts (first inserts are either log files or base files),
               // delete all files for the corresponding failed commit, if present (same as COW)
               hoodieRollbackRequests.addAll(getHoodieRollbackRequests(partitionPath, filesToDelete.get()));
-
+              if (metaClient.getTableConfig().getTableVersion().lesserThan(HoodieTableVersion.EIGHT)) {
+                hoodieRollbackRequests.addAll(getRollbackRequestToAppendForVersionSix(partitionPath, instantToRollback, commitMetadataOptional.get(), table));
+              }
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java
user: login: yihua
body: Can we have the diverging logic within `getHoodieRollbackRequests` instead of polluting the core logic here?  Does other place calling `getHoodieRollbackRequests` need revisiting too?
created_at: 2025-03-05T18:23:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED);
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: yihua
body: We should not differentiate the reader logic at this layer.  If there are gaps regarding the file group reader reading the base and log files written in table version 6, let's fix that within the file group reader.  Otherwise, we'll never deprecate the legacy log scanner and reader path.
created_at: 2025-03-05T18:25:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED);
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Map<String, String> parameters) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && Boolean.parseBoolean(parameters.getOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue().toString()));
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Configuration conf) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && conf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue());
+  }
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: yihua
body: Let's remove all these new config logic and fix forward.

In fact, basic functionality of reading file groups in table version 6 is validated in `TestMergeModeCommitTimeOrdering`, `TestMergeModeEventTimeOrdering`.  So we should dig deeper and understand the additional issue discovered if there is any.
created_at: 2025-03-05T18:29:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -632,6 +637,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES.contains(configProperty.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: yihua
body: What is the rationale here?  It looks like we need to validate this in another place instead of here.
created_at: 2025-03-05T18:32:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -479,7 +481,7 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: yihua
body: Could you fix `HoodieFileGroupReader`, `BaseHoodieLogRecordReader`, and `HoodieBaseFileGroupRecordBuffer` on the same logic?  `*LogRecordScanner` classes are going to be deprecated.
created_at: 2025-03-05T18:34:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,7 +103,8 @@ protected HoodieMergedLogRecordScanner(HoodieStorage storage, String basePath, L
                                          InternalSchema internalSchema,
                                          Option<String> keyFieldOverride,
                                          boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger,
-                                         Option<HoodieTableMetaClient> hoodieTableMetaClientOption) {
+                                         Option<HoodieTableMetaClient> hoodieTableMetaClientOption,
+                                         boolean allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
user: login: yihua
body: Similar on changes in the log record scanner class.
created_at: 2025-03-05T18:36:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: yihua
body: Let's add a JIRA ticket and tag the fix version `1.0.2`.
created_at: 2025-03-05T18:41:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
+      return HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), jobConf)
+          && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
+          && !(split instanceof BootstrapBaseFileSplit);
+    } else if (split instanceof CombineFileSplit) {
+      for (Path path : ((CombineFileSplit) split).getPaths()) {
+        HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTablePath(jobConf, path)).build();
+        boolean isFileGroupReaderEnabled = HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), jobConf)
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: yihua
body: Similarly, let's remove this logic and fix the file group reader logic forward.
created_at: 2025-03-05T18:46:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -304,9 +304,8 @@ object DefaultSource {
         Option(schema)
       }
 
-      val useNewParquetFileFormat = parameters.getOrElse(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(),
-        HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue().toString).toBoolean &&
-        !metaClient.isMetadataTable && (globPaths == null || globPaths.isEmpty)
+      val useNewParquetFileFormat = !metaClient.isMetadataTable && (globPaths == null || globPaths.isEmpty) &&
+        HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig.getTableVersion, parameters.asJava)
path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala
user: login: yihua
body: Similarly, remove this.
created_at: 2025-03-05T18:47:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,7 +146,7 @@ public HoodieData<WriteStatus> compact(
 
     boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()   // the engine needs to support fg reader first
         && !metaClient.isMetadataTable()
-        && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
+        && HoodieReaderConfig.isFileGroupReaderEnabled(metaClient.getTableConfig().getTableVersion(), config)
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java
user: login: nsivabalan
body: https://github.com/apache/hudi/pull/12935
created_at: 2025-03-07T22:51:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: nsivabalan
body: nope. this piece of code was added back by me just for table version 6. 
https://github.com/apache/hudi/pull/12805/files 

I added 2 validations as part of that patch. And @lokeshj1703  is removing the 2nd one from it. 
@lokeshj1703 : can you help us understand why do we need to remove the validation? 

If there was a test written specifically targetted for table version 8, but when you tried to run it against table version 6 and it failed -> And to get the test to succeed, if we are removing this validation, its not something that we wanted to do. 

Lets reason about table version 6 and take a call.  
created_at: 2025-03-07T22:54:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED);
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: nsivabalan
body: https://github.com/apache/hudi/pull/12935
created_at: 2025-03-07T23:57:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -632,6 +637,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES.contains(configProperty.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: nsivabalan
body: @lokeshj1703 : can you respond to this. 
created_at: 2025-03-07T23:58:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -479,7 +481,7 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: nsivabalan
body: @lokeshj1703 : can you follow up here. 
created_at: 2025-03-08T00:00:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -479,7 +481,7 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: nsivabalan
body: This may not be immediate necessity since we don't have support to reading MDT using new File group reader. But when @linliu-code gets to replacing it, we might need the support. So, we can take a call to delay the support for couple of weeks. 

created_at: 2025-03-08T00:02:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -527,19 +529,39 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
-      Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
-      return tablePath.get().toString();
+      return getTablePath(jobConf, inputPath);
     }
   }
 
+  private static String getTablePath(JobConf jobConf, Path inputPath) throws IOException {
+    FileSystem fs = inputPath.getFileSystem(jobConf);
+    HoodieStorage storage = new HoodieHadoopStorage(fs);
+    Option<StoragePath> tablePath = TablePathUtils.getTablePath(storage, convertToStoragePath(inputPath));
+    return tablePath.get().toString();
+  }
+
   /**
    * `schema.on.read` and skip merge not implemented
    */
-  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) {
-    return jobConf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue())
-        && !jobConf.getBoolean(HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.key(), HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue())
-        && !(split instanceof BootstrapBaseFileSplit);
+  public static boolean shouldUseFilegroupReader(final JobConf jobConf, final InputSplit split) throws IOException {
+    if (split instanceof FileSplit || split instanceof RealtimeSplit) {
+      HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(getStorageConf(jobConf)).setBasePath(getTableBasePath(split, jobConf)).build();
path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
user: login: nsivabalan
body: we will file a ticket for review once we have the follow up patch. 
created_at: 2025-03-08T00:04:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: lokeshj1703
body: This was failing due to MDT compaction scheduling with table version 6. This was not related to table version 8. 
If we delete compaction completed instant in DT but the corresponding deltacommit still exists in MDT. The MDT writer goes and tries to create compaction instant in MDT using time - one minus the inflight instant and fails check in this PR.

DT
dc10
dc20.inf
MDT
dc10
dc20
dc19 - compaction fails since dc20 already exists.

This check wouldn't have failed with older compaction scheduling logic of MDT where compaction instant is created using suffix
created_at: 2025-03-10T06:18:04Z
author_association: COLLABORATOR
diff_hunk: @@ -109,9 +109,6 @@ public Option<HoodieCompactionPlan> execute() {
           .getWriteTimeline().filterCompletedAndCompactionInstants().getInstantsAsStream()
           .filter(instant -> compareTimestamps(instant.requestedTime(), GREATER_THAN_OR_EQUALS, instantTime))
           .collect(Collectors.toList());
-      ValidationUtils.checkArgument(conflictingInstants.isEmpty(),
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
user: login: lokeshj1703
body: We can add this check for table version 6 only after MDT backward compatible changes are in.
created_at: 2025-03-10T06:19:31Z
author_association: COLLABORATOR
diff_hunk: @@ -632,6 +637,7 @@ static boolean validateConfigVersion(ConfigProperty<?> configProperty, HoodieTab
     // validate that the table version is greater than or equal to the config version
     HoodieTableVersion firstVersion = HoodieTableVersion.fromReleaseVersion(configProperty.getSinceVersion().get());
     boolean valid = tableVersion.greaterThan(firstVersion) || tableVersion.equals(firstVersion);
+    valid = valid || CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES.contains(configProperty.key());
path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
user: login: lokeshj1703
body: These configs are added with since version 1.0.0 but are still required for table version 6. Therefore I had to add a check here so that these configs are not dropped for tables with version 6.
created_at: 2025-03-10T06:21:56Z
author_association: COLLABORATOR
diff_hunk: @@ -479,7 +481,7 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA
           continue;
         }
         if (logBlock.getBlockType() != COMMAND_BLOCK) {
-          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT)) {
+          if (this.tableVersion.lesserThan(HoodieTableVersion.EIGHT) && !allowInflightInstants) {
path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
user: login: lokeshj1703
body: @yihua This was required for RLI. RLI directly calls AbstractHoodieLogRecordScanner via `org.apache.hudi.metadata.HoodieTableMetadataUtil#getRevivedAndDeletedKeysFromMergedLogs`.
created_at: 2025-03-10T06:27:05Z
author_association: COLLABORATOR
diff_hunk: @@ -89,4 +95,18 @@ public class HoodieReaderConfig extends HoodieConfig {
       "hoodie.write.record.merge.custom.implementation.classes";
   public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
       "hoodie.datasource.write.record.merger.impls";
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, HoodieConfig config) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT) && config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED);
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Map<String, String> parameters) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && Boolean.parseBoolean(parameters.getOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue().toString()));
+  }
+
+  public static boolean isFileGroupReaderEnabled(HoodieTableVersion tableVersion, Configuration conf) {
+    return tableVersion.greaterThanOrEquals(HoodieTableVersion.EIGHT)
+        && conf.getBoolean(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), HoodieReaderConfig.FILE_GROUP_READER_ENABLED.defaultValue());
+  }
path: hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
user: login: linliu-code
body: `Configuration conf` introduce hadoop dependencies into hudi-common. Should we use StorageConfiguration instead?
created_at: 2025-03-12T18:17:36Z
author_association: COLLABORATOR

PR comments:
user: login: lokeshj1703
created_at: 2025-02-28T13:26:36Z
body: @nsivabalan I have fixed the usage of configs so that default value is false for table version less than 8. Also added tests. PTAL.
author_association: COLLABORATOR
user: login: nsivabalan
created_at: 2025-03-01T04:20:35Z
body: @lokeshj1703 : can you also follow up on CI failures. 
author_association: CONTRIBUTOR
user: login: lokeshj1703
created_at: 2025-03-01T11:14:39Z
body: @nsivabalan I have addressed the review comments. PTAL.
author_association: COLLABORATOR
user: login: nsivabalan
created_at: 2025-03-04T04:32:00Z
body: @lokeshj1703 : once CI succeeds, we can go ahead and land the patch. 
author_association: CONTRIBUTOR
user: login: hudi-bot
created_at: 2025-03-05T06:14:11Z
body: <!--
Meta data
{
  "version" : 1,
  "metaDataEntries" : [ {
    "hash" : "e566d06850fc39652a93070751432a245f64b91f",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3946",
    "triggerID" : "e566d06850fc39652a93070751432a245f64b91f",
    "triggerType" : "PUSH"
  }, {
    "hash" : "2bed89847bf0a3750dd852f33350715a39d320fb",
    "status" : "UNKNOWN",
    "url" : "TBD",
    "triggerID" : "2bed89847bf0a3750dd852f33350715a39d320fb",
    "triggerType" : "PUSH"
  }, {
    "hash" : "274e042749a92bc8b29d5bee085ea17ebd682778",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3947",
    "triggerID" : "274e042749a92bc8b29d5bee085ea17ebd682778",
    "triggerType" : "PUSH"
  }, {
    "hash" : "b83612530ce251d99fbb62129f957cc608f566d7",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3965",
    "triggerID" : "b83612530ce251d99fbb62129f957cc608f566d7",
    "triggerType" : "PUSH"
  }, {
    "hash" : "a067c3dba47e975b1369b19d5f3684b0b86c91c0",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3968",
    "triggerID" : "a067c3dba47e975b1369b19d5f3684b0b86c91c0",
    "triggerType" : "PUSH"
  }, {
    "hash" : "e87c7ebbb9dbc3af70efda1ed2c83a557de72f2b",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3989",
    "triggerID" : "e87c7ebbb9dbc3af70efda1ed2c83a557de72f2b",
    "triggerType" : "PUSH"
  }, {
    "hash" : "800de68bf283b7c3b5bf927caa1bdbd03f6d4ced",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3990",
    "triggerID" : "800de68bf283b7c3b5bf927caa1bdbd03f6d4ced",
    "triggerType" : "PUSH"
  }, {
    "hash" : "58f3d71e0e20b52bc389babfa2ab39fc8f8245a6",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4010",
    "triggerID" : "58f3d71e0e20b52bc389babfa2ab39fc8f8245a6",
    "triggerType" : "PUSH"
  }, {
    "hash" : "19585fd355c82488490523192667aaa5cac79f01",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4011",
    "triggerID" : "19585fd355c82488490523192667aaa5cac79f01",
    "triggerType" : "PUSH"
  }, {
    "hash" : "d6ff9f3c4eaf069110a37c03cb0a25f004e51cf5",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4012",
    "triggerID" : "d6ff9f3c4eaf069110a37c03cb0a25f004e51cf5",
    "triggerType" : "PUSH"
  }, {
    "hash" : "a102f8eac1781e53a6d6d20d9ca804d3fd65ac53",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4015",
    "triggerID" : "a102f8eac1781e53a6d6d20d9ca804d3fd65ac53",
    "triggerType" : "PUSH"
  }, {
    "hash" : "c78d63235cf255ed685865aa294a8603630108f1",
    "status" : "FAILURE",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4064",
    "triggerID" : "c78d63235cf255ed685865aa294a8603630108f1",
    "triggerType" : "PUSH"
  } ]
}-->
## CI report:

* 2bed89847bf0a3750dd852f33350715a39d320fb UNKNOWN
* c78d63235cf255ed685865aa294a8603630108f1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4064) 

<details>
<summary>Bot commands</summary>
  @hudi-bot supports the following commands:

 - `@hudi-bot run azure` re-run the last Azure build
</details>
author_association: COLLABORATOR
user: login: lokeshj1703
created_at: 2025-03-05T11:35:45Z
body: @codope Azure CI passed
<img width="952" alt="image" src="https://github.com/user-attachments/assets/9b634886-8f74-45aa-a54e-947bb21c948d" />

author_association: COLLABORATOR
user: login: nsivabalan
created_at: 2025-03-05T18:33:48Z
body: @yihua : yes, I was also in a dilemma on this. Since a 0.x reader (say 0.14.1 reader) will not be using File Group reader way of reading, we wanted to maintain parity. If not, we need to test and certify new FG reader for tbl version 6 as well. 
For eg, all the tests we did to certify new FG reader for 1.0 release, we might have to repeat it for tbl version 6 as well. 

Also, in 1.2, we will deprecate b/w compatible writer support for table version 6, since likely we will bump up the table versions by 2(we might be in table version 10). All this will go away. So, its not that we will keep carrying this baggage forever. 
Open to discuss more. let me know. 


author_association: CONTRIBUTOR