muttakin2301/ccr2000 huggingface dataset:
Code review comment: 1, `dataRecord.getOldKey()` might be `null`?  2, Could `dataRecord.getOldKey()` method name be updated?

Code under review:
private void mergeDelete(final DataRecord dataRecord, final Map<DataRecord.Key, DataRecord> dataRecords) {
        DataRecord beforeDataRecord = dataRecords.get(dataRecord.getOldKey());
        ShardingSpherePreconditions.checkState(null == beforeDataRecord
                || !IngestDataChangeType.DELETE.equals(beforeDataRecord.getType()), () -> new PipelineUnexpectedDataRecordOrderException(beforeDataRecord, dataRecord));
        if (null != beforeDataRecord && IngestDataChangeType.UPDATE.equals(beforeDataRecord.getType()) && checkUpdatedPrimaryKey(beforeDataRecord)) {
            DataRecord mergedDataRecord = new DataRecord(dataRecord.getPosition(), dataRecord.getColumnCount());
            for (int i = 0; i < dataRecord.getColumnCount(); i++) {
                boolean isUniqueKey = dataRecord.getColumn(i).isUniqueKey();
                mergedDataRecord.addColumn(new Column(dataRecord.getColumn(i).getName(),
                        isUniqueKey ? beforeDataRecord.getColumn(i).getOldValue() : beforeDataRecord.getColumn(i).getValue(), null, true, isUniqueKey));
            }
            mergedDataRecord.setTableName(dataRecord.getTableName());
            mergedDataRecord.setType(IngestDataChangeType.DELETE);
            dataRecords.remove(beforeDataRecord.getKey());
            dataRecords.put(mergedDataRecord.getKey(), mergedDataRecord);
        } else {
            dataRecords.put(dataRecord.getOldKey(), dataRecord);
        }
    }

Line commented on: DataRecord beforeDataRecord = dataRecords.get(dataRecord.getOldKey());

Revised code:
private void mergeDelete(final DataRecord dataRecord, final Map<DataRecord.Key, DataRecord> dataRecords) {
        DataRecord beforeDataRecord = dataRecords.get(dataRecord.getOldKey());
        ShardingSpherePreconditions.checkState(null == beforeDataRecord
                || !IngestDataChangeType.DELETE.equals(beforeDataRecord.getType()), () -> new PipelineUnexpectedDataRecordOrderException(beforeDataRecord, dataRecord));
        if (null != beforeDataRecord && IngestDataChangeType.UPDATE.equals(beforeDataRecord.getType()) && checkUpdatedPrimaryKey(beforeDataRecord)) {
            DataRecord mergedDataRecord = new DataRecord(dataRecord.getPosition(), dataRecord.getColumnCount());
            for (int i = 0; i < dataRecord.getColumnCount(); i++) {
                mergedDataRecord.addColumn(new Column(dataRecord.getColumn(i).getName(),
                        dataRecord.getColumn(i).isUniqueKey() ? beforeDataRecord.getColumn(i).getOldValue() : beforeDataRecord.getColumn(i).getValue(), true, dataRecord.getColumn(i).isUniqueKey()));
            }
            mergedDataRecord.setTableName(dataRecord.getTableName());
            mergedDataRecord.setType(IngestDataChangeType.DELETE);
            dataRecords.remove(beforeDataRecord.getKey());
            dataRecords.put(mergedDataRecord.getKey(), mergedDataRecord);
        } else {
            dataRecords.put(dataRecord.getOldKey(), dataRecord);
        }
    }
Code review comment: Wmpty string does not match any special characters,so don't judge

Code under review:
public String mask(final Object plainValue) {
        String result = null == plainValue ? null : String.valueOf(plainValue);
        final int index = result.indexOf(specialCharacters);
        char[] chars = result.toCharArray();
        for (int i = 0; i < chars.length; i++) {
            if (i < index) {
                chars[i] = '*';
            }
        }
        return new String(chars);
    }

Line commented on: String result = null == plainValue ? null : String.valueOf(plainValue);

Revised code:
public String mask(final Object plainValue) {
        String result = null == plainValue ? null : String.valueOf(plainValue);
        if (Strings.isNullOrEmpty(result)) {
            return result;
        }
        int index = result.indexOf(specialCharacters);
        char[] chars = result.toCharArray();
        for (int i = 0; i < index; i++) {
            chars[i] = '*';
        }
        return new String(chars);
    }
Code review comment: If we handle in backend: 1. there is a performance issue, we still need to scan  the HLL or Bitmap column 2. we need to support return binary content for HLL or Bitmap, so we shouldn't send null directly in ResultSink, we must pass a config from FE to BE.

Code under review:
private void expandStar(TableName tblName, TupleDescriptor desc) {
        for (Column col : desc.getTable().getBaseSchema()) {
            PrimitiveType type = col.getDataType();
            if (!fromInsert && (type == PrimitiveType.HLL || type == PrimitiveType.BITMAP)) {
                resultExprs.add(new NullLiteral());
            } else {
                resultExprs.add(new SlotRef(tblName, col.getName()));
            }
            colLabels.add(col.getName());
        }
    }

Line commented on: PrimitiveType type = col.getDataType();

Revised code:
private void expandStar(TableName tblName, TupleDescriptor desc) {
        for (Column col : desc.getTable().getBaseSchema()) {
            resultExprs.add(new SlotRef(tblName, col.getName()));
            colLabels.add(col.getName());
        }
    }
Code review comment: firstHeartbeat here will be null, if heartbeatsBySender is empty  I see exceptions like  ``` java.lang.NullPointerException: Cannot invoke "java.util.Map$Entry.getValue()" because "heartbeat" is null 	at org.apache.beam.runners.dataflow.worker.windmill.work.refresh.ActiveWorkRefresher.sendHeartbeatSafely(ActiveWorkRefresher.java:182) 	at org.apache.beam.runners.dataflow.worker.windmill.work.refresh.ActiveWorkRefresher.refreshActiveWork(ActiveWorkRefresher.java:149) 	at org.apache.beam.runners.dataflow.worker.windmill.work.refresh.ActiveWorkRefresher.lambda$start$0(ActiveWorkRefresher.java:94) 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) 	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 	at java.base/java.lang.Thread.run(Thread.java:1583) ```

Code under review:
private void refreshActiveWork() {
    Instant refreshDeadline = clock.get().minus(Duration.millis(activeWorkRefreshPeriodMillis));
    Map<HeartbeatSender, Heartbeats> heartbeatsBySender =
        aggregateHeartbeatsBySender(refreshDeadline);

    List<CompletableFuture<Void>> fanOutRefreshActiveWork = new ArrayList<>();

    
    
    @Nullable Map.Entry<HeartbeatSender, Heartbeats> firstHeartbeat = null;
    for (Map.Entry<HeartbeatSender, Heartbeats> heartbeat : heartbeatsBySender.entrySet()) {
      if (firstHeartbeat == null) {
        firstHeartbeat = heartbeat;
      } else {
        fanOutRefreshActiveWork.add(
            CompletableFuture.runAsync(
                () -> sendHeartbeatSafely(heartbeat), fanOutActiveWorkRefreshExecutor));
      }
    }

    sendHeartbeatSafely(firstHeartbeat);
    fanOutRefreshActiveWork.forEach(CompletableFuture::join);
  }

Line commented on: sendHeartbeatSafely(firstHeartbeat);

Revised code:
private void refreshActiveWork() {
    Instant refreshDeadline = clock.get().minus(Duration.millis(activeWorkRefreshPeriodMillis));
    Map<HeartbeatSender, Heartbeats> heartbeatsBySender =
        aggregateHeartbeatsBySender(refreshDeadline);

    List<CompletableFuture<Void>> fanOutRefreshActiveWork = new ArrayList<>();

    
    
    @Nullable Map.Entry<HeartbeatSender, Heartbeats> firstHeartbeat = null;
    for (Map.Entry<HeartbeatSender, Heartbeats> heartbeat : heartbeatsBySender.entrySet()) {
      if (firstHeartbeat == null) {
        firstHeartbeat = heartbeat;
      } else {
        fanOutRefreshActiveWork.add(
            CompletableFuture.runAsync(
                () -> sendHeartbeatSafely(heartbeat), fanOutActiveWorkRefreshExecutor));
      }
    }

    sendHeartbeatSafely(firstHeartbeat);
    fanOutRefreshActiveWork.forEach(CompletableFuture::join);
  }
Code review comment: In which case do we have dominant and recessive repositories?

Code under review:
public List<RemoteRepository> aggregateRepositories(List<RemoteRepository> dominant, List<RemoteRepository> recessive) {
        return dominant.isEmpty() ? recessive : remoteRepoManager.aggregateRepositories(repoSession, dominant, recessive, false);
    }

Line commented on: return dominant.isEmpty() ? recessive : remoteRepoManager.aggregateRepositories(repoSession, dominant, recessive, false);

Revised code:
public List<RemoteRepository> aggregateRepositories(List<RemoteRepository> dominant, List<RemoteRepository> recessive) {
        return dominant.isEmpty() ? recessive : remoteRepoManager.aggregateRepositories(repoSession, dominant, recessive, false);
    }
Code review comment: Do we need that logic if we only ever retain a single checkpoint?

Code under review:
private File discoverRetainedCheckpoint() throws Exception {
        
        File attemptDir = temporaryFolder.getRoot();
        Path checkpointDir = null;

        for (int i = 0; i <= 1000 && checkpointDir == null; i++) {
            Thread.sleep(5);
            try (Stream<Path> files = Files.walk(Paths.get(attemptDir.getPath()))) {
                checkpointDir =
                        files.filter(Files::isRegularFile)
                                .filter(path -> path.endsWith("_metadata"))
                                .map(path -> path.getParent())
                                .sorted(
                                        Comparator.comparingInt(
                                                UnalignedCheckpointStressITCase
                                                        ::getCheckpointNumberFromPath))
                                .reduce((first, second) -> second)
                                .orElse(null);
            }
        }
        if (checkpointDir == null) {
            List<Path> files =
                    Files.walk(Paths.get(attemptDir.getPath())).collect(Collectors.toList());
            throw new IllegalStateException("Failed to find _metadata file among " + files);
        }
        return checkpointDir.toFile();
    }

Line commented on: return checkpointDir.toFile();

Revised code:
private File discoverRetainedCheckpoint() throws Exception {
        
        File rootDir = temporaryFolder.getRoot();
        Path checkpointDir = null;

        for (int i = 0; i <= 1000 && checkpointDir == null; i++) {
            Thread.sleep(5);
            try (Stream<Path> files = Files.walk(Paths.get(rootDir.getPath()))) {
                checkpointDir =
                        files.filter(Files::isRegularFile)
                                .filter(path -> path.endsWith("_metadata"))
                                .map(path -> path.getParent())
                                .sorted(
                                        Comparator.comparingInt(
                                                UnalignedCheckpointStressITCase
                                                        ::getCheckpointNumberFromPath))
                                .reduce((first, second) -> second)
                                .orElse(null);
            }
        }
        if (checkpointDir == null) {
            List<Path> files =
                    Files.walk(Paths.get(rootDir.getPath())).collect(Collectors.toList());
            throw new IllegalStateException("Failed to find _metadata file among " + files);
        }
        return checkpointDir.toFile();
    }
Code review comment: maybe we should change the name of this session variable, since it is not used for strict sondistency insert anymore.

Code under review:
public void disableStrictConsistencyDmlOnce() throws DdlException {
        if (!enableStrictConsistencyDml) {
            return;
        }
        setIsSingleSetVar(true);
        VariableMgr.setVar(this,
                new SetVar(SessionVariable.ENABLE_STRICT_CONSISTENCY_DML, new StringLiteral("false")));
    }

Line commented on: new SetVar(SessionVariable.ENABLE_STRICT_CONSISTENCY_DML, new StringLiteral("false")));

Revised code:
public void disableStrictConsistencyDmlOnce() throws DdlException {
        if (!enableStrictConsistencyDml) {
            return;
        }
        setIsSingleSetVar(true);
        VariableMgr.setVar(this,
                new SetVar(SessionVariable.ENABLE_STRICT_CONSISTENCY_DML, new StringLiteral("false")));
    }
Code review comment: Add new resource class to GsonUtils.java

Code under review:
private static Resource getResourceInstance(ResourceType type, String name) throws DdlException {
        Resource resource;
        switch (type) {
            case SPARK:
                resource = new SparkResource(name);
                break;
            case ODBC_CATALOG:
                resource = new OdbcCatalogResource(name);
                break;
            case S3:
                resource = new S3Resource(name);
                break;
            case JDBC:
                resource = new JdbcResource(name);
                break;
            case HDFS:
                resource = new HdfsResource(name);
                break;
            case HMS:
                resource = new HMSResource(name);
                break;
            default:
                throw new DdlException("Unknown resource type: " + type);
        }

        return resource;
    }

Line commented on: resource = new HdfsResource(name);

Revised code:
private static Resource getResourceInstance(ResourceType type, String name) throws DdlException {
        Resource resource;
        switch (type) {
            case SPARK:
                resource = new SparkResource(name);
                break;
            case ODBC_CATALOG:
                resource = new OdbcCatalogResource(name);
                break;
            case S3:
                resource = new S3Resource(name);
                break;
            case JDBC:
                resource = new JdbcResource(name);
                break;
            case HDFS:
                resource = new HdfsResource(name);
                break;
            case HMS:
                resource = new HMSResource(name);
                break;
            default:
                throw new DdlException("Unknown resource type: " + type);
        }

        return resource;
    }
Code review comment: > Why do we need to reduce the backpressure here?  This is a commit from #13827. As far as I understand, the motivation was: 1. to have higher backpressure (.1ms vs 1ms) for **some** parts of the stream 2. to have unbalanced streams 3. reduce running time (was ~1m, now ~10s)  On average, the change decreases sleep time only by 60% (1-42/1000*10).  > I'm worried that any solution on top of it will only work with a certain minimal flow.  With 1K elements, the sink will sleep 23 times for 1ms, which I think is enough and 1K is a reasonable number of elements for this test.

Code under review:
public void invoke(Long value, Context context) throws Exception {
			int parallelism = state.lastRecordInPartitions.length;
			int partition = (int) (value % parallelism);
			long lastRecord = state.lastRecordInPartitions[partition];
			if (value < lastRecord) {
				state.numOutOfOrderness++;
				info("Out of order records current={} and last={}", value, lastRecord);
			} else if (value == lastRecord) {
				state.numDuplicates++;
				info("Duplicate record {}", value);
			} else if (lastRecord != -1) {
				long expectedValue = lastRecord + parallelism * parallelism;
				if (value != expectedValue) {
					state.numLostValues++;
					info("Lost records {}-{}", expectedValue, value);
				}
			}
			state.lastRecordInPartitions[partition] = value;
			state.numOutput++;

			if (state.completedCheckpoints < minCheckpoints) {
				
				if (random.nextInt(1000) == 42) {
					Thread.sleep(1);
				}
			}
			
		}

Line commented on: if (random.nextInt(1000) == 42) {

Revised code:
public void invoke(Long value, Context context) throws Exception {
			int parallelism = state.lastRecordInPartitions.length;
			int partition = (int) (value % parallelism);
			long lastRecord = state.lastRecordInPartitions[partition];
			if (value < lastRecord) {
				state.numOutOfOrderness++;
				info("Out of order records current={} and last={}", value, lastRecord);
			} else if (value == lastRecord) {
				state.numDuplicates++;
				info("Duplicate record {}", value);
			} else if (lastRecord != -1) {
				long expectedValue = lastRecord + parallelism * parallelism;
				if (value != expectedValue) {
					state.numLostValues++;
					info("Lost records {}-{}", expectedValue, value);
				}
			}
			state.lastRecordInPartitions[partition] = value;
			state.numOutput++;

			if (state.completedCheckpoints < minCheckpoints) {
				
				if (random.nextInt(1000) == 42) {
					Thread.sleep(1);
				}
			}
			
		}
Code review comment: > Maybe we should disallow -Xrunjdwp:transport completely in hosted?  Yes, it will prevent confusion about why debugging doesn't work at least.  > Right now we warn or fail deployment for hosted (depending on feature flag value) and warn for non-hosted for JVM options like these.   :+1:

Code under review:
public void requireThatJvmOptionsAreLogged()  throws IOException, SAXException {
        verifyLoggingOfJvmOptions(true,
                                  "options",
                                  "-Xms2G foo     bar",
                                  "foo", "bar");
        verifyLoggingOfJvmOptions(true,
                                  "options",
                                  "$(touch /tmp/hello-from-gc-options)",
                                  "$(touch", "/tmp/hello-from-gc-options)");

        verifyLoggingOfJvmOptions(false,
                                  "options",
                                  "$(touch /tmp/hello-from-gc-options)",
                                  "$(touch", "/tmp/hello-from-gc-options)");

        
        verifyLoggingOfJvmOptions(true, "options", "-Xms2G");
        verifyLoggingOfJvmOptions(true, "options", "-verbose:gc");
        verifyLoggingOfJvmOptions(true, "options", "-Djava.library.path=/opt/vespa/lib64:/home/y/lib64 -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005");
        verifyLoggingOfJvmOptions(false, "options", "-Xms2G");
    }

Line commented on: verifyLoggingOfJvmOptions(true, "options", "-Djava.library.path=/opt/vespa/lib64:/home/y/lib64 -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005");

Revised code:
public void requireThatJvmOptionsAreLogged()  throws IOException, SAXException {
        verifyLoggingOfJvmOptions(true,
                                  "options",
                                  "-Xms2G foo     bar",
                                  "foo", "bar");
        verifyLoggingOfJvmOptions(true,
                                  "options",
                                  "$(touch /tmp/hello-from-gc-options)",
                                  "$(touch", "/tmp/hello-from-gc-options)");

        verifyLoggingOfJvmOptions(true,
                                  "options",
                                  "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005",
                                  "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005");

        verifyLoggingOfJvmOptions(false,
                                  "options",
                                  "$(touch /tmp/hello-from-gc-options)",
                                  "$(touch", "/tmp/hello-from-gc-options)");

        
        verifyLoggingOfJvmOptions(true, "options", "-Xms2G");
        verifyLoggingOfJvmOptions(true, "options", "-verbose:gc");
        verifyLoggingOfJvmOptions(true, "options", "-Djava.library.path=/opt/vespa/lib64:/home/y/lib64");
        verifyLoggingOfJvmOptions(false, "options", "-Xms2G");
    }
Code review comment: it will be good to clarify that this is the client that contains the model we want to copy

Code under review:
public static void main(final String[] args) {
        

        FormTrainingClient client = new FormTrainingClientBuilder()
            .credential(new AzureKeyCredential("{key}"))
            .endpoint("https:
            .buildClient();

        String targetResourceId = "target-resource-Id";
        String targetResourceRegion = "target-resource-region";
        final String copyModelId = "copy-model-Id";

        
        final CopyAuthorization modelCopyAuthorization =
            client.getCopyAuthorization(targetResourceId, targetResourceRegion);

        
        SyncPoller<OperationResult, CustomFormModelInfo> copyPoller = client.beginCopyModel(copyModelId, modelCopyAuthorization);

        
        CustomFormModelInfo modelCopy = copyPoller.getFinalResult();

        System.out.printf("Copied model has model Id: %s, model status: %s, was created on: %s,"
                + " last updated on: %s.%n",
            modelCopy.getModelId(),
            modelCopy.getStatus(),
            modelCopy.getCreatedOn(),
            modelCopy.getLastUpdatedOn());
    }


Revised code:
public static void main(final String[] args) {
        
        FormTrainingClient sourceClient = new FormTrainingClientBuilder()
            .credential(new AzureKeyCredential("{key}"))
            .endpoint("https:
            .buildClient();

        
        FormTrainingClient targetClient = new FormTrainingClientBuilder()
            .credential(new AzureKeyCredential("{key}"))
            .endpoint("https:
            .buildClient();

        String targetResourceId = "target-resource-Id";
        String targetResourceRegion = "target-resource-region";

        
        CopyAuthorization modelCopyAuthorization = targetClient.getCopyAuthorization(targetResourceId,
            targetResourceRegion);

        
        String copyModelId = "copy-model-Id";
        
        SyncPoller<OperationResult, CustomFormModelInfo> copyPoller = sourceClient.beginCopyModel(copyModelId,
            modelCopyAuthorization);
        copyPoller.waitForCompletion();

        
        CustomFormModel copiedModel = targetClient.getCustomModel(modelCopyAuthorization.getModelId());

        System.out.printf("Copied model has model Id: %s, model status: %s, was created on: %s,"
                + " transfer completed on: %s.%n",
            copiedModel.getModelId(),
            copiedModel.getModelStatus(),
            copiedModel.getRequestedOn(),
            copiedModel.getCompletedOn());
    }
Code review comment: Leaving comment to track reverting this TODO before merge.

Code under review:
private void resetSessionContainerIfNeeded(DatabaseAccount databaseAccount) {
        boolean isRegionScopingOfSessionTokensPossible = this.isRegionScopingOfSessionTokensPossible(databaseAccount, this.useMultipleWriteLocations, this.isRegionScopedSessionCapturingEnabledOnClientOrSystemConfig);

        

            this.sessionContainer = new RegionScopedSessionContainer(this.serviceEndpoint.getHost(), this.sessionCapturingDisabled, this.globalEndpointManager);
            this.diagnosticsClientConfig.withRegionScopedSessionContainerOptions((RegionScopedSessionContainer) this.sessionContainer);

    }


Revised code:
private void resetSessionContainerIfNeeded(DatabaseAccount databaseAccount) {
        boolean isRegionScopingOfSessionTokensPossible = this.isRegionScopingOfSessionTokensPossible(databaseAccount, this.useMultipleWriteLocations, this.isRegionScopedSessionCapturingEnabledOnClientOrSystemConfig);

        if (isRegionScopingOfSessionTokensPossible) {
            this.sessionContainer = new RegionScopedSessionContainer(this.serviceEndpoint.getHost(), this.sessionCapturingDisabled, this.globalEndpointManager);
            this.diagnosticsClientConfig.withRegionScopedSessionContainerOptions((RegionScopedSessionContainer) this.sessionContainer);
        }
    }
Code review comment: ```suggestion         Consumer<MockEventContext> eventProcessor = mockEventContext -> eventRaised(); ```

Code under review:
public MockEventProcessorTest(MockEventProcessorPerfOptions perfStressOptions) {
        super(perfStressOptions);
        Consumer<MockErrorContext> errorprocessor = mockErrorContext -> errorRaised();

        Consumer<MockEventContext> eventPrcessor = mockEventContext -> eventRaised();

        Duration errorAfter = perfStressOptions.getErrorAfterInSeconds() > 0
            ? Duration.ofSeconds(perfStressOptions.getErrorAfterInSeconds()) : null;

        mockEventProcessor = new MockEventProcessor(2, perfStressOptions.getMaxEventsPerSecond(), errorAfter,
            errorprocessor, eventPrcessor);
    }

Line commented on: Consumer<MockEventContext> eventPrcessor = mockEventContext -> eventRaised();

Revised code:
public MockEventProcessorTest(MockEventProcessorPerfOptions perfStressOptions) {
        super(perfStressOptions);
        Consumer<MockErrorContext> errorProcessor = mockErrorContext -> errorRaised(mockErrorContext.getThrowable());

        Consumer<MockEventContext> eventProcessor = mockEventContext -> eventRaised();

        Duration errorAfter = perfStressOptions.getErrorAfterInSeconds() > 0
            ? Duration.ofSeconds(perfStressOptions.getErrorAfterInSeconds()) : null;

        mockEventProcessor = new MockEventProcessor(perfStressOptions.getPartitions(), perfStressOptions.getMaxEventsPerSecond(), errorAfter,
            errorProcessor, eventProcessor);
    }
Code review comment: I thought hte check on tracker existence was sufficient but added window check too.

Code under review:
private Progress getProgressFromWindowObservingTruncate(double elementCompleted) {
    synchronized (splitLock) {
      if (currentWindow != null) {
        return scaleProgress(
            Progress.from(elementCompleted, 1 - elementCompleted),
            windowCurrentIndex,
            windowStopIndex);
      }
    }
    return null;
  }

Line commented on: Progress.from(elementCompleted, 1 - elementCompleted),

Revised code:
private Progress getProgressFromWindowObservingTruncate(double elementCompleted) {
    synchronized (splitLock) {
      if (currentWindow != null) {
        return scaleProgress(
            Progress.from(elementCompleted, 1 - elementCompleted),
            windowCurrentIndex,
            windowStopIndex);
      }
    }
    return null;
  }
Code review comment: you can use Strings.isNullOrEmpty() in guava

Code under review:
protected void doRefresh() {
        String metadataLocation = null;
        try {
            Table table = metaClients.run(client -> client.getTable(database, tableName));
            validateTableIsIceberg(table, fullName);

            metadataLocation = table.getParameters().get(METADATA_LOCATION_PROP);
            if (null == metadataLocation || metadataLocation.length() == 0) {
                throw new NotFoundException("Property 'metadata_location' can not be found in table %s.%s. " +
                        "Probably this table is created by Spark2, which is not supported.", database, tableName);
            }

        } catch (NoSuchObjectException e) {
            if (currentMetadataLocation() != null) {
                throw new NoSuchTableException("No such table: %s.%s", database, tableName);
            }

        } catch (TException e) {
            String errMsg = String.format("Failed to get table info from metastore %s.%s", database, tableName);
            throw new RuntimeException(errMsg, e);

        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Interrupted during refresh", e);
        }

        refreshFromMetadataLocation(metadataLocation, metadataRefreshMaxRetries);
    }

Line commented on: if (null == metadataLocation || metadataLocation.length() == 0) {

Revised code:
protected void doRefresh() {
        String metadataLocation = null;
        try {
            Table table = metaClients.run(client -> client.getTable(database, tableName));
            validateTableIsIceberg(table, fullName);

            metadataLocation = table.getParameters().get(METADATA_LOCATION_PROP);
            if (Strings.isNullOrEmpty(metadataLocation)) {
                throw new NotFoundException("Property 'metadata_location' can not be found in table %s.%s. " +
                        "Probably this table is created by Spark2, which is not supported.", database, tableName);
            }

        } catch (NoSuchObjectException e) {
            if (currentMetadataLocation() != null) {
                throw new NoSuchTableException("No such table: %s.%s", database, tableName);
            }

        } catch (TException e) {
            String errMsg = String.format("Failed to get table info from metastore %s.%s", database, tableName);
            throw new RuntimeException(errMsg, e);

        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Interrupted during refresh", e);
        }

        refreshFromMetadataLocation(metadataLocation, metadataRefreshMaxRetries);
    }
Code review comment: This doesn't work in JDK 8 AFAIK

Code under review:
private void createBuildFile() throws IOException {
        Path buildFile = projectDir.resolve("build.gradle");
        
        try (InputStream is = getClass().getResourceAsStream("/TaskDependenciesFunctionalTest.gradle")) {
            String txt = new String(is.readAllBytes());
            Files.writeString(buildFile, txt);
        }
    }

Line commented on: String txt = new String(is.readAllBytes());

Revised code:
private void createBuildFile() throws IOException {
        Path buildFile = projectDir.resolve("build.gradle");

        try (InputStream is = getClass().getResourceAsStream("/TaskDependenciesFunctionalTest.gradle");
                BufferedInputStream bis = new BufferedInputStream(is)) {
            Files.copy(bis, buildFile);
        }
    }
Code review comment: Maybe it's easier to discuss having a code change. I published a hotfix PR: https://github.com/apache/flink/pull/15046

Code under review:
public static boolean isStateImmutableInStateBackend(KeyedStateBackend<?> keyedStateBackend) {
        
        
        KeyedStateBackend<?> rootKeyedStateBackend =
                keyedStateBackend instanceof DelegateKeyedStateBackend
                        ? ((DelegateKeyedStateBackend<?>) keyedStateBackend)
                                .getDelegatedKeyedStateBackend()
                        : keyedStateBackend;

        boolean isRocksDbState =
                ROCKSDB_KEYED_STATE_BACKEND.equals(
                        rootKeyedStateBackend.getClass().getCanonicalName());
        
        
        boolean isHeapTimer =
                keyedStateBackend.requiresLegacySynchronousTimerSnapshots(
                        CheckpointType.CHECKPOINT);

        return isRocksDbState && !isHeapTimer;
    }

Line commented on: rootKeyedStateBackend.getClass().getCanonicalName());

Revised code:
public static boolean isStateImmutableInStateBackend(KeyedStateBackend<?> keyedStateBackend) {
        
        return keyedStateBackend.isStateImmutableInStateBackend(CheckpointType.CHECKPOINT);
    }
Code review comment: Wondering if this `ChallengeData` can be cached and singleton.

Code under review:
public Uni<ChallengeData> getChallenge(RoutingContext context, DefaultTenantConfigResolver resolver) {
        String bearerToken = extractBearerToken(context);

        if (bearerToken == null) {
            return Uni.createFrom().item(new ChallengeData(HttpResponseStatus.UNAUTHORIZED.code(), null, null));
        }

        return Uni.createFrom().item(new ChallengeData(HttpResponseStatus.FORBIDDEN.code(), null, null));
    }

Line commented on: return Uni.createFrom().item(new ChallengeData(HttpResponseStatus.UNAUTHORIZED.code(), null, null));

Revised code:
public Uni<ChallengeData> getChallenge(RoutingContext context, DefaultTenantConfigResolver resolver) {
        String bearerToken = extractBearerToken(context);

        if (bearerToken == null) {
            return Uni.createFrom().item(UNAUTHORIZED_CHALLENGE);
        }

        return Uni.createFrom().item(FORBIDDEN_CHALLENGE);
    }
Code review comment: I think using explicit try/catch is a lot easier to read. try {    some_code()    fail(); } catch (IllegalArgumentException e) {    assertXX }

Code under review:
public void requireThatInvalidJvmGcOptionsFailDeployment() throws IOException, SAXException {
        expectedException.expect(IllegalArgumentException.class);
        expectedException.expectMessage(containsString("Invalid JVM GC options from services.xml: bar,foo"));
        buildModelWithJvmOptions(new TestProperties().setHostedVespa(true).failDeploymentWithInvalidJvmOptions(true),
                                    new TestLogger(),
                                    "gc-options",
                                    "-XX:+ParallelGCThreads=8 foo     bar");
    }

Line commented on: expectedException.expect(IllegalArgumentException.class);

Revised code:
public void requireThatInvalidJvmGcOptionsFailDeployment() throws IOException, SAXException {
        expectedException.expect(IllegalArgumentException.class);
        expectedException.expectMessage(containsString("Invalid JVM GC options from services.xml: bar,foo"));
        buildModelWithJvmOptions(new TestProperties().setHostedVespa(true).failDeploymentWithInvalidJvmOptions(true),
                                    new TestLogger(),
                                    "gc-options",
                                    "-XX:+ParallelGCThreads=8 foo     bar");
    }
Code review comment: Why does this function return 0 when one of these two is SYSTEM?

Code under review:
public int compareTo(@NotNull PrivilegeEntry o) {
            
            if (this.object == null || o.object == null) {
                return 0;
            }
            return this.object.compareTo(o.object);
        }

Line commented on: if (this.object == null || o.object == null) {

Revised code:
public int compareTo(@NotNull PrivilegeEntry o) {
            
            if (this.object == null && o.object == null) {
                return 0;
            } else if (this.object == null) {
                return -1;
            } else if (o.object == null) {
                return 1;
            }
            return this.object.compareTo(o.object);
        }
Code review comment: how about adding a unit test case for hudi?

Code under review:
private boolean containsMaterializedColumn(LogicalScanOperator scanOperator, Set<ColumnRefOperator> scanColumns) {
        if (scanOperator instanceof LogicalHiveScanOperator || scanOperator instanceof LogicalHudiScanOperator) {
            return scanColumns.size() != 0 && !scanOperator.getPartitionColumns().containsAll(
                    scanColumns.stream().map(ColumnRefOperator::getName).collect(Collectors.toList()));
        }
        return scanColumns.size() != 0;
    }

Line commented on: if (scanOperator instanceof LogicalHiveScanOperator || scanOperator instanceof LogicalHudiScanOperator) {

Revised code:
private boolean containsMaterializedColumn(LogicalScanOperator scanOperator, Set<ColumnRefOperator> scanColumns) {
        return scanColumns.size() != 0 && !scanOperator.getPartitionColumns().containsAll(
                scanColumns.stream().map(ColumnRefOperator::getName).collect(Collectors.toList()));
    }
Code review comment: I don't think so, but we can definitely add the test directories after main source sets.

Code under review:
public Set<File> combinedOutputSourceDirs() {
        Set<File> sourcesDirs = new HashSet<>();
        sourcesDirs.addAll(getSourceSets().getByName(SourceSet.MAIN_SOURCE_SET_NAME).getOutput().getClassesDirs().getFiles());
        sourcesDirs.addAll(getSourceSets().getByName(SourceSet.TEST_SOURCE_SET_NAME).getOutput().getClassesDirs().getFiles());
        return sourcesDirs;
    }

Line commented on: Set<File> sourcesDirs = new HashSet<>();

Revised code:
public Set<File> combinedOutputSourceDirs() {
        Set<File> sourcesDirs = new LinkedHashSet<>();
        sourcesDirs.addAll(getSourceSets().getByName(SourceSet.MAIN_SOURCE_SET_NAME).getOutput().getClassesDirs().getFiles());
        sourcesDirs.addAll(getSourceSets().getByName(SourceSet.TEST_SOURCE_SET_NAME).getOutput().getClassesDirs().getFiles());
        return sourcesDirs;
    }
Code review comment: Good catch. Yes, if an external table provider implements the interface, then it will get loaded here, I haven't thought about it. I can think of multiple ways to handle it. One is, as you suggested, to have a separate interface, another is to introduce something like a "@DoNotAutoLoad" annotation. I think it can be addressed separately.

Code under review:
public Schema create(SchemaPlus parentSchema, String name, Map<String, Object> operand) {
      return this;
    }

Line commented on: return this;

Revised code:
public Schema create(SchemaPlus parentSchema, String name, Map<String, Object> operand) {
      return this;
    }
Code review comment: Sorry I missed this but I think we need something better here.  Because the exception is caught globally not only for the close() now and it could happen in the code that is in the try. So I would rather have an IllegalStateException with a proper message and the cause.

Code under review:
private List<String> getChangeLogs(LiquibaseMongodbBuildTimeConfig liquibaseBuildConfig) {
        ChangeLogParameters changeLogParameters = new ChangeLogParameters();

        ChangeLogParserFactory changeLogParserFactory = ChangeLogParserFactory.getInstance();

        try (var classLoaderResourceAccessor = new ClassLoaderResourceAccessor(
                Thread.currentThread().getContextClassLoader())) {

            Set<String> resources = new LinkedHashSet<>(
                    findAllChangeLogFiles(liquibaseBuildConfig.changeLog, changeLogParserFactory,
                            classLoaderResourceAccessor, changeLogParameters));

            LOGGER.debugf("Liquibase changeLogs: %s", resources);

            return new ArrayList<>(resources);

        } catch (Exception ex) {
            
            throw new AssertionError(ex);
        }
    }

Line commented on: throw new AssertionError(ex);

Revised code:
private List<String> getChangeLogs(LiquibaseMongodbBuildTimeConfig liquibaseBuildConfig) {
        ChangeLogParameters changeLogParameters = new ChangeLogParameters();

        ChangeLogParserFactory changeLogParserFactory = ChangeLogParserFactory.getInstance();

        try (var classLoaderResourceAccessor = new ClassLoaderResourceAccessor(
                Thread.currentThread().getContextClassLoader())) {

            Set<String> resources = new LinkedHashSet<>(
                    findAllChangeLogFiles(liquibaseBuildConfig.changeLog, changeLogParserFactory,
                            classLoaderResourceAccessor, changeLogParameters));

            LOGGER.debugf("Liquibase changeLogs: %s", resources);

            return new ArrayList<>(resources);

        } catch (Exception ex) {
            
            throw new IllegalStateException(
                    "Error while loading the liquibase changelogs: %s".formatted(ex.getMessage()), ex);
        }
    }
Code review comment: https://github.com/reactor/reactor-core/blob/6058a391f614de6213fb85970272fc5b342bd181/reactor-core/src/main/java/reactor/util/concurrent/Queues.java#L88  Looks like that'd be 256 by default. Pretty high.  We limit number of buffers. Shouldn't max concurrency == numBuffers ? Or maxConcurrency be <= numbBuffers with numBuffers being default ?

Code under review:
public static ParallelTransferOptions populateAndApplyDefaults(ParallelTransferOptions other) {
        other = other == null ? new ParallelTransferOptions(null, null, null) : other;
        return new ParallelTransferOptions(
            other.getBlockSize() == null ? Integer.valueOf(BlobAsyncClient.BLOB_DEFAULT_UPLOAD_BLOCK_SIZE)
                : other.getBlockSize(),
            other.getNumBuffers() == null ? Integer.valueOf(BlobAsyncClient.BLOB_DEFAULT_NUMBER_OF_BUFFERS)
                : other.getNumBuffers(),
            other.getProgressReceiver(),
            other.getMaxSingleUploadSize() == null ? Integer.valueOf(BlockBlobAsyncClient.MAX_UPLOAD_BLOB_BYTES)
                : other.getMaxSingleUploadSize(),
            
            other.getMaxConcurrency() == null ? Integer.valueOf(Queues.SMALL_BUFFER_SIZE) : other.getMaxConcurrency());
    }

Line commented on: other.getMaxConcurrency() == null ? Integer.valueOf(Queues.SMALL_BUFFER_SIZE) : other.getMaxConcurrency());

Revised code:
public static ParallelTransferOptions populateAndApplyDefaults(ParallelTransferOptions other) {
        other = other == null ? new ParallelTransferOptions(null, null, null) : other;
        return new ParallelTransferOptions(
            other.getBlockSize() == null ? Integer.valueOf(BlobAsyncClient.BLOB_DEFAULT_UPLOAD_BLOCK_SIZE)
                : other.getBlockSize(),
            other.getMaxConcurrency() == null ? Integer.valueOf(BlobAsyncClient.BLOB_DEFAULT_NUMBER_OF_BUFFERS)
                : other.getMaxConcurrency(),
            other.getProgressReceiver(),
            other.getMaxSingleUploadSize() == null ? Integer.valueOf(BlockBlobAsyncClient.MAX_UPLOAD_BLOB_BYTES)
                : other.getMaxSingleUploadSize());
    }
Code review comment: Int compare date shoule cast int， because int are 4byte， date is 16byte。 int is more chance to SIMD

Code under review:
public static boolean canCompareDate(PrimitiveType t1, PrimitiveType t2) {
        if (t1 == PrimitiveType.DATE) {
            if (t2 == PrimitiveType.DATE || t2.isStringType() || t2.isIntegerType()) {
                return true;
            }
            return false;
        } else if (t2 == PrimitiveType.DATE) {
            if (t1.isStringType() || t1.isIntegerType()) {
                return true;
            }
            return false;
        } else {
            return false;
        }
    }

Line commented on: if (t2 == PrimitiveType.DATE || t2.isStringType() || t2.isIntegerType()) {

Revised code:
public static boolean canCompareDate(PrimitiveType t1, PrimitiveType t2) {
        return (t1 == PrimitiveType.DATE && t2 == PrimitiveType.DATE);
    }
Code review comment: `config.getOptionalValue()` throws `IllegalArgumentException` if the property cannot be converted to the specified type. It was a bug in our integration - SM Config extension catches the exception as well: https://github.com/smallrye/smallrye-config/blob/master/implementation/src/main/java/io/smallrye/config/inject/ConfigExtension.java#L93-L104

Code under review:
public void validateConfigProperties(Map<String, Set<String>> properties) {
        Config config = ConfigProviderResolver.instance().getConfig();
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        if (cl == null) {
            ConfigDeploymentTemplate.class.getClassLoader();
        }
        for (Entry<String, Set<String>> entry : properties.entrySet()) {
            Set<String> propertyTypes = entry.getValue();
            for (String propertyType : propertyTypes) {
                Class<?> propertyClass = load(propertyType, cl);
                
                if (propertyClass.isArray() || propertyClass.getTypeParameters().length > 0) {
                    propertyClass = String.class;
                }
                try {
                    if (!config.getOptionalValue(entry.getKey(), propertyClass).isPresent()) {
                        throw new DeploymentException(
                                "No config value of type " + entry.getValue() + " exists for: " + entry.getKey());
                    }
                } catch (IllegalArgumentException e) {
                    throw new DeploymentException(e);
                }
            }
        }
    }

Line commented on: throw new DeploymentException(e);

Revised code:
public void validateConfigProperties(Map<String, Set<String>> properties) {
        Config config = ConfigProviderResolver.instance().getConfig();
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        if (cl == null) {
            ConfigDeploymentTemplate.class.getClassLoader();
        }
        for (Entry<String, Set<String>> entry : properties.entrySet()) {
            Set<String> propertyTypes = entry.getValue();
            for (String propertyType : propertyTypes) {
                Class<?> propertyClass = load(propertyType, cl);
                
                if (propertyClass.isArray() || propertyClass.getTypeParameters().length > 0) {
                    propertyClass = String.class;
                }
                try {
                    if (!config.getOptionalValue(entry.getKey(), propertyClass).isPresent()) {
                        throw new DeploymentException(
                                "No config value of type " + entry.getValue() + " exists for: " + entry.getKey());
                    }
                } catch (IllegalArgumentException e) {
                    throw new DeploymentException(e);
                }
            }
        }
    }
Code review comment: updated to do exact match first, then relaxed match

Code under review:
public static String defaultApiVersion(String id, Provider provider) {
        if (id == null || provider == null) {
            return null;
        }
        ResourceId resourceId = ResourceId.fromString(id);
        String resourceTypeWithoutNamespace = getFullResourceTypeWithoutNamespace(resourceId);
        String fullResourceType = resourceId.fullResourceType();
        for (ProviderResourceType prt : provider.resourceTypes()) {
            
            
            
            if (prt.resourceType().equalsIgnoreCase(resourceTypeWithoutNamespace) 
                    
                    || fullResourceType.contains(prt.resourceType())) {
                return prt.defaultApiVersion() == null ? prt.apiVersions().get(0) : prt.defaultApiVersion();
            }
        }
        ResourceId parent = resourceId.parent();
        if (parent != null && !CoreUtils.isNullOrEmpty(parent.id())) {
            return defaultApiVersion(parent.id(), provider);
        } else {
            
            return provider.resourceTypes().get(0).apiVersions().get(0);
        }
    }

Line commented on: if (prt.resourceType().equalsIgnoreCase(resourceTypeWithoutNamespace)

Revised code:
public static String defaultApiVersion(String id, Provider provider) {
        if (id == null || provider == null) {
            return null;
        }
        ResourceId resourceId = ResourceId.fromString(id);
        String resourceTypeWithoutNamespace = getFullResourceTypeWithoutNamespace(resourceId);
        for (ProviderResourceType prt : provider.resourceTypes()) {
            
            
            if (prt.resourceType().equalsIgnoreCase(resourceTypeWithoutNamespace)) {
                return prt.defaultApiVersion() == null ? prt.apiVersions().get(0) : prt.defaultApiVersion();
            }
        }

        ResourceId parent = resourceId.parent();
        if (parent != null && !CoreUtils.isNullOrEmpty(parent.id())) {
            return defaultApiVersion(parent.id(), provider);
        } else {
            
            return provider.resourceTypes().get(0).apiVersions().get(0);
        }
    }