muttakin2301/ccr2000 huggingface dataset:
Code review comment: > Previously desiredState = "FAILED" and newState = "CANCELLING" would return true. Now they return false.  That's intended. As a side-effect it basically solves the failing cancel problem we've discussed previously.  > How about, we use this:  TBH I find these early returns super hard to read. Usually I'd try to avoid them unless there the nesting would be too deep / we have a really long method that can't be easily broken into smaller fragment.s

Code under review:
boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionStateTransition) {
        if (taskExecutionStateTransition.getExecutionState() != ExecutionState.FAILED) {
            return getExecutionGraph().updateState(taskExecutionStateTransition);
        }

        
        
        
        
        Optional<ExecutionVertexID> idOpt =
                executionGraph.findExecutionVertexId(taskExecutionStateTransition.getID());
        final boolean successfulUpdate =
                getExecutionGraph().updateState(taskExecutionStateTransition);
        if (!successfulUpdate) {
            return false;
        }

        Throwable cause = extractErrorOrUseDefault(taskExecutionStateTransition);

        checkState(idOpt.isPresent());
        ExecutionVertexID id = idOpt.get();

        if (getNonEmptyExecution(id).getFailureInfo().isPresent()) {
            failureCollection.add(new LocalFailure(cause, id));
        }
        onFailure(cause);
        return true;
    }

Line commented on: return true;

Revised code:
boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionStateTransition) {
        
        final Optional<AccessExecution> maybeExecution =
                executionGraph.findExecution(taskExecutionStateTransition.getID());
        final Optional<String> maybeTaskName =
                executionGraph.findVertexWithAttempt(taskExecutionStateTransition.getID());

        final ExecutionState desiredState = taskExecutionStateTransition.getExecutionState();
        boolean successfulUpdate = getExecutionGraph().updateState(taskExecutionStateTransition);
        if (successfulUpdate && desiredState == ExecutionState.FAILED) {
            final AccessExecution execution =
                    maybeExecution.orElseThrow(NoSuchElementException::new);
            final String taskName = maybeTaskName.orElseThrow(NoSuchElementException::new);
            final ExecutionState currentState = execution.getState();
            if (currentState == desiredState) {
                failureCollection.add(ExceptionHistoryEntry.create(execution, taskName));
                onFailure(
                        ErrorInfo.handleMissingThrowable(
                                taskExecutionStateTransition.getError(userCodeClassLoader)));
            }
        }
        return successfulUpdate;
    }
Code review comment: adding @yrodiere for awareness and opinions as well..

Code under review:
public void testMetrics() {
        assertEquals(0L, getCounterValueOrNull("hibernate.query.executions",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        assertEquals(0L, getCounterValueOrNull("hibernate.entities.inserts",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        assertEquals(0L, getCounterValueOrNull("hibernate.cache.query.requests",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME),
                new Tag("result", "miss")));
        Arc.container().requestContext().activate();
        try {
            DummyEntity entity = new DummyEntity();
            entity.number = 12345L;
            em.persist(entity);
            em.flush();
            em.createQuery("from DummyEntity e").getResultList();
            assertEquals(1L, getCounterValueOrNull("hibernate.query.executions",
                    new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
            assertEquals(1L, getCounterValueOrNull("hibernate.entities.inserts",
                    new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        } finally {
            Arc.container().requestContext().terminate();
        }
    }

Line commented on: assertEquals(0L, getCounterValueOrNull("hibernate.query.executions",

Revised code:
public void testMetrics() {
        assertEquals(0L, getCounterValueOrNull("hibernate.query.executions",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        assertEquals(0L, getCounterValueOrNull("hibernate.entities.inserts",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        assertEquals(0L, getCounterValueOrNull("hibernate.cache.query.requests",
                new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME),
                new Tag("result", "miss")));
        Arc.container().requestContext().activate();
        try {
            DummyEntity entity = new DummyEntity();
            entity.number = 12345L;
            em.persist(entity);
            em.flush();
            em.createQuery("from DummyEntity e").getResultList();
            assertEquals(1L, getCounterValueOrNull("hibernate.query.executions",
                    new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
            assertEquals(1L, getCounterValueOrNull("hibernate.entities.inserts",
                    new Tag("entityManagerFactory", PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME)));
        } finally {
            Arc.container().requestContext().terminate();
        }
    }
Code review comment: At present, we can't get `SessionVariable` from  `HiveMetastore.java`, so session variable is not easy to be added. I add the variable in `Config.java` instead.

Code under review:
public HivePartitionStats getTableStatistics(String dbName, String tblName) {
        org.apache.hadoop.hive.metastore.api.Table table = client.getTable(dbName, tblName);
        HiveCommonStats commonStats = toHiveCommonStats(table.getParameters());
        long totalRowNums = commonStats.getRowNums();
        if (totalRowNums == -1) {
            return HivePartitionStats.empty();
        }

        List<String> dataColumns = table.getSd().getCols().stream()
                .map(FieldSchema::getName)
                .collect(toImmutableList());
        List<ColumnStatisticsObj> statisticsObjs = client.getTableColumnStats(dbName, tblName, dataColumns);
        if (statisticsObjs.isEmpty()) {
            
            try {
                if (table.getParameters().keySet().stream().anyMatch(k -> k.startsWith("spark.sql.statistics.colStats."))) {
                    statisticsObjs = HiveMetastoreApiConverter.getColStatsFromSparkParams(table);
                }
            } catch (Exception e) {
                LOG.warn("Failed to get column stats from table [{}.{}]", dbName, tblName);
            }
        }
        Map<String, HiveColumnStats> columnStatistics =
                HiveMetastoreApiConverter.toSinglePartitionColumnStats(statisticsObjs, totalRowNums);
        return new HivePartitionStats(commonStats, columnStatistics);
    }

Line commented on: if (table.getParameters().keySet().stream().anyMatch(k -> k.startsWith("spark.sql.statistics.colStats."))) {

Revised code:
public HivePartitionStats getTableStatistics(String dbName, String tblName) {
        org.apache.hadoop.hive.metastore.api.Table table = client.getTable(dbName, tblName);
        HiveCommonStats commonStats = toHiveCommonStats(table.getParameters());
        long totalRowNums = commonStats.getRowNums();
        if (totalRowNums == -1) {
            return HivePartitionStats.empty();
        }

        List<String> dataColumns = table.getSd().getCols().stream()
                .map(FieldSchema::getName)
                .collect(toImmutableList());
        List<ColumnStatisticsObj> statisticsObjs = client.getTableColumnStats(dbName, tblName, dataColumns);
        if (statisticsObjs.isEmpty() && Config.enable_reuse_spark_column_statistics) {
            
            try {
                if (table.getParameters().keySet().stream().anyMatch(k -> k.startsWith("spark.sql.statistics.colStats."))) {
                    statisticsObjs = HiveMetastoreApiConverter.getColStatsFromSparkParams(table);
                }
            } catch (Exception e) {
                LOG.warn("Failed to get column stats from table [{}.{}]", dbName, tblName);
            }
        }
        Map<String, HiveColumnStats> columnStatistics =
                HiveMetastoreApiConverter.toSinglePartitionColumnStats(statisticsObjs, totalRowNums);
        return new HivePartitionStats(commonStats, columnStatistics);
    }
Code review comment: runningJobs is only used for replay, so no need to add a lock

Code under review:
public void cancelRunningJobs() {
        
        if (!GlobalStateMgr.isCheckpointThread()) {
            cancelJobExecutorService.submit(() -> {
                try {
                    
                    while (!GlobalStateMgr.getServingState().isReady()) {
                        try {
                            
                            
                            Thread.sleep(1000);
                        } catch (InterruptedException e) {
                            LOG.warn("InsertOverwriteJobManager runAfterCatalogReady interrupted exception.", e);
                        }
                    }
                    if (runningJobs != null) {
                        for (InsertOverwriteJob job : runningJobs) {
                            LOG.info("start to cancel unfinished insert overwrite job:{}", job.getJobId());
                            try {
                                InsertOverwriteJobRunner runner = new InsertOverwriteJobRunner(job);
                                runner.cancel();
                            } finally {
                                deregisterOverwriteJob(job.getJobId());
                            }
                        }
                        runningJobs.clear();
                    }
                } catch (Exception e) {
                    LOG.warn("cancel running jobs failed. cancel thread will exit", e);
                }
            });
        }
    }

Line commented on: if (runningJobs != null) {

Revised code:
public void cancelRunningJobs() {
        
        if (GlobalStateMgr.isCheckpointThread()) {
            return;
        }
        cancelJobExecutorService.submit(() -> {
            try {
                
                while (!GlobalStateMgr.getServingState().isReady()) {
                    try {
                        
                        
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        LOG.warn("InsertOverwriteJobManager runAfterCatalogReady interrupted exception.", e);
                    }
                }
                if (runningJobs != null) {
                    for (InsertOverwriteJob job : runningJobs) {
                        LOG.info("start to cancel unfinished insert overwrite job:{}", job.getJobId());
                        try {
                            InsertOverwriteJobRunner runner = new InsertOverwriteJobRunner(job);
                            runner.cancel();
                        } finally {
                            deregisterOverwriteJob(job.getJobId());
                        }
                    }
                    runningJobs.clear();
                }
            } catch (Exception e) {
                LOG.warn("cancel running jobs failed. cancel thread will exit", e);
            }
        });
    }
Code review comment: By reading the spec, I can't really tell what the intended behavior is.  I'll try and see if we can keep the paths of the original request and see how the tests behave

Code under review:
public Object extractParameter(ResteasyReactiveRequestContext context) {
        int index = findPathParamIndex(context.getLocatorTarget().getClassPath(), context.getLocatorTarget().getPath());
        if (index >= 0) {
            return context.getLocatorPathParam(index);
        }
        return null;
    }

Line commented on: int index = findPathParamIndex(context.getLocatorTarget().getClassPath(), context.getLocatorTarget().getPath());

Revised code:
public Object extractParameter(ResteasyReactiveRequestContext context) {
        int index = findPathParamIndex(context.getLocatorTarget().getClassPath(), context.getLocatorTarget().getPath());
        if (index >= 0) {
            return context.getLocatorPathParam(index);
        }
        return null;
    }
Code review comment: Yes, there is no case where mgr is  null.

Code under review:
public long loadGlobalFunction(DataInputStream in, long checksum) throws IOException {
        GlobalFunctionMgr mgr = GlobalFunctionMgr.read(in);
        if (mgr != null) {
            this.globalFunctionMgr = mgr;
        }
        LOG.info("finished replay global function from image");
        return checksum;
    }

Line commented on: if (mgr != null) {

Revised code:
public long loadGlobalFunction(DataInputStream in, long checksum) throws IOException {
        this.globalFunctionMgr = GlobalFunctionMgr.read(in);
        LOG.info("finished replay global function from image");
        return checksum;
    }
Code review comment: These methods seem like they are both writing the bom file. #Resolved

Code under review:
public void generate() {
        TreeSet<BomDependency> inputDependencies = scan();
        TreeSet<BomDependency> externalDependencies = resolveExternalDependencies();

        
        
        DependencyAnalyzer analyzer = new DependencyAnalyzer(inputDependencies, externalDependencies);
        TreeSet<BomDependency> outputDependencies = analyzer.analyze();
        outputDependencies.retainAll(inputDependencies);

        
        analyzer = new DependencyAnalyzer(outputDependencies, externalDependencies);
        outputDependencies = analyzer.analyze();

        
        boolean validationPassed = analyzer.validate();

        
        if(validationPassed) {
            rewriteBomFile();
            writeBom(outputDependencies);
        }
        else {
            logger.info("Validation for the BOM failed. Exiting...");
        }
    }

Line commented on: TreeSet<BomDependency> inputDependencies = scan();

Revised code:
public void generate() {
        List<BomDependency> inputDependencies = scan();
        List<BomDependency> externalDependencies = resolveExternalDependencies();

        
        
        DependencyAnalyzer analyzer = new DependencyAnalyzer(inputDependencies, externalDependencies);
        analyzer.reduce();
        Collection<BomDependency> outputDependencies = analyzer.getBomEligibleDependencies();

        
        analyzer = new DependencyAnalyzer(outputDependencies, externalDependencies);
        boolean validationFailed = analyzer.validate();
        outputDependencies = analyzer.getBomEligibleDependencies();

        
        if(!validationFailed) {
            
            rewriteExistingBomFile();
            writeBom(outputDependencies);
        }
        else {
            logger.trace("Validation for the BOM failed. Exiting...");
        }
    }
Code review comment: I don't think they can? I think even before the migration there's always a default value?

Code under review:
public Read withAttemptTimeout(Duration timeout) {
      checkArgument(timeout.isLongerThan(Duration.ZERO), "attempt timeout must be positive");
      BigtableReadOptions readOptions = getBigtableReadOptions();
      return toBuilder()
          .setBigtableReadOptions(readOptions.toBuilder().setAttemptTimeout(timeout).build())
          .build();
    }

Line commented on: checkArgument(timeout.isLongerThan(Duration.ZERO), "attempt timeout must be positive");

Revised code:
public Read withAttemptTimeout(Duration timeout) {
      checkArgument(timeout.isLongerThan(Duration.ZERO), "attempt timeout must be positive");
      BigtableReadOptions readOptions = getBigtableReadOptions();
      return toBuilder()
          .setBigtableReadOptions(readOptions.toBuilder().setAttemptTimeout(timeout).build())
          .build();
    }
Code review comment: `verifyJobGraphs` verifies JobID and JobName, if we don't change both of them, we should remove such verification. Otherwise we' better to check update.

Code under review:
public void testPutAndRemoveJobGraph() throws Exception {
		ZooKeeperSubmittedJobGraphStore jobGraphs = createZooKeeperSubmittedJobGraphStore("/testPutAndRemoveJobGraph");

		try {
			SubmittedJobGraphListener listener = mock(SubmittedJobGraphListener.class);

			jobGraphs.start(listener);

			SubmittedJobGraph jobGraph = createSubmittedJobGraph(new JobID());

			
			assertEquals(0, jobGraphs.getJobIds().size());

			
			jobGraphs.putJobGraph(jobGraph);

			
			Collection<JobID> jobIds = jobGraphs.getJobIds();
			assertEquals(1, jobIds.size());

			JobID jobId = jobIds.iterator().next();

			verifyJobGraphs(jobGraph, jobGraphs.recoverJobGraph(jobId));

			
			jobGraph = createSubmittedJobGraph(jobGraph.getJobId());
			jobGraphs.putJobGraph(jobGraph);

			
			jobIds = jobGraphs.getJobIds();
			assertEquals(1, jobIds.size());

			jobId = jobIds.iterator().next();

			verifyJobGraphs(jobGraph, jobGraphs.recoverJobGraph(jobId));

			
			jobGraphs.removeJobGraph(jobGraph.getJobId());

			
			assertEquals(0, jobGraphs.getJobIds().size());

			
			verify(listener, atMost(1)).onAddedJobGraph(any(JobID.class));
			verify(listener, never()).onRemovedJobGraph(any(JobID.class));

			
			jobGraphs.removeJobGraph(jobGraph.getJobId());
		}
		finally {
			jobGraphs.stop();
		}
	}

Line commented on: jobGraphs.putJobGraph(jobGraph);

Revised code:
public void testPutAndRemoveJobGraph() throws Exception {
		ZooKeeperSubmittedJobGraphStore jobGraphs = createZooKeeperSubmittedJobGraphStore("/testPutAndRemoveJobGraph");

		try {
			SubmittedJobGraphListener listener = mock(SubmittedJobGraphListener.class);

			jobGraphs.start(listener);

			SubmittedJobGraph jobGraph = createSubmittedJobGraph(new JobID(), "JobName");

			
			assertEquals(0, jobGraphs.getJobIds().size());

			
			jobGraphs.putJobGraph(jobGraph);

			
			Collection<JobID> jobIds = jobGraphs.getJobIds();
			assertEquals(1, jobIds.size());

			JobID jobId = jobIds.iterator().next();

			verifyJobGraphs(jobGraph, jobGraphs.recoverJobGraph(jobId));

			
			jobGraph = createSubmittedJobGraph(jobGraph.getJobId(), "Updated JobName");
			jobGraphs.putJobGraph(jobGraph);

			
			jobIds = jobGraphs.getJobIds();
			assertEquals(1, jobIds.size());

			jobId = jobIds.iterator().next();

			verifyJobGraphs(jobGraph, jobGraphs.recoverJobGraph(jobId));

			
			jobGraphs.removeJobGraph(jobGraph.getJobId());

			
			assertEquals(0, jobGraphs.getJobIds().size());

			
			verify(listener, atMost(1)).onAddedJobGraph(any(JobID.class));
			verify(listener, never()).onRemovedJobGraph(any(JobID.class));

			
			jobGraphs.removeJobGraph(jobGraph.getJobId());
		}
		finally {
			jobGraphs.stop();
		}
	}
Code review comment: Can be merged with the previous line.

Code under review:
public void testEncryptRsaEcbOAEPwithSHA512andMGF1() {
        byte[] message = "Ballerina crypto test           ".getBytes(StandardCharsets.UTF_8);
        BValueArray messageValue = new BValueArray(message);

        BValue[] args = {messageValue, new BString(confRoot.resolve("testKeystore.p12").toString()),
                new BString("ballerina"), new BString("ballerina"),
                new BString("OAEPwithSHA512andMGF1")};
        BValue[] returnValues = BRunUtil.invoke(compileResult, "testEncryptRsaEcb", args);
        Assert.assertFalse(returnValues == null || returnValues.length == 0 || returnValues[0] == null);
        Assert.assertTrue(returnValues[0] instanceof BValueArray);

        BValue[] args1 = {returnValues[0], new BString(confRoot.resolve("testKeystore.p12").toString()),
                new BString("ballerina"), new BString("ballerina"), new BString("ballerina"),
                new BString("OAEPwithSHA512andMGF1")};
        returnValues = BRunUtil.invoke(compileResult, "testDecryptRsaEcb", args1);
        Assert.assertFalse(returnValues == null || returnValues.length == 0 || returnValues[0] == null);
        Assert.assertEquals(((BValueArray) returnValues[0]).getBytes(), message);
    }

Line commented on: new BString("OAEPwithSHA512andMGF1")};

Revised code:
public void testEncryptRsaEcbOAEPwithSHA512andMGF1() {
        byte[] message = "Ballerina crypto test           ".getBytes(StandardCharsets.UTF_8);
        BValueArray messageValue = new BValueArray(message);

        BValue[] args = {messageValue, new BString(confRoot.resolve("testKeystore.p12").toString()),
                new BString("ballerina"), new BString("ballerina"), new BString("OAEPwithSHA512andMGF1")};
        BValue[] returnValues = BRunUtil.invoke(compileResult, "testEncryptRsaEcb", args);
        Assert.assertFalse(returnValues == null || returnValues.length == 0 || returnValues[0] == null);
        Assert.assertTrue(returnValues[0] instanceof BValueArray);

        BValue[] args1 = {returnValues[0], new BString(confRoot.resolve("testKeystore.p12").toString()),
                new BString("ballerina"), new BString("ballerina"), new BString("ballerina"),
                new BString("OAEPwithSHA512andMGF1")};
        returnValues = BRunUtil.invoke(compileResult, "testDecryptRsaEcb", args1);
        Assert.assertFalse(returnValues == null || returnValues.length == 0 || returnValues[0] == null);
        Assert.assertEquals(((BValueArray) returnValues[0]).getBytes(), message);
    }
Code review comment: +1. We can remove that variable.

Code under review:
private boolean checkUnionHasSameType(BUnionType unionType, BType baseType) {
        LinkedHashSet<BType> memberTypes = unionType.getMemberTypes();
        boolean hasSameOrderedType = false;
        for (BType type : memberTypes) {
            type = getReferredType(type);
            if (type.tag == TypeTags.FINITE) {
                for (BLangExpression expr : ((BFiniteType) type).getValueSpace()) {
                    hasSameOrderedType = isSameOrderedType(expr.getBType(), baseType);
                    if (!hasSameOrderedType) {
                        return false;
                    }
                }
            } else if (type.tag == TypeTags.UNION) {
                if (!checkUnionHasSameType((BUnionType) type, baseType)) {
                    return false;
                }
                hasSameOrderedType = true;
            } else if (type.tag == TypeTags.TUPLE || type.tag == TypeTags.ARRAY) {
                if (!isSameOrderedType(type, baseType)) {
                    return false;
                }
                hasSameOrderedType = true;
            } else if (type.tag == TypeTags.INTERSECTION) {
                if (!isSameOrderedType(getEffectiveTypeForIntersection(type), baseType)) {
                    return false;
                }
                hasSameOrderedType = true;
            } else if (isSimpleBasicType(type.tag)) {
                hasSameOrderedType = isSameOrderedType(type, baseType) || isNil(type);
                if (!hasSameOrderedType) {
                    return false;
                }
            }
        }
        return hasSameOrderedType;
    }

Line commented on: boolean hasSameOrderedType = false;

Revised code:
private boolean checkUnionHasSameType(BUnionType unionType, BType baseType) {
        LinkedHashSet<BType> memberTypes = unionType.getMemberTypes();
        for (BType type : memberTypes) {
            type = getReferredType(type);
            if (type.tag == TypeTags.FINITE) {
                for (BLangExpression expr : ((BFiniteType) type).getValueSpace()) {
                    if (!isSameOrderedType(expr.getBType(), baseType)) {
                        return false;
                    }
                }




            } else if (type.tag == TypeTags.TUPLE || type.tag == TypeTags.ARRAY) {
                if (!isSameOrderedType(type, baseType)) {
                    return false;
                }
            } else if (type.tag == TypeTags.INTERSECTION) {
                if (!isSameOrderedType(getEffectiveTypeForIntersection(type), baseType)) {
                    return false;
                }
            } else if (isSimpleBasicType(type.tag)) {
                if (!isSameOrderedType(type, baseType) && !isNil(type)) {
                    return false;
                }
            }
        }
        return true;
    }
Code review comment: Done, though the purpose of this empty line was to separate the "given" from the "when" in the test

Code under review:
void assertReviserReturnsRevisedIndex() {
        IndexMetaData originalMetaData = new IndexMetaData("test_idx_tableName");
        originalMetaData.getColumns().add("column1");
        originalMetaData.getColumns().add("column2");
        originalMetaData.setUnique(true);
        SingleIndexReviser reviser = new SingleIndexReviser();
        
        Optional<IndexMetaData> optionalRevised = reviser.revise("tableName", originalMetaData, null);
        
        assertTrue(optionalRevised.isPresent());
        IndexMetaData actual = optionalRevised.get();
        assertThat(originalMetaData.isUnique(), is(actual.isUnique()));
        assertThat(originalMetaData.getColumns(), is(actual.getColumns()));
        assertThat("test_idx", is(actual.getName()));
    }


Revised code:
void assertReviserReturnsRevisedIndex() {
        IndexMetaData originalMetaData = new IndexMetaData("test_idx_tableName");
        originalMetaData.getColumns().add("column1");
        originalMetaData.getColumns().add("column2");
        originalMetaData.setUnique(true);
        SingleIndexReviser reviser = new SingleIndexReviser();
        Optional<IndexMetaData> optionalRevised = reviser.revise("tableName", originalMetaData, null);
        assertTrue(optionalRevised.isPresent());
        IndexMetaData actual = optionalRevised.get();
        assertThat(originalMetaData.isUnique(), is(actual.isUnique()));
        assertThat(originalMetaData.getColumns(), is(actual.getColumns()));
        assertThat("test_idx", is(actual.getName()));
    }
Code review comment: Wondering if we should make things a bit more complex. E.g. either the package should be `my.package` or it should start with `my.package.` (with a final dot). I don't think we want to catch `my.package2` and if I'm not mistaken, we catch it with what you did.

Code under review:
private List<Predicate<String>> initPredicates(Collection<String> exclusions) {
        final String packMatch = ".*";
        List<Predicate<String>> predicates = new ArrayList<>();
        for (String exclusionExpression : exclusions) {
            if (exclusionExpression.endsWith(packMatch)) {
                
                final String pack = exclusionExpression.substring(0, exclusionExpression.length() - packMatch.length());
                predicates.add(new Predicate<String>() {
                    @Override
                    public boolean test(String packageName) {
                        return packageName.startsWith(pack);
                    }
                });
            } else {
                
                predicates.add(new Predicate<String>() {
                    @Override
                    public boolean test(String packageName) {
                        return packageName.equals(exclusionExpression);
                    }
                });
            }
        }
        return predicates;
    }

Line commented on: final String pack = exclusionExpression.substring(0, exclusionExpression.length() - packMatch.length());

Revised code:
Override
                    public boolean test(String packageName) {
                        return packageName.equals(pack) || packageName.startsWith(pack + ".");
                    }
Code review comment: It is required since when we run the non native image (standard java jar) we do not want the TSM to be started.

Code under review:
public void configureRuntimeProperties(NarayanaSTMRecorder recorder) {
        recorder.disableTransactionStatusManager();
    }

Line commented on: recorder.disableTransactionStatusManager();

Revised code:
public void configureRuntimeProperties(NarayanaSTMRecorder recorder) {
        recorder.disableTransactionStatusManager();
    }
Code review comment: ```suggestion                     String.format("The jni reader fields' size {%s} is not matched with paimon fields' size {%s}. Please refresh table and try again", ```

Code under review:
private void initReader() throws IOException {
        ReadBuilder readBuilder = table.newReadBuilder();
        if (this.fields.length != this.paimonAllFieldNames.size()) {
            throw new IOException(
                    String.format("The jni reader fields' size {%s} is not matched with paimon fields' size {%s}",
                            fields.length, paimonAllFieldNames.size()));
        }
        readBuilder.withProjection(getProjected());
        readBuilder.withFilter(getPredicates());
        reader = readBuilder.newRead().createReader(getSplit());
    }

Line commented on: String.format("The jni reader fields' size {%s} is not matched with paimon fields' size {%s}",

Revised code:
private void initReader() throws IOException {
        ReadBuilder readBuilder = table.newReadBuilder();
        if (this.fields.length != this.paimonAllFieldNames.size()) {
            throw new IOException(
                    String.format(
                            "The jni reader fields' size {%s} is not matched with paimon fields' size {%s}."
                                    + " Please refresh table and try again",
                            fields.length, paimonAllFieldNames.size()));
        }
        readBuilder.withProjection(getProjected());
        readBuilder.withFilter(getPredicates());
        reader = readBuilder.newRead().createReader(getSplit());
    }
Code review comment: No, it's just important to not accept all exceptions. Otherwise the AssertionError thrown by `snsWriterFnLogs.verifyWarn` fullfills this, which is kind of pointless. But that's exactly what happened, because logs for the wrong logger were captured.  Even more, the message of the assertion error contained exactly that string we expected to be thrown because it's also the string we were checking for in the logs.

Code under review:
public void testRetries() throws Throwable {
    thrown.expect(IOException.class);
    thrown.expectMessage("Error writing to SNS");
    thrown.expectMessage("No more attempts allowed");

    final PublishRequest request1 = createSampleMessage("my message that will not be published");
    final TupleTag<PublishResult> results = new TupleTag<>();
    final AmazonSNS amazonSnsErrors = getAmazonSnsMockErrors();
    p.apply(Create.of(request1))
        .apply(
            SnsIO.write()
                .withTopicName(topicName)
                .withRetryConfiguration(
                    SnsIO.RetryConfiguration.create(4, standardSeconds(10), millis(1)))
                .withAWSClientsProvider(new Provider(amazonSnsErrors))
                .withResultOutputTag(results));

    try {
      p.run();
    } catch (final Pipeline.PipelineExecutionException e) {
      
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 1));
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 2));
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 3));
      throw e.getCause();
    }
  }

Line commented on: thrown.expect(IOException.class);

Revised code:
public void testRetries() throws Throwable {
    thrown.expect(IOException.class);
    thrown.expectMessage("Error writing to SNS");
    thrown.expectMessage("No more attempts allowed");

    final PublishRequest request1 = createSampleMessage("my message that will not be published");
    final TupleTag<PublishResult> results = new TupleTag<>();
    final AmazonSNS amazonSnsErrors = getAmazonSnsMockErrors();
    p.apply(Create.of(request1))
        .apply(
            SnsIO.write()
                .withTopicName(topicName)
                .withRetryConfiguration(
                    SnsIO.RetryConfiguration.create(4, standardSeconds(10), millis(1)))
                .withAWSClientsProvider(new Provider(amazonSnsErrors))
                .withResultOutputTag(results));

    try {
      p.run();
    } catch (final Pipeline.PipelineExecutionException e) {
      
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 1));
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 2));
      snsWriterFnLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 3));
      throw e.getCause();
    }
  }
Code review comment: Can you add some comment highlighting and justifying that you are iterating here over all of the state handles?

Code under review:
public String toString() {
        synchronized (registeredStates) {
            return "SharedStateRegistry{" + "registeredStates=" + registeredStates + '}';
        }
    }

Line commented on: synchronized (registeredStates) {

Revised code:
public String toString() {
        synchronized (registeredStates) {
            return "SharedStateRegistry{" + "registeredStates=" + registeredStates + '}';
        }
    }
Code review comment: Shall we add tests for the not-covered cases?

Code under review:
private boolean isOverridden(Method method1, Method method2, Class<?> clazz) {
        
        if ((Modifier.isStatic(method1.getModifiers()) ^ Modifier.isStatic(method2.getModifiers())) ||
                method1.getParameterCount() != method2.getParameterCount()) {
            
            
            
            throw new JInteropException(
                    OVERLOADED_METHODS, "Overloaded methods cannot be differentiated. Please specify the " +
                    "parameterTypes for each parameter in 'paramTypes' field in the annotation");
        }
        
        Method currentMethod;
        Method otherMethod;
        if (method2.getReturnType().isAssignableFrom(method1.getReturnType())) {
            currentMethod = method1;
            otherMethod = method2;
        } else if (method1.getReturnType().isAssignableFrom(method2.getReturnType())) {
            currentMethod = method2;
            otherMethod = method1;
        } else {
            return false;
        }

        try {
            Method superMethod = clazz.getSuperclass().getDeclaredMethod(currentMethod.getName(),
                    currentMethod.getParameterTypes());
            if (Modifier.isStatic(currentMethod.getModifiers())) {
                return superMethod.equals(otherMethod);
            }
            return Arrays.equals(superMethod.getParameterTypes(), otherMethod.getParameterTypes()) &&
                    superMethod.getReturnType().equals(otherMethod.getReturnType());
        } catch (NoSuchMethodException e) {
            return false;
        }
    }

Line commented on: return false;

Revised code:
private boolean isOverridden(Method method1, Method method2, Class<?> clazz) {
        if (method1.getParameterCount() != method2.getParameterCount()) {
            
            
            
            throw new JInteropException(OVERLOADED_METHODS, "Overloaded methods cannot be differentiated. " +
                    "Please specify the parameter types for each parameter in 'paramTypes' field in the annotation");
        }
        
        Method currentMethod;
        Method otherMethod;
        if (method2.getReturnType().isAssignableFrom(method1.getReturnType())) {
            currentMethod = method1;
            otherMethod = method2;
        } else if (method1.getReturnType().isAssignableFrom(method2.getReturnType())) {
            currentMethod = method2;
            otherMethod = method1;
        } else {
            return false;
        }

        try {
            Method superMethod = clazz.getSuperclass()
                    .getDeclaredMethod(currentMethod.getName(), currentMethod.getParameterTypes());
            return Arrays.equals(superMethod.getParameterTypes(), otherMethod.getParameterTypes()) &&
                    superMethod.getReturnType().equals(otherMethod.getReturnType());
        } catch (NoSuchMethodException e) {
            return false;
        }
    }
Code review comment: based on our discussion converting all the exception to SBRE .

Code under review:
private Throwable mapError(Throwable throwable, ServiceBusErrorSource errorSource) {
        
        if (!(throwable instanceof AmqpException) && !(throwable instanceof ServiceBusReceiverException)) {
            return new ServiceBusReceiverException(throwable, errorSource);
        }
        return throwable;
    }

Line commented on: if (!(throwable instanceof AmqpException) && !(throwable instanceof ServiceBusReceiverException)) {

Revised code:
private Throwable mapError(Throwable throwable, ServiceBusErrorSource errorSource) {
        
        if (!(throwable instanceof ServiceBusReceiverException)) {
            return new ServiceBusReceiverException(throwable, errorSource);
        }
        return throwable;
    }
Code review comment: Instead of having `flux` and `iterable` fields in this class, if an instance of IterableStream is created using an iterable, can this just be converted to `this.flux = Flux.fromIterable(Objects.requireNonNull(iterable, "'iterable' cannot be null."));`. Simplifies code in other methods too where you don't have to check if flux is null or iterable is null.

Code under review:
public IterableStream(Iterable<T> iterable) {
        this.iterable = Objects.requireNonNull(iterable, "'iterable' cannot be null.");
        this.flux = null;
    }

Line commented on: this.iterable = Objects.requireNonNull(iterable, "'iterable' cannot be null.");

Revised code:
public IterableStream(Iterable<T> iterable) {
        this.iterable = Objects.requireNonNull(iterable, "'iterable' cannot be null.");
        this.flux = null;
    }
Code review comment: Thanks. Method `isShadow()` in the algorithm is used to determine whether the sql contains shadow coloring.  Shadow tables and shadow algorithms are separately configured and referenced by ID.  If the user customizes the algorithm, he needs to get the configured shadow tables to customize his own judgment algorithm.  So the beginning of `isContainTable` is not used.

Code under review:
public boolean isShadow(final Collection<String> shadowTableNames, final PreciseColumnShadowValue<Comparable<?>> shadowValue) {
        boolean containTable = shadowTableNames.contains(shadowValue.getLogicTableName());
        boolean isSameOperation = shadowOperationType == shadowValue.getShadowOperationType();
        boolean isSameColumnName = Objects.equals(props.get(COLUMN), shadowValue.getColumnName());
        boolean isRegexMatch = String.valueOf(shadowValue.getValue()).matches(props.get(REGEX).toString());
        return containTable && isSameOperation && isSameColumnName && isRegexMatch;
    }

Line commented on: boolean containTable = shadowTableNames.contains(shadowValue.getLogicTableName());

Revised code:
public boolean isShadow(final Collection<String> shadowTableNames, final PreciseColumnShadowValue<Comparable<?>> shadowValue) {
        boolean containTable = shadowTableNames.contains(shadowValue.getLogicTableName());
        boolean isSameOperation = shadowOperationType == shadowValue.getShadowOperationType();
        boolean isSameColumnName = Objects.equals(props.get(COLUMN), shadowValue.getColumnName());
        boolean isRegexMatch = String.valueOf(shadowValue.getValue()).matches(props.get(REGEX).toString());
        return containTable && isSameOperation && isSameColumnName && isRegexMatch;
    }
Code review comment: The `count != 0` is not under the lock

Code under review:
public void awaitZero() throws InterruptedException {
        if (count != 0) {
            latch.await(); 
        }
    }

Line commented on: if (count != 0) {

Revised code:
public void awaitZero() throws InterruptedException {
        sync.acquireSharedInterruptibly(1);
    }
Code review comment: I agree with the change but I think the comment should be:  ```suggestion             //We must not ignore the returned CompletionStage! ```

Code under review:
public void disposeStageSession(@Disposes Stage.Session reactiveSession) {
        if (reactiveSession != null) {
            
            
            
            
            
            reactiveSession.close().toCompletableFuture().join();
        }
    }


Revised code:
public void disposeStageSession(@Disposes Stage.Session reactiveSession) {
        if (reactiveSession != null) {
            
            
            
            
            
            reactiveSession.close().toCompletableFuture().join();
        }
    }
Code review comment: Could you also add some comments after the `SET`? Because it fails to parse the statement when comment is around SQL Client commands.

Code under review:
public void testInitFile() throws Exception {
        List<String> statements =
                Arrays.asList(
                        "-- Define Table \n"
                                + "CREATE TABLE source ("
                                + "id INT,"
                                + "val STRING"
                                + ") WITH ("
                                + "  'connector' = 'values'"
                                + "); "
                                + "-- Define Table \n",
                        "SET key = value;\n");
        String initFile = createSqlFile(statements, "init-sql.sql");

        String[] args = new String[] {"-i", initFile};
        String output = runSqlClient(args, "SET;\nQUIT;\n");
        assertThat(output, containsString("key=value"));
    }

Line commented on: "SET key = value;\n");

Revised code:
public void testInitFile() throws Exception {
        List<String> statements =
                Arrays.asList(
                        "-- define table \n"
                                + "CREATE TABLE source ("
                                + "id INT,"
                                + "val STRING"
                                + ") WITH ("
                                + "  'connector' = 'values'"
                                + "); \n",
                        " -- define config \nSET key = value;\n");
        String initFile = createSqlFile(statements, "init-sql.sql");

        String[] args = new String[] {"-i", initFile};
        String output = runSqlClient(args, "SET;\nQUIT;\n");
        assertThat(output, containsString("key=value"));
    }
Code review comment: Does base class method call need to be "synchronized"/single threaded as well?

Code under review:
public void populatePropertyBag() {
        
            super.populatePropertyBag();

        synchronized(this) {
            setProperty(
                this,
                Constants.Properties.CHANGE_FEED_START_FROM_TYPE,
                ChangeFeedStartFromTypes.NOW);
        }
    }

Line commented on: super.populatePropertyBag();

Revised code:
public void populatePropertyBag() {
        
            super.populatePropertyBag();

        synchronized(this) {
            setProperty(
                this,
                Constants.Properties.CHANGE_FEED_START_FROM_TYPE,
                ChangeFeedStartFromTypes.NOW);
        }
    }
Code review comment: except sort node, all other node should use same logical to do limit merge, right?

Code under review:
public PlanFragment visitPhysicalLimit(PhysicalLimit<Plan> physicalLimit, PlanTranslatorContext context) {
        PlanFragment inputFragment = physicalLimit.child(0).accept(this, context);
        PlanNode child = inputFragment.getPlanRoot();
        if (child instanceof OlapScanNode) {
            child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
            return inputFragment;
        }

        if (child instanceof SortNode) {
            ((SortNode) child).setOffset(physicalLimit.getOffset());
            child.setLimit(physicalLimit.getLimit());
            inputFragment.getChildren().forEach(fragment -> {
                PlanNode root = fragment.getPlanRoot();
                if (root instanceof SortNode) {
                    root.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
                }
            });
            return inputFragment;
        }

        if (child instanceof AggregationNode) {
            child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
            return inputFragment;
        }
        return inputFragment;
    }

Line commented on: if (child instanceof AggregationNode) {

Revised code:
public PlanFragment visitPhysicalLimit(PhysicalLimit<Plan> physicalLimit, PlanTranslatorContext context) {
        PlanFragment inputFragment = physicalLimit.child(0).accept(this, context);
        PlanNode child = inputFragment.getPlanRoot();

        
        
        
        
        if (child instanceof SortNode) {
            SortNode sort = (SortNode) child;
            sort.setLimit(physicalLimit.getLimit());
            sort.setOffset(physicalLimit.getOffset());
            return inputFragment;
        }
        if (child instanceof ExchangeNode) {
            ExchangeNode exchangeNode = (ExchangeNode) child;
            exchangeNode.setLimit(physicalLimit.getLimit());
            
            
            exchangeNode.setOffset(physicalLimit.getOffset());
            if (exchangeNode.getChild(0) instanceof SortNode) {
                SortNode sort = (SortNode) exchangeNode.getChild(0);
                sort.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
                sort.setOffset(0);
            }
            return inputFragment;
        }
        
        child.setLimit(physicalLimit.getLimit() + physicalLimit.getOffset());
        return inputFragment;
    }
Code review comment: but the user can only pass binaryData, right? so it is already serialized.

Code under review:
public CloudEvent(String source, String type, BinaryData data, CloudEventDataFormat format, String dataContentType) {
        if (Objects.isNull(source)) {
            throw LOGGER.logExceptionAsError(new NullPointerException("'source' cannot be null."));
        }
        if (Objects.isNull(type)) {
            throw LOGGER.logExceptionAsError(new NullPointerException("'type' cannot be null."));
        }
        if (Objects.isNull(data)) {
            throw LOGGER.logExceptionAsError(new NullPointerException("'data' cannot be null."));
        }
        if (Objects.isNull(format)) {
            throw LOGGER.logExceptionAsError(new NullPointerException("'format' cannot be null."));
        }
        this.source = source;
        this.type = type;
        if (CloudEventDataFormat.BYTES == format) {
            this.dataBase64 = Base64.getEncoder().encodeToString(data.toBytes());
        } else {
            this.data = data.toString();
        }
        this.dataContentType = dataContentType;
        this.id = UUID.randomUUID().toString();
        this.specVersion = CloudEvent.SPEC_VERSION;
    }

Line commented on: this.data = data.toString();

Revised code:
public CloudEvent(String source, String type, BinaryData data, CloudEventDataFormat format, String dataContentType) {
        Objects.requireNonNull(source, "'source' cannot be null.");
        Objects.requireNonNull(type, "'type' cannot be null.");
        this.source = source;
        this.type = type;
        if (data != null) {
            Objects.requireNonNull(format, "'format' cannot be null when 'data' isn't null.");
            if (CloudEventDataFormat.BYTES == format) {
                this.dataBase64 = Base64.getEncoder().encodeToString(data.toBytes());
            } else {
                try {
                    this.data = BINARY_DATA_OBJECT_MAPPER.readTree(data.toBytes());
                } catch (IOException e) {
                    throw LOGGER.logExceptionAsError(new IllegalArgumentException("'data' isn't in valid Json format",
                        e));
                }
            }
        }
        this.dataContentType = dataContentType;
        this.id = UUID.randomUUID().toString();
        this.specVersion = CloudEvent.SPEC_VERSION;
        this.binaryData = data;
    }