muttakin2301/ccr2000 huggingface dataset:
Code review comment: I am thinking whether we should special case, `int? status_code`. But checking int is sufficient for now.

Code under review:
private static int getStatusCode(BMap<String, BValue> errorDetails) {
        if (!errorDetails.hasKey(STATUS_CODE)) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        BValue specifiedStatusCode = errorDetails.get(STATUS_CODE);
        if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        long specifiedIntCode = ((BInteger) specifiedStatusCode).intValue();
        if (specifiedIntCode >= DEFAULT_ERROR_RETURN_STATUS_CODE) {
            if (specifiedIntCode > Integer.MAX_VALUE) {
                return Integer.MAX_VALUE;
            }
            return (int) specifiedIntCode;
        }
        return DEFAULT_ERROR_RETURN_STATUS_CODE;
    }

Line commented on: if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {

Revised code:
private static int getStatusCode(BMap<String, BValue> errorDetails) {
        if (!errorDetails.hasKey(STATUS_CODE)) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        BValue specifiedStatusCode = errorDetails.get(STATUS_CODE);
        if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        long specifiedIntCode = ((BInteger) specifiedStatusCode).intValue();
        if (specifiedIntCode >= DEFAULT_ERROR_RETURN_STATUS_CODE) {
            if (specifiedIntCode > Integer.MAX_VALUE) {
                return Integer.MAX_VALUE;
            }
            return (int) specifiedIntCode;
        }
        return DEFAULT_ERROR_RETURN_STATUS_CODE;
    }
Code review comment: You can't actually set the proto value to null, it will crash the program. This is an invalid input case though, since all user counters must be scoped to their step. I removed the test case which caused me to add this in the first place. If this occurs, we should drop the metric. I'll add a test for that too

Code under review:
private MonitoringInfo counterUpdateToMonitoringInfo(MetricUpdate<Long> metricUpdate) {
    SimpleMonitoringInfoBuilder builder = new SimpleMonitoringInfoBuilder(true);
    MetricName metricName = metricUpdate.getKey().metricName();
    if (metricName instanceof MonitoringInfoMetricName) {
      MonitoringInfoMetricName monitoringInfoName = (MonitoringInfoMetricName) metricName;
      
      builder.setUrn(monitoringInfoName.getUrn());
      for (Entry<String, String> e : monitoringInfoName.getLabels().entrySet()) {
        builder.setLabel(e.getKey(), e.getValue());
      }
    } else {
      
      builder.setUrnForUserMetric(
          metricUpdate.getKey().metricName().getNamespace(),
          metricUpdate.getKey().metricName().getName());
      String stepName = "";
      if (metricUpdate.getKey().stepName() != null) {
        stepName = metricUpdate.getKey().stepName();
      }
      builder.setPTransformLabel(stepName);
    }
    builder.setInt64Value(metricUpdate.getUpdate());
    builder.setTimestampToNow();
    return builder.build();
  }

Line commented on: builder.setPTransformLabel(stepName);

Revised code:
private MonitoringInfo counterUpdateToMonitoringInfo(MetricUpdate<Long> metricUpdate) {
    SimpleMonitoringInfoBuilder builder = new SimpleMonitoringInfoBuilder(true);
    MetricName metricName = metricUpdate.getKey().metricName();
    if (metricName instanceof MonitoringInfoMetricName) {
      MonitoringInfoMetricName monitoringInfoName = (MonitoringInfoMetricName) metricName;
      
      builder.setUrn(monitoringInfoName.getUrn());
      for (Entry<String, String> e : monitoringInfoName.getLabels().entrySet()) {
        builder.setLabel(e.getKey(), e.getValue());
      }
    } else { 
      
      builder.setUrnForUserMetric(
          metricUpdate.getKey().metricName().getNamespace(),
          metricUpdate.getKey().metricName().getName());
      
      
      if (this.stepName == null) {
        return null;
      }
      builder.setPTransformLabel(metricUpdate.getKey().stepName());
    }
    builder.setInt64Value(metricUpdate.getUpdate());
    builder.setTimestampToNow();
    return builder.build();
  }
Code review comment: ```suggestion             throw createLauncherException("Unable to generate API Documentation.", e.getCause()); ```

Code under review:
public void execute(Project project) {
        Path sourceRootPath = project.sourceRoot();
        Path targetDir = project.sourceRoot().resolve(ProjectDirConstants.TARGET_DIR_NAME);
        Path outputPath = targetDir.resolve(ProjectDirConstants.TARGET_API_DOC_DIRECTORY);
        this.out.println("Generating API Documentation");
        try {
            Files.createDirectories(outputPath);
            BallerinaDocGenerator.generateAPIDocs(project, outputPath.toString(), excludeIndex);
            this.out.println("\t" + sourceRootPath.relativize(outputPath).toString());

        } catch (IOException e) {
            throw createLauncherException("Unable to generate API Documentation.");
        }

    }

Line commented on: throw createLauncherException("Unable to generate API Documentation.");

Revised code:
public void execute(Project project) {
        Path sourceRootPath = project.sourceRoot();
        Target target;
        if (outputPath == null) {
            try {
                target = new Target(sourceRootPath);
                outputPath = target.getDocPath();
            } catch (IOException e) {
                throw createLauncherException("error occurred while generating docs: " + e.getMessage());
            }
        }
        this.out.println("Generating API Documentation");
        try {
            BallerinaDocGenerator.generateAPIDocs(project, outputPath.toString(), excludeIndex);
            this.out.println("Saved to: " + sourceRootPath.relativize(outputPath).toString());

        } catch (IOException e) {
            throw createLauncherException("Unable to generate API Documentation.", e.getCause());
        }

    }
Code review comment: Should this config item auto_check_statistics_in_minutes changed to auto_check_statistics_in_seconds?

Code under review:
public StatisticsAutoCollector() {
        super("Automatic Analyzer",
                TimeUnit.SECONDS.toMillis(Config.auto_check_statistics_in_minutes),
                new AnalysisTaskExecutor(Config.full_auto_analyze_simultaneously_running_task_num));
    }

Line commented on: TimeUnit.SECONDS.toMillis(Config.auto_check_statistics_in_minutes),

Revised code:
public StatisticsAutoCollector() {
        super("Automatic Analyzer",
                TimeUnit.MINUTES.toMillis(Config.auto_check_statistics_in_minutes),
                new AnalysisTaskExecutor(Config.full_auto_analyze_simultaneously_running_task_num));
    }
Code review comment: Good idea, a short comment would be helpful.

Code under review:
public void close() throws Exception {
        cacheCleanScheduler.shutdownNow();
        if (!cacheCleanScheduler.awaitTermination(5, TimeUnit.SECONDS)) {
            LOG.warn(
                    "Unable to cleanly shutdown cache clean scheduler of "
                            + "ChangelogHandleReaderWithCache in 5s");
        }

        Iterator<RefCountedFile> iterator = cache.values().iterator();
        while (iterator.hasNext()) {
            RefCountedFile cacheFile = iterator.next();
            iterator.remove();
            Files.deleteIfExists(cacheFile.getFile().toPath());
        }
    }

Line commented on: while (iterator.hasNext()) {

Revised code:
public void close() throws IOException {
                try {
                    super.close();
                } finally {
                    cache.computeIfPresent(
                            dfsPath,
                            (key, value) -> {
                                value.release();
                                if (value.getReferenceCounter() == NO_USING_REF_COUNT) {
                                    cacheCleanScheduler.schedule(
                                            () -> cleanCacheFile(dfsPath),
                                            cacheIdleMillis,
                                            TimeUnit.MILLISECONDS);
                                }
                                return value;
                            });
                }
            }
Code review comment: @strongduanmu Got it. Maybe we need a `init()` interface function to initialize some local members so as to avoid calculating some local values like `partitionRangeMap`

Code under review:
public Collection<String> doSharding(final Collection<String> availableTargetNames, final RangeShardingValue<Long> shardingValue) {
        Preconditions.checkNotNull(properties.get(RANGE_PARTITION_SPLIT_VALUE), "Range sharding algorithm range partition split value cannot be null.");
        Map<Integer, Range<Long>> partitionRangeMap = getPartitionRangeMap();
        Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
        for (long value = shardingValue.getValueRange().lowerEndpoint(); value <= shardingValue.getValueRange().upperEndpoint(); value++) {
            for (String each : availableTargetNames) {
                if (each.endsWith(getPartition(partitionRangeMap, value))) {
                    result.add(each);
                }
            }
        }
        return result;
    }

Line commented on: if (each.endsWith(getPartition(partitionRangeMap, value))) {

Revised code:
public Collection<String> doSharding(final Collection<String> availableTargetNames, final RangeShardingValue<Long> shardingValue) {
        Preconditions.checkNotNull(properties.get(RANGE_PARTITION_SPLIT_VALUE), "Range sharding algorithm range partition split value cannot be null.");
        Map<Integer, Range<Long>> partitionRangeMap = getPartitionRangeMap();
        Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
        int lowerEndpointPartition = getPartition(partitionRangeMap, shardingValue.getValueRange().lowerEndpoint());
        int upperEndpointPartition = getPartition(partitionRangeMap, shardingValue.getValueRange().upperEndpoint());
        for (int partition = lowerEndpointPartition; partition <= upperEndpointPartition; partition++) {
            for (String each : availableTargetNames) {
                if (each.endsWith(partition + "")) {
                    result.add(each);
                }
            }
        }
        return result;
    }
Code review comment: Yes, eventually, but we thought there were issues with having it in staging (in CD also) due to visiting being slow when running tests. If we don't do any visiting in the way we talk about here in CD tests (I think we should, though) we can enable it for all zones in CD. What do you think?

Code under review:
public int distributionBits() {
        
        List<Zone> zonesWith16DistributionBits = Arrays.asList(createZone(Environment.prod, "us-west-1"),
                                                               createZone(Environment.prod, "us-central-1"),
                                                               createZone(Environment.prod, "eu-west-1"),
                                                               createZone(Environment.prod, "ap-northeast-1"),
                                                               createZone(Environment.prod, "ap-northeast-2"),
                                                               createZone(Environment.prod, "us-east-3"));
        if ((zone.system() == SystemName.cd && zone.environment() == Environment.prod) || 
                zonesWith16DistributionBits.contains(zone))
            return 16;
        else
            return DistributionBitCalculator.getDistributionBits(getNodeCountPerGroup(), getDistributionMode());
    }

Line commented on: if ((zone.system() == SystemName.cd && zone.environment() == Environment.prod) ||

Revised code:
public int distributionBits() {
        
        List<Zone> zonesWith16DistributionBits = Arrays.asList(createZone(Environment.prod, "us-west-1"),
                                                               createZone(Environment.prod, "us-central-1"),
                                                               createZone(Environment.prod, "eu-west-1"),
                                                               createZone(Environment.prod, "ap-northeast-1"),
                                                               createZone(Environment.prod, "ap-northeast-2"),
                                                               createZone(Environment.prod, "us-east-3"));
        if (zone.system() == SystemName.cd || zonesWith16DistributionBits.contains(zone))
            return 16;
        else
            return DistributionBitCalculator.getDistributionBits(getNodeCountPerGroup(), getDistributionMode());
    }
Code review comment: Maybe a parameterized logging like `LOG.warn("Statistic convert error, type: {}, statistic: {}", type.toSql(), statistic, e)` is better.

Code under review:
public static Optional<Double> convertStatisticsToDouble(Type type, String statistic) {
        if (!type.canStatistic()) {
            throw new StarRocksPlannerException("Error statistic type : " + type.toSql(), ErrorType.INTERNAL_ERROR);
        }
        try {
            switch (type.getPrimitiveType()) {
                case BOOLEAN:
                    if (statistic.equalsIgnoreCase("TRUE")) {
                        return Optional.of(1D);
                    } else {
                        return Optional.of(0D);
                    }
                case DATE:
                    return Optional.of((double) getLongFromDateTime(DateUtils.parseStringWithDefaultHSM(
                            statistic, DateUtils.DATE_FORMATTER_UNIX)));
                case DATETIME:
                    return Optional.of((double) getLongFromDateTime(DateUtils.parseStringWithDefaultHSM(
                            statistic, DateUtils.DATE_TIME_FORMATTER_UNIX)));
                case CHAR:
                case VARCHAR:
                    return Optional.empty();
                default:
                    return Optional.of(Double.parseDouble(statistic));
            }
        } catch (Exception e) {
            LOG.warn("Statistic convert error, type" + type.toSql() + ", statistic " + statistic + ", " + e.getMessage());
            return Optional.empty();
        }
    }

Line commented on: LOG.warn("Statistic convert error, type" + type.toSql() + ", statistic " + statistic + ", " + e.getMessage());

Revised code:
public static Optional<Double> convertStatisticsToDouble(Type type, String statistic) {
        if (!type.canStatistic()) {
            throw new StarRocksPlannerException("Error statistic type : " + type.toSql(), ErrorType.INTERNAL_ERROR);
        }
        try {
            switch (type.getPrimitiveType()) {
                case BOOLEAN:
                    if (statistic.equalsIgnoreCase("TRUE")) {
                        return Optional.of(1D);
                    } else {
                        return Optional.of(0D);
                    }
                case DATE:
                    return Optional.of((double) getLongFromDateTime(DateUtils.parseStringWithDefaultHSM(
                            statistic, DateUtils.DATE_FORMATTER_UNIX)));
                case DATETIME:
                    return Optional.of((double) getLongFromDateTime(DateUtils.parseStringWithDefaultHSM(
                            statistic, DateUtils.DATE_TIME_FORMATTER_UNIX)));
                case CHAR:
                case VARCHAR:
                    return Optional.empty();
                default:
                    return Optional.of(Double.parseDouble(statistic));
            }
        } catch (Exception e) {
            LOG.warn(String.format("Statistic convert error, type %s, statistic %s, %s",
                    type.toSql(), statistic, e.getMessage()));
            return Optional.empty();
        }
    }
Code review comment: do we work with Paths too?

Code under review:
public boolean isFileDownload() {
        if (responseType == null) {
            return false;
        }
        return File.class.equals(responseType.getRawType());
    }

Line commented on: if (responseType == null) {

Revised code:
public boolean isFileDownload() {
        if (responseType == null) {
            return false;
        }
        Class<?> rawType = responseType.getRawType();
        return File.class.equals(rawType) || Path.class.equals(rawType);
    }
Code review comment: This `checkState` was preventing from some bugs and data loses. Maybe replace with: ``` checkState(!bufferBuilders[targetChannel].isPresent() || bufferBuilders[targetChannel].isFinished()); ``` ?

Code under review:
private BufferBuilder requestNewBufferBuilder(int targetChannel) throws IOException, InterruptedException {
		BufferBuilder bufferBuilder = targetPartition.getBufferProvider().requestBufferBuilderBlocking();
		bufferBuilders[targetChannel] = Optional.of(bufferBuilder);
		targetPartition.addBufferConsumer(bufferBuilder.createBufferConsumer(), targetChannel);
		return bufferBuilder;
	}

Line commented on: return bufferBuilder;

Revised code:
private BufferBuilder requestNewBufferBuilder(int targetChannel) throws IOException, InterruptedException {
		checkState(!bufferBuilders[targetChannel].isPresent() || bufferBuilders[targetChannel].get().isFinished());

		BufferBuilder bufferBuilder = targetPartition.getBufferProvider().requestBufferBuilderBlocking();
		bufferBuilders[targetChannel] = Optional.of(bufferBuilder);
		targetPartition.addBufferConsumer(bufferBuilder.createBufferConsumer(), targetChannel);
		return bufferBuilder;
	}
Code review comment: 1. there is no `else` - do you think this would be simpler? ``` if (requestedPartitionsFlag) { 	return; }  synchronized (...) ```  2. yes - but thereby, I'm making `SingleInputGate` adhere the same standards as `UnionInputGate` for this overload of `requestPartitions` - the behaviour should be consistent, one way or the other. Should I add a note to the method's JavaDoc?

Code under review:
public void requestPartitions() throws IOException, InterruptedException {
		if (!requestedPartitionsFlag) {
			synchronized (requestLock) {
				if (isReleased) {
					throw new IllegalStateException("Already released.");
				}

				
				if (numberOfInputChannels != inputChannels.size()) {
					throw new IllegalStateException("Bug in input gate setup logic: mismatch between" +
							"number of total input channels and the currently set number of input " +
							"channels.");
				}

				for (InputChannel inputChannel : inputChannels.values()) {
					inputChannel.requestSubpartition(consumedSubpartitionIndex);
				}
			}

			requestedPartitionsFlag = true;
		}
	}

Line commented on: if (!requestedPartitionsFlag) {

Revised code:
public void requestPartitions() throws IOException, InterruptedException {
		if (requestedPartitionsFlag) {
			return;
		}

		synchronized (requestLock) {
			if (isReleased) {
				throw new IllegalStateException("Already released.");
			}

			
			if (numberOfInputChannels != inputChannels.size()) {
				throw new IllegalStateException("Bug in input gate setup logic: mismatch between" +
						"number of total input channels and the currently set number of input " +
						"channels.");
			}

			for (InputChannel inputChannel : inputChannels.values()) {
				inputChannel.requestSubpartition(consumedSubpartitionIndex);
			}
		}

		requestedPartitionsFlag = true;
	}
Code review comment: it check for atomic single document. For atomic single operation, document can't be null but can be empty.  For batch operation, document**s** can not be null and empty list.

Code under review:
public PiiEntityCollection recognizePiiEntities(String document, String language) {
        Objects.requireNonNull(document, "'document' cannot be null.");
        return client.recognizePiiEntities(document, language).block();
    }

Line commented on: Objects.requireNonNull(document, "'document' cannot be null.");

Revised code:
public PiiEntityCollection recognizePiiEntities(String document, String language) {
        Objects.requireNonNull(document, "'document' cannot be null.");
        return client.recognizePiiEntities(document, language).block();
    }
Code review comment: This is done intentionally, till the time we enable telemetry by default i don't want cx to get this log. Otherwise every customer will be getting this every 10 min , and might create a cri with question.

Code under review:
private Mono<Void> sendClientTelemetry() {
        return Mono.delay(Duration.ofSeconds(clientTelemetrySchedulingSec))
            .flatMap(t -> {
                if (this.isClosed) {
                    logger.warn("client already closed");
                    return Mono.empty();
                }

                if(!Configs.isClientTelemetryEnabled(this.isClientTelemetryEnabled)) {
                    logger.trace("client telemetry not enabled");
                    return Mono.empty();
                }

                readHistogram();
                try {
                    logger.info("ClientTelemetry {}", OBJECT_MAPPER.writeValueAsString(this.clientTelemetryInfo));
                } catch (JsonProcessingException e) {
                    e.printStackTrace();
                }
                clearDataForNextRun();
                return this.sendClientTelemetry();
            }).onErrorResume(ex -> {
                logger.error("sendClientTelemetry() - Unable to send client telemetry" +
                    ". Exception: {}", ex.toString(), ex);
                clearDataForNextRun();
                return this.sendClientTelemetry();
            }).subscribeOn(scheduler);
    }

Line commented on: logger.trace("client telemetry not enabled");

Revised code:
private Mono<Void> sendClientTelemetry() {
        return Mono.delay(Duration.ofSeconds(clientTelemetrySchedulingSec))
            .flatMap(t -> {
                if (this.isClosed) {
                    logger.warn("client already closed");
                    return Mono.empty();
                }

                if(!Configs.isClientTelemetryEnabled(this.isClientTelemetryEnabled)) {
                    logger.trace("client telemetry not enabled");
                    return Mono.empty();
                }

                readHistogram();
                try {
                    logger.info("ClientTelemetry {}", OBJECT_MAPPER.writeValueAsString(this.clientTelemetryInfo));
                } catch (JsonProcessingException e) {
                    logger.error("Error which parsing client telemetry into json. ", e);
                }
                clearDataForNextRun();
                return this.sendClientTelemetry();
            }).onErrorResume(ex -> {
                logger.error("sendClientTelemetry() - Unable to send client telemetry" +
                    ". Exception: ", ex);
                clearDataForNextRun();
                return this.sendClientTelemetry();
            }).subscribeOn(scheduler);
    }
Code review comment: I think I fixed it in b83b280ce2fb493eb647ffa589613c0b0362f39a which is part of #5774

Code under review:
private ExpectedJobIdJobManagerRunnerFactory(JobID expectedJobId) {
			this.expectedJobId = expectedJobId;
		}

Line commented on: this.expectedJobId = expectedJobId;

Revised code:
private ExpectedJobIdJobManagerRunnerFactory(JobID expectedJobId) {
			this.expectedJobId = expectedJobId;
		}
Code review comment: To verify that concurrent failovers are working fine, I think it's better to let the DefaultScheduler do the `recordModification` in `restartTasksWithDelay` rather than doing it directly in the test.  This may require a task failure to be triggered. The failure should affect task X and happen when task X is in a certain stage, including (waiting for assigning resource), (resource is assigned but waiting for other tasks to finish resource assignment to do `deployAll`).

Code under review:
public void skipDeploymentIfVertexVersionOutdated() {
		final JobGraph jobGraph = singleNonParallelJobVertexJobGraph();

		final DefaultScheduler scheduler = createSchedulerAndStartScheduling(jobGraph);

		final List<ExecutionVertexID> initiallyScheduledVertices = testExecutionVertexOperations.getDeployedVertices();

		final ArchivedExecutionVertex onlyExecutionVertex = Iterables.getOnlyElement(scheduler.requestJob().getAllExecutionVertices());
		final ExecutionAttemptID attemptId = onlyExecutionVertex.getCurrentExecutionAttempt().getAttemptId();
		scheduler.updateTaskExecutionState(new TaskExecutionState(jobGraph.getJobID(), attemptId, ExecutionState.FAILED));

		testExecutionSlotAllocator.disableAutoCompletePendingRequests();
		taskRestartExecutor.triggerScheduledTasks();
		executionVertexVersioner.recordModification(new ExecutionVertexID(getOnlyJobVertex(jobGraph).getID(), 0));
		testExecutionSlotAllocator.completePendingRequests();

		assertThat(initiallyScheduledVertices, is(equalTo(testExecutionVertexOperations.getDeployedVertices())));
	}

Line commented on: executionVertexVersioner.recordModification(new ExecutionVertexID(getOnlyJobVertex(jobGraph).getID(), 0));

Revised code:
public void skipDeploymentIfVertexVersionOutdated() {
		testExecutionSlotAllocator.disableAutoCompletePendingRequests();

		final JobGraph jobGraph = nonParallelSourceSinkJobGraph();
		final List<JobVertex> sortedJobVertices = jobGraph.getVerticesSortedTopologicallyFromSources();
		final ExecutionVertexID sourceExecutionVertexId = new ExecutionVertexID(sortedJobVertices.get(0).getID(), 0);
		final ExecutionVertexID sinkExecutionVertexId = new ExecutionVertexID(sortedJobVertices.get(1).getID(), 0);

		final DefaultScheduler scheduler = createSchedulerAndStartScheduling(jobGraph);
		testExecutionSlotAllocator.completePendingRequest(sourceExecutionVertexId);

		final ArchivedExecutionVertex sourceExecutionVertex = scheduler.requestJob().getAllExecutionVertices().iterator().next();
		final ExecutionAttemptID attemptId = sourceExecutionVertex.getCurrentExecutionAttempt().getAttemptId();
		scheduler.updateTaskExecutionState(new TaskExecutionState(jobGraph.getJobID(), attemptId, ExecutionState.FAILED));
		testRestartBackoffTimeStrategy.setCanRestart(false);

		testExecutionSlotAllocator.enableAutoCompletePendingRequests();
		taskRestartExecutor.triggerScheduledTasks();

		assertThat(testExecutionVertexOperations.getDeployedVertices(), containsInAnyOrder(sourceExecutionVertexId, sinkExecutionVertexId));
		assertThat(scheduler.requestJob().getState(), is(equalTo(JobStatus.RUNNING)));
	}
Code review comment: Why use double quotes here? IIUC, this is invalid in ADD JAR command?

Code under review:
public void testAddCommand() {
        TableEnvironmentInternal tableEnvInternal = (TableEnvironmentInternal) tableEnv;
        Parser parser = tableEnvInternal.getParser();

        
        Operation operation = parser.parse("add jar test.jar").get(0);
        assertThat(operation).isInstanceOf(AddJarOperation.class);
        assertThat(((AddJarOperation) operation).getPath()).isEqualTo("test.jar");
        
        operation = parser.parse("add jar \"${hiveconf:common-key}.jar\"").get(0);
        assertThat(operation).isInstanceOf(AddJarOperation.class);
        assertThat(((AddJarOperation) operation).getPath()).isEqualTo("\"common-val.jar\"");

        
        assertThatThrownBy(() -> tableEnv.executeSql("add jar t1.jar t2.jar"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage(
                        "Only support add one jar in one single statement. Usage: add JAR <value>");
        assertThatThrownBy(() -> tableEnv.executeSql("add File t1.txt"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage(
                        "Only support add one jar in one single statement. Usage: add JAR <value>");
        assertThatThrownBy(() -> tableEnv.executeSql("add Archive t1.tgz"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage(
                        "Only support add one jar in one single statement. Usage: add JAR <value>");
    }

Line commented on: assertThat(((AddJarOperation) operation).getPath()).isEqualTo("\"common-val.jar\"");

Revised code:
public void testAddCommand() {
        TableEnvironmentInternal tableEnvInternal = (TableEnvironmentInternal) tableEnv;
        Parser parser = tableEnvInternal.getParser();

        
        Operation operation = parser.parse("add jar test.jar").get(0);
        assertThat(operation).isInstanceOf(AddJarOperation.class);
        assertThat(((AddJarOperation) operation).getPath()).isEqualTo("test.jar");
        
        operation = parser.parse("add jar ${hiveconf:common-key}.jar").get(0);
        assertThat(operation).isInstanceOf(AddJarOperation.class);
        assertThat(((AddJarOperation) operation).getPath()).isEqualTo("common-val.jar");

        
        assertThatThrownBy(() -> tableEnv.executeSql("add jar t1.jar t2.jar"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage(
                        "Add multiple jar in one single statement is not supported yet. Usage: ADD JAR <file_path>");
        assertThatThrownBy(() -> tableEnv.executeSql("add file t1.txt"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage("ADD FILE is not supported yet. Usage: ADD JAR <file_path>");
        assertThatThrownBy(() -> tableEnv.executeSql("add archive t1.tgz"))
                .isInstanceOf(UnsupportedOperationException.class)
                .hasMessage("ADD ARCHIVE is not supported yet. Usage: ADD JAR <file_path>");
    }
Code review comment: Shall we do these assertions in Ballerina itself?

Code under review:
public void testOptionalRecordRemove() {
        BValue[] returns = BRunUtil.invoke(compileResult, "removeOptional");

        Assert.assertTrue(((BBoolean) returns[0]).booleanValue());
    }

Line commented on: Assert.assertTrue(((BBoolean) returns[0]).booleanValue());

Revised code:
public void testOptionalRecordRemove() {
        BValue[] returns = BRunUtil.invoke(compileResult, "removeOptional");
    }
Code review comment: I'm thinking maybe that we give the full  instructions will be more helpful. For example, `In order to use runenr determined sharding, please use --experiments=enable_streaming_auto_sharding, --experiments=use_runner_v2, --streaming, --enable_streaming_engine`. But it really depends on you.

Code under review:
void maybeRecordPCollectionWithAutoSharding(PCollection<?> pcol) {
    
    checkArgument(
        options.isEnableStreamingEngine(),
        "Runner determined sharding not available in Dataflow for GroupIntoBatches for"
            + " non-Streaming-Engine jobs.");
    checkArgument(
        hasExperiment(options, "enable_streaming_auto_sharding"),
        "Runner determined sharding not enabled in Dataflow for GroupIntoBatches."
            + " Try adding the experiment: --experiments=enable_streaming_auto_sharding.");
    
    boolean useJRH = hasExperiment(options, "beam_fn_api") && !useUnifiedWorker(options);
    checkArgument(
        !useJRH,
        "Runner determined sharding not available in Dataflow for GroupIntoBatches for portable"
            + " jobs not using runner v2. Try adding the experiment --experiments=use_runner_v2.");
    pcollectionsRequiringAutoSharding.add(pcol);
  }

Line commented on: + " non-Streaming-Engine jobs.");

Revised code:
void maybeRecordPCollectionWithAutoSharding(PCollection<?> pcol) {
    
    checkArgument(
        options.isEnableStreamingEngine(),
        "Runner determined sharding not available in Dataflow for GroupIntoBatches for"
            + " non-Streaming-Engine jobs.");
    checkArgument(
        hasExperiment(options, "enable_streaming_auto_sharding"),
        "Runner determined sharding not enabled in Dataflow for GroupIntoBatches."
            + " Try adding the experiment: --experiments=enable_streaming_auto_sharding.");
    
    boolean useJRH = hasExperiment(options, "beam_fn_api") && !useUnifiedWorker(options);
    checkArgument(
        !useJRH,
        "Runner determined sharding not available in Dataflow for GroupIntoBatches for portable"
            + " jobs not using runner v2. Try adding the experiment --experiments=use_runner_v2.");
    pcollectionsRequiringAutoSharding.add(pcol);
  }
Code review comment: I think this is a bit of a legacy issue where instance `equals()` checking has been done in contexts where what you really want is a semantical equality check instead of a value equality check (and descriptions do not matter for state semantics). So most likely intentional but not optimal...!  I personally feel descriptions don't really belong _within_ `ClusterState` in the first place, since they're really just metadata related to `set-node-state` and friends that is of zero importance to the content nodes and/or ideal state algorithm. But changing this will likely require re-wiring quite a bit of the cluster controller code that uses `NodeState` et al internally.

Code under review:
public boolean equals(Object o) {
        if (!(o instanceof NodeState)) { return false; }
        NodeState ns = (NodeState) o;
        
        if (state != ns.state
            || Math.abs(capacity - ns.capacity)         > 0.0000000001
            || Math.abs(initProgress - ns.initProgress) > 0.0000000001
            || startTimestamp != ns.startTimestamp
            || minUsedBits != ns.minUsedBits)
        {
            return false;
        }
        return true;
    }


Revised code:
public boolean equals(Object o) {
        if (!(o instanceof NodeState)) { return false; }
        NodeState ns = (NodeState) o;
        
        if (state != ns.state
            || Math.abs(capacity - ns.capacity)         > 0.0000000001
            || Math.abs(initProgress - ns.initProgress) > 0.0000000001
            || startTimestamp != ns.startTimestamp
            || minUsedBits != ns.minUsedBits)
        {
            return false;
        }
        return true;
    }
Code review comment: Interesting! Definitely relevant beyond Kotlin (for example https://checkerframework.org/releases/0.9.7/doc/checkers/nullness/quals/PolyNull.html) but your change is more general and obviously correct.

Code under review:
private void addExposedTypes(Invokable<?, ?> invokable, Class<?> cause) {
    addExposedTypes(invokable.getReturnType(), cause);
    for (Annotation annotation : invokable.getAnnotations()) {
      Class<? extends Annotation> annotationClass = annotation.annotationType();
      if (!Modifier.isPublic(annotationClass.getModifiers())) {
        
        
        
        continue;
      }
      LOG.debug(
          "Adding exposed types from {}, which is an annotation on invokable {}",
          annotation,
          invokable);
      addExposedTypes(annotationClass, cause);
    }
    for (Parameter parameter : invokable.getParameters()) {
      LOG.debug(
          "Adding exposed types from {}, which is a parameter on invokable {}",
          parameter,
          invokable);
      addExposedTypes(parameter, cause);
    }
    for (TypeToken<?> exceptionType : invokable.getExceptionTypes()) {
      LOG.debug(
          "Adding exposed types from {}, which is an exception type on invokable {}",
          exceptionType,
          invokable);
      addExposedTypes(exceptionType, cause);
    }
  }


Revised code:
private void addExposedTypes(Invokable<?, ?> invokable, Class<?> cause) {
    addExposedTypes(invokable.getReturnType(), cause);
    for (Annotation annotation : invokable.getAnnotations()) {
      Class<? extends Annotation> annotationClass = annotation.annotationType();
      if (!Modifier.isPublic(annotationClass.getModifiers())) {
        
        
        
        
        continue;
      }
      LOG.debug(
          "Adding exposed types from {}, which is an annotation on invokable {}",
          annotation,
          invokable);
      addExposedTypes(annotationClass, cause);
    }
    for (Parameter parameter : invokable.getParameters()) {
      LOG.debug(
          "Adding exposed types from {}, which is a parameter on invokable {}",
          parameter,
          invokable);
      addExposedTypes(parameter, cause);
    }
    for (TypeToken<?> exceptionType : invokable.getExceptionTypes()) {
      LOG.debug(
          "Adding exposed types from {}, which is an exception type on invokable {}",
          exceptionType,
          invokable);
      addExposedTypes(exceptionType, cause);
    }
  }
Code review comment: `ByteArrayOutputStream.toString(Charset)` is only available in Java versions past Java 8 which is why CI is failing validation.

Code under review:
public void logWithSupplier(LogLevel logLevel) {

        setupLogLevel(logLevel.getLogLevel());
        setPropertyToOriginalOrClear(Configuration.getGlobalConfiguration().get(PROPERTY_AZURE_LOG_LEVEL));
        Supplier<String> supplier = () -> String.format("Param 1: %s, Param 2: %s, Param 3: %s", "test1", "test2", "test3");
        logMessageWithInfoLevelForSupplier(new ClientLogger(ClientLoggerTests.class), logLevel, "", supplier);

        String logValues = logCaptureStream.toString(StandardCharsets.UTF_8);
        assertTrue(logValues.contains(supplier.get() + System.lineSeparator()));
    }

Line commented on: String logValues = logCaptureStream.toString(StandardCharsets.UTF_8);

Revised code:
public void logWithSupplier(LogLevel logLevel) throws UnsupportedEncodingException {

        setupLogLevel(logLevel.getLogLevel());
        setPropertyToOriginalOrClear(Configuration.getGlobalConfiguration().get(PROPERTY_AZURE_LOG_LEVEL));
        Supplier<String> supplier = () -> String.format("Param 1: %s, Param 2: %s, Param 3: %s", "test1", "test2", "test3");
        ClientLogger logger = new ClientLogger(ClientLoggerTests.class);
        logHelper(() -> logger.log(logLevel, supplier), (args) -> logger.log(logLevel, supplier), supplier);

        String logValues = logCaptureStream.toString("UTF-8");
        assertTrue(logValues.contains(supplier.get() + System.lineSeparator()));
    }
Code review comment: Nit: Add a period (`.`) after `SQLXMLFeature is enabled`. I.e. `... SQLXMLFeature is enabled.`

Code under review:
public void verifyJDKXMLParsersAreIncluded() {
        final ClassInclusionReport report = ClassInclusionReport.load();
        
        
        
        report.assertContains(org.postgresql.jdbc.PgSQLXML.class);
        report.assertContains(UUIDJdbcType.class);

        
        
        
        report.assertContains("com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl");
    }


Revised code:
public void verifyJDKXMLParsersAreIncluded() {
        final ClassInclusionReport report = ClassInclusionReport.load();
        
        
        
        report.assertContains(org.postgresql.jdbc.PgSQLXML.class);
        report.assertContains(UUIDJdbcType.class);

        
        
        
        
        
        
        report.assertContains("com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl");
    }
Code review comment: Or, which is IMO the best solution, in an end handler added to the `RoutingContext`?

Code under review:
private void doHandle(RoutingContext ctx, ManagedContext requestContext) {
        QuarkusHttpUser user = (QuarkusHttpUser) ctx.user();
        if (user != null) {
            Arc.container().instance(CurrentIdentityAssociation.class).get().setIdentity(user.getSecurityIdentity());
        }
        SmallRyeHealthReporter reporter = Arc.container().instance(SmallRyeHealthReporter.class).get();
        Context context = Vertx.currentContext();
        getHealth(reporter, ctx).emitOn(MutinyHelper.executor(context))
                .subscribe().with(health -> {
                    if (requestContext != null) {
                        requestContext.terminate();
                    }
                    HttpServerResponse resp = ctx.response();
                    if (health.isDown()) {
                        resp.setStatusCode(503);
                    }
                    resp.headers().set(HttpHeaders.CONTENT_TYPE, "application/json; charset=UTF-8");
                    Buffer buffer = Buffer.buffer(256); 
                    try (BufferOutputStream outputStream = new BufferOutputStream(buffer);) {
                        reporter.reportHealth(outputStream, health);
                        resp.end(buffer);
                    } catch (IOException e) {
                        throw new UncheckedIOException(e);
                    }
                });
    }

Line commented on: if (requestContext != null) {

Revised code:
private void doHandle(RoutingContext ctx, ManagedContext requestContext) {
        QuarkusHttpUser user = (QuarkusHttpUser) ctx.user();
        if (user != null) {
            Arc.container().instance(CurrentIdentityAssociation.class).get().setIdentity(user.getSecurityIdentity());
        }
        SmallRyeHealthReporter reporter = Arc.container().instance(SmallRyeHealthReporter.class).get();
        Context context = Vertx.currentContext();
        getHealth(reporter, ctx).emitOn(MutinyHelper.executor(context))
                .subscribe().with(health -> {
                    if (requestContext != null) {
                        requestContext.terminate();
                    }
                    HttpServerResponse resp = ctx.response();
                    if (health.isDown()) {
                        resp.setStatusCode(503);
                    }
                    resp.headers().set(HttpHeaders.CONTENT_TYPE, "application/json; charset=UTF-8");
                    Buffer buffer = Buffer.buffer(256); 
                    try (BufferOutputStream outputStream = new BufferOutputStream(buffer);) {
                        reporter.reportHealth(outputStream, health);
                        resp.end(buffer);
                    } catch (IOException e) {
                        throw new UncheckedIOException(e);
                    }
                }, failure -> {
                    if (requestContext != null) {
                        requestContext.terminate();
                    }
                });
    }
Code review comment: You can do an assert fail here instead with a message.

Code under review:
static void assertConfigurationEquals(ConfigurationSetting expected, ConfigurationSetting actual) {
        if (expected != null && actual != null) {
            actual = cleanResponse(expected, actual);
        } else if (expected == actual) {
            return;
        } else if (expected == null || actual == null) {
            assertTrue(false);
        }

        assertEquals(expected.getKey(), actual.getKey());
        assertEquals(expected.getLabel(), actual.getLabel());
        assertEquals(expected.getValue(), actual.getValue());
        assertEquals(expected.getETag(), actual.getETag());
        assertEquals(expected.getLastModified(), actual.getLastModified());
        assertEquals(expected.getContentType(), actual.getContentType());

        final Map<String, String> expectedTags = expected.getTags();
        final Map<String, String> actualTags = actual.getTags();

        boolean expectedIsNullOrEmpty = ImplUtils.isNullOrEmpty(expectedTags);
        boolean actualIsNullOrEmpty = ImplUtils.isNullOrEmpty(actualTags);

        if (expectedIsNullOrEmpty) {
            assertTrue(actualIsNullOrEmpty);
        } else {
            assertEquals(expectedTags, actualTags);

            assertTrue(true);
        }
    }

Line commented on: assertTrue(false);

Revised code:
static void assertConfigurationEquals(ConfigurationSetting expected, ConfigurationSetting actual) {
        if (expected != null && actual != null) {
            actual = cleanResponse(expected, actual);
        } else if (expected == actual) {
            return;
        } else if (expected == null || actual == null) {
            assertFalse("One of input settings is null", true);
        }

        equals(expected, actual);
    }
Code review comment: Hm, I think that regex is quite expensive in terms of allocations ;-).

Code under review:
String escape(CharSequence value) {
        if (Objects.requireNonNull(value).length() == 0) {
            return value.toString();
        }
        for (int i = 0; i < value.length(); i++) {
            String replacement = replacementFor(value.charAt(i));
            if (replacement != null) {
                
                return doEscape(value, i, new StringBuilder(value.subSequence(0, i)).append(replacement));
            }
        }
        return value.toString();
    }

Line commented on: for (int i = 0; i < value.length(); i++) {

Revised code:
String escape(CharSequence value) {
        if (Objects.requireNonNull(value).length() == 0) {
            return value.toString();
        }
        for (int i = 0; i < value.length(); i++) {
            String replacement = replacementFor(value.charAt(i));
            if (replacement != null) {
                
                return doEscape(value, i, new StringBuilder(value.subSequence(0, i)).append(replacement));
            }
        }
        return value.toString();
    }
Code review comment: need to cache database? how to refresh it

Code under review:
public Database getDB(String dbName) {
        if (databases.containsKey(dbName)) {
            return databases.get(dbName);
        }
        Database db;
        try {
            db = delegate.getDB(dbName);
        } catch (NoSuchNamespaceException e) {
            LOG.error("Database {} not found", dbName, e);
            return null;
        }

        databases.put(dbName, db);
        return db;
    }

Line commented on: return databases.get(dbName);

Revised code:
public Database getDB(String dbName) {
        if (databases.containsKey(dbName)) {
            return databases.get(dbName);
        }
        Database db;
        try {
            db = delegate.getDB(dbName);
        } catch (NoSuchNamespaceException e) {
            LOG.error("Database {} not found", dbName, e);
            return null;
        }

        databases.put(dbName, db);
        return db;
    }