muttakin2301/ccr2000 huggingface dataset:
Code review comment: Well, it all depends on what your service returns so I prefer to do it on a per call basis.  I'm not very familiar with this health thing so I don't know if it can also produce XML. But I decided go per call to be on the safe side.  I can change it if you prefer though.

Code under review:
public void testHealth() {
        
        try {
            RestAssured.defaultParser = Parser.JSON;
            RestAssured.when().get("/health").then()
                    .body("outcome", is("UP"),
                            "checks.state", contains("UP"),
                            "checks.name", contains("basic"));
        } finally {
            RestAssured.reset();
        }
    }

Line commented on: RestAssured.defaultParser = Parser.JSON;

Revised code:
public void testHealth() {
        
        try {
            RestAssured.defaultParser = Parser.JSON;
            RestAssured.when().get("/health").then()
                    .body("outcome", is("UP"),
                            "checks.state", contains("UP"),
                            "checks.name", contains("basic"));
        } finally {
            RestAssured.reset();
        }
    }
Code review comment: It should not be necessary to set the same value as the default in the def file. We should instead avoid setting the wrong values at higher levels in the model. The handling of `getMemoryPercentage` in `ContainerCluster.getConfig(QrStartConfig)` should be moved to the `ApplicationContainerCluster` class. Only those clusters are created by `ContainerModelBuilder`, and that's the only place where we set `memoryPercentage`.  `ContainerCluster.set/getMemoryPercentage` should also be moved to `ApplicationContainerCluster`.

Code under review:
public void getConfig(QrStartConfig.Builder builder) {
        builder.jvm.heapsize(512);
        builder.jvm.heapSizeAsPercentageOfPhysicalMemory(0);
        builder.jvm.availableProcessors(2);
        builder.jvm.verbosegc(false);
    }

Line commented on: builder.jvm.heapSizeAsPercentageOfPhysicalMemory(0);

Revised code:
public void getConfig(QrStartConfig.Builder builder) {
        builder.jvm
                .verbosegc(false)
                .availableProcessors(2)
                .heapsize(512)
                .heapSizeAsPercentageOfPhysicalMemory(0);
    }
Code review comment: I'm curious what the intention of this comment is: Doesn't it just describes what the next line does? It feels a bit obsolete. WDYT?

Code under review:
public void testWriteFile() throws Exception {
        
        File file = TEMPORARY_FOLDER.newFolder();
        Path basePath = new Path(file.toURI());

        List<String> data = Arrays.asList("first line", "second line", "third line");

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        env.enableCheckpointing(100);

        DataStream<String> stream =
                env.addSource(new FiniteTestSource<>(data), TypeInformation.of(String.class));
        Configuration configuration = new Configuration();
        
        HadoopPathBasedBulkFormatBuilder<String, String, ?> builder =
                new HadoopPathBasedBulkFormatBuilder<>(
                        basePath,
                        new TestHadoopPathBasedBulkWriterFactory(),
                        configuration,
                        new DateTimeBucketAssigner<>("yyyy-MM-dd"));
        TestStreamingFileSinkFactory<String> streamingFileSinkFactory =
                new TestStreamingFileSinkFactory<>();
        stream.addSink(streamingFileSinkFactory.createSink(builder, 1000));

        env.execute();
        validateResult(data, configuration, basePath);
    }


Revised code:
public void testWriteFile() throws Exception {
        File file = TEMPORARY_FOLDER.newFolder();
        Path basePath = new Path(file.toURI());

        List<String> data = Arrays.asList("first line", "second line", "third line");

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        env.enableCheckpointing(100);

        
        
        DataStream<String> stream =
                env.addSource(new FiniteTestSource<>(data), TypeInformation.of(String.class));
        Configuration configuration = new Configuration();
        
        HadoopPathBasedBulkFormatBuilder<String, String, ?> builder =
                new HadoopPathBasedBulkFormatBuilder<>(
                        basePath,
                        new TestHadoopPathBasedBulkWriterFactory(),
                        configuration,
                        new BasePathBucketAssigner<>());
        TestStreamingFileSinkFactory<String> streamingFileSinkFactory =
                new TestStreamingFileSinkFactory<>();
        stream.addSink(streamingFileSinkFactory.createSink(builder, 1000));

        env.execute();
        validateResult(data, configuration, basePath);
    }
Code review comment: A bit out-of-scope, but I'm wondering whether we should also assert the completion of the `CompletableFuture` that is returned by `closeAsync` here? ...just to be sure that the service has actually stopped properly after the completion of the `terminationFuture` succeeded. WDYT? :thinking:

Code under review:
void testActorTerminationWhenServiceShutdown() throws Exception {
        final ActorSystem rpcActorSystem = AkkaUtils.createDefaultActorSystem();
        final RpcService rpcService =
                new AkkaRpcService(
                        rpcActorSystem, AkkaRpcServiceConfiguration.defaultConfiguration());

        try {
            SimpleRpcEndpoint rpcEndpoint =
                    new SimpleRpcEndpoint(rpcService, SimpleRpcEndpoint.class.getSimpleName());

            rpcEndpoint.start();

            CompletableFuture<Void> terminationFuture = rpcEndpoint.getTerminationFuture();

            rpcService.closeAsync();

            terminationFuture.get();
        } finally {
            rpcActorSystem.terminate();
            AkkaFutureUtils.toJava(rpcActorSystem.whenTerminated()).get();
        }
    }

Line commented on: rpcService.closeAsync();

Revised code:
void testActorTerminationWhenServiceShutdown() throws Exception {
        final ActorSystem rpcActorSystem = AkkaUtils.createDefaultActorSystem();
        final RpcService rpcService =
                new AkkaRpcService(
                        rpcActorSystem, AkkaRpcServiceConfiguration.defaultConfiguration());

        try {
            SimpleRpcEndpoint rpcEndpoint =
                    new SimpleRpcEndpoint(rpcService, SimpleRpcEndpoint.class.getSimpleName());

            rpcEndpoint.start();

            CompletableFuture<Void> terminationFuture = rpcEndpoint.getTerminationFuture();

            rpcService.closeAsync();

            terminationFuture.get();
        } finally {
            rpcActorSystem.terminate();
            AkkaFutureUtils.toJava(rpcActorSystem.whenTerminated()).get();
        }
    }
Code review comment: We can't since it's override.

Code under review:
public void testRegisterHealthOnBlockingThreadStep1() {
        
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }

        try {
            RestAssured.defaultParser = Parser.JSON;
            
            for (int i = 0; i < 3; i++) {
                RestAssured.when().get("/q/health").then()
                        .body("status", is("UP"),
                                "checks.status", contains("UP"),
                                "checks.name", contains("blocking"));
            }
        } finally {
            RestAssured.reset();
        }

        if (!inMemoryLogHandler.getRecords().isEmpty()) {
            LogRecord logRecord = inMemoryLogHandler.getRecords().get(0);
            assertEquals(Level.WARNING, logRecord.getLevel());
            assertFalse(logRecord.getMessage().contains("has been blocked for"),
                    "The blocking health check ran on eventloop thread");
        }
    }

Line commented on: throw new RuntimeException(e);

Revised code:
public void testRegisterHealthOnBlockingThreadStep1() {
        
        given()
                .when().get("/start-health")
                .then().statusCode(200);

        try {
            RestAssured.defaultParser = Parser.JSON;
            
            for (int i = 0; i < 3; i++) {
                RestAssured.when().get("/q/health").then()
                        .body("status", is("UP"),
                                "checks.status", contains("UP"),
                                "checks.name", contains("blocking"));
            }
        } finally {
            RestAssured.reset();
        }
    }
Code review comment: Yeah, I just noticed that I deactivated this test.

Code under review:
void testMetricsInitialization() {
        
        
        
        

        
        

        
        
    }


Revised code:
void testMetricsInitialization() {
        assertNull(getGaugeValueOrNull("mongodb.connection-pool.size", getTags()));
        assertNull(getGaugeValueOrNull("mongodb.connection-pool.checked-out-count", getTags()));

        
        String name = client.listDatabaseNames().first();

        assertEquals(1L, getGaugeValueOrNull("mongodb.connection-pool.size", getTags()));
        assertEquals(0L, getGaugeValueOrNull("mongodb.connection-pool.checked-out-count", getTags()));

        client.close();
        assertEquals(0L, getGaugeValueOrNull("mongodb.connection-pool.size", getTags()));
        assertEquals(0L, getGaugeValueOrNull("mongodb.connection-pool.checked-out-count", getTags()));
    }
Code review comment: ```suggestion 		final PreferredLocationsRetriever locationsRetriever = new DefaultPreferredLocationsRetriever( ```

Code under review:
public void testStateLocationsWillBeReturnedIfExist() {
		final TaskManagerLocation stateLocation = new LocalTaskManagerLocation();

		final TestingInputsLocationsRetriever.Builder locationRetrieverBuilder =
			new TestingInputsLocationsRetriever.Builder();

		final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
		final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
		locationRetrieverBuilder.connectConsumerToProducer(consumerId, producerId);

		final TestingInputsLocationsRetriever inputsLocationsRetriever = locationRetrieverBuilder.build();

		inputsLocationsRetriever.markScheduled(producerId);

		final DefaultPreferredLocationsRetriever locationsRetriever = new DefaultPreferredLocationsRetriever(
			id -> Optional.of(stateLocation),
			inputsLocationsRetriever);

		final CompletableFuture<Collection<TaskManagerLocation>> preferredLocations =
			locationsRetriever.getPreferredLocations(consumerId, Collections.emptySet());

		assertThat(preferredLocations.getNow(null), contains(stateLocation));
	}

Line commented on: final DefaultPreferredLocationsRetriever locationsRetriever = new DefaultPreferredLocationsRetriever(

Revised code:
public void testStateLocationsWillBeReturnedIfExist() {
		final TaskManagerLocation stateLocation = new LocalTaskManagerLocation();

		final TestingInputsLocationsRetriever.Builder locationRetrieverBuilder =
			new TestingInputsLocationsRetriever.Builder();

		final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
		final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
		locationRetrieverBuilder.connectConsumerToProducer(consumerId, producerId);

		final TestingInputsLocationsRetriever inputsLocationsRetriever = locationRetrieverBuilder.build();

		inputsLocationsRetriever.markScheduled(producerId);

		final PreferredLocationsRetriever locationsRetriever = new DefaultPreferredLocationsRetriever(
			id -> Optional.of(stateLocation),
			inputsLocationsRetriever);

		final CompletableFuture<Collection<TaskManagerLocation>> preferredLocations =
			locationsRetriever.getPreferredLocations(consumerId, Collections.emptySet());

		assertThat(preferredLocations.getNow(null), contains(stateLocation));
	}
Code review comment: nit: "Flink configuration directory (%s) in environment should not be null"

Code under review:
public static void main(String[] args) {
		
		EnvironmentInformation.logEnvironmentInfo(LOG, KubernetesSessionClusterEntrypoint.class.getSimpleName(), args);
		SignalHandler.register(LOG);
		JvmShutdownSafeguard.installAsShutdownHook(LOG);

		final String configDir = System.getenv(ConfigConstants.ENV_FLINK_CONF_DIR);
		Preconditions.checkNotNull(configDir,
			"%s environment should not be null!", ConfigConstants.ENV_FLINK_CONF_DIR);
		final Configuration configuration = GlobalConfiguration.loadConfiguration(configDir);

		ClusterEntrypoint entrypoint = new KubernetesSessionClusterEntrypoint(configuration);
		ClusterEntrypoint.runClusterEntrypoint(entrypoint);
	}

Line commented on: "%s environment should not be null!", ConfigConstants.ENV_FLINK_CONF_DIR);

Revised code:
public static void main(String[] args) {
		
		EnvironmentInformation.logEnvironmentInfo(LOG, KubernetesSessionClusterEntrypoint.class.getSimpleName(), args);
		SignalHandler.register(LOG);
		JvmShutdownSafeguard.installAsShutdownHook(LOG);

		final String configDir = System.getenv(ConfigConstants.ENV_FLINK_CONF_DIR);
		Preconditions.checkNotNull(
			configDir,
			"Flink configuration directory (%s) in environment should not be null!",
			ConfigConstants.ENV_FLINK_CONF_DIR);
		final Configuration configuration = GlobalConfiguration.loadConfiguration(configDir);

		ClusterEntrypoint entrypoint = new KubernetesSessionClusterEntrypoint(configuration);
		ClusterEntrypoint.runClusterEntrypoint(entrypoint);
	}
Code review comment: Makes sense. Sounds good to me!

Code under review:
public void executeBundles(ProcessContext context) {
        String body = context.element();
        try {
          
          mapper.readTree(body);
          client.executeFhirBundle(fhirStore, body);
        } catch (IOException | HealthcareHttpException e) {
          failedBundles.inc();
          context.output(HealthcareIOError.of(body, e));
        }
      }

Line commented on: client.executeFhirBundle(fhirStore, body);

Revised code:
public void executeBundles(ProcessContext context) {
        String body = context.element();
        try {
          
          mapper.readTree(body);
          client.executeFhirBundle(fhirStore.get(), body);
        } catch (IOException | HealthcareHttpException e) {
          failedBundles.inc();
          context.output(HealthcareIOError.of(body, e));
        }
      }
Code review comment: Yes. I asked about this even with the older APIs. Their resolver on the backend checks the operation against the last path part. All the other APIs with optional versions also list them as required in swagger. You could elide the double slash (e.g. "/keys/{key-name}//export") in the URL in that case, but in my experience with the KV APIs I haven't seen it make a difference.

Code under review:
public Mono<KeyVaultKey> exportKey(String name, String environment) {
        try {
            return exportKeyWithResponse(name, "", environment).flatMap(FluxUtil::toMono);
        } catch (RuntimeException ex) {
            return monoError(logger, ex);
        }
    }

Line commented on: return exportKeyWithResponse(name, "", environment).flatMap(FluxUtil::toMono);

Revised code:
public Mono<KeyVaultKey> exportKey(String name, String environment) {
        try {
            return exportKeyWithResponse(name, "", environment).flatMap(FluxUtil::toMono);
        } catch (RuntimeException ex) {
            return monoError(logger, ex);
        }
    }
Code review comment: NIT: I'm wondering if something like ```         assertEquals(                 uploader.getBasePath().getPath(),                 String.format(                         "/%s/%s/%s",                         rootPath, jobID.toHexString(), StateChangeFsUploader.PATH_SUFFIX)); ``` will be more readable, WDYT?

Code under review:
void testBasePath() throws IOException {
        JobID jobID = JobID.generate();
        Path oriBasePath = new Path("file:

        ChangelogStorageMetricGroup metrics =
                new ChangelogStorageMetricGroup(createUnregisteredTaskManagerJobMetricGroup());

        StateChangeFsUploader uploader =
                new StateChangeFsUploader(
                        jobID,
                        oriBasePath,
                        oriBasePath.getFileSystem(),
                        false,
                        4096,
                        metrics,
                        TaskChangelogRegistry.NO_OP);

        Path basePath = uploader.getBasePath();

        assertThat(basePath.getName()).isEqualTo("dstl");
        assertThat(basePath.getParent().getName()).isEqualTo(jobID.toHexString());
        assertThat(basePath.getParent().getParent()).isEqualTo(oriBasePath);
    }

Line commented on: assertThat(basePath.getParent().getParent()).isEqualTo(oriBasePath);

Revised code:
void testBasePath() throws IOException {
        JobID jobID = JobID.generate();
        String rootPath = "/dstl-root-path";
        Path oriBasePath = new Path(rootPath);

        ChangelogStorageMetricGroup metrics =
                new ChangelogStorageMetricGroup(createUnregisteredTaskManagerJobMetricGroup());

        StateChangeFsUploader uploader =
                new StateChangeFsUploader(
                        jobID,
                        oriBasePath,
                        oriBasePath.getFileSystem(),
                        false,
                        4096,
                        metrics,
                        TaskChangelogRegistry.NO_OP);

        assertEquals(
                uploader.getBasePath().getPath(),
                String.format(
                        "%s/%s/%s",
                        rootPath, jobID.toHexString(), StateChangeFsUploader.PATH_SUB_DIR));
    }
Code review comment: We have a few that are using `@ConfigRoot.name = ConfigItem.PARENT`, e.g. `CommandLineRuntimeConfig`. But those are very special and it doesn't make sense to pass them to `ConfigInstantiator`, AFAICS.

Code under review:
public static void handleObject(Object o) {
        final SmallRyeConfig config = (SmallRyeConfig) ConfigProvider.getConfig();
        final String clsNameSuffix = getClassNameSuffix(o);
        if (clsNameSuffix == null) {
            
            return;
        }

        final Class<?> cls = o.getClass();
        final String name;
        ConfigRoot configRoot = cls.getAnnotation(ConfigRoot.class);
        if (configRoot != null && !configRoot.name().equals(ConfigItem.HYPHENATED_ELEMENT_NAME)) {
            name = configRoot.name();
            if (name.startsWith("<<")) {
                throw new IllegalArgumentException("Found unsupported @ConfigRoot.name = " + name + " on " + cls);
            }
        } else {
            name = dashify(cls.getSimpleName().substring(0, cls.getSimpleName().length() - clsNameSuffix.length()));
        }
        handleObject(QUARKUS_PROPERTY_PREFIX + name, o, config, gatherQuarkusPropertyNames(config));
    }

Line commented on: throw new IllegalArgumentException("Found unsupported @ConfigRoot.name = " + name + " on " + cls);

Revised code:
public static void handleObject(Object o) {
        final SmallRyeConfig config = (SmallRyeConfig) ConfigProvider.getConfig();
        final String clsNameSuffix = getClassNameSuffix(o);
        if (clsNameSuffix == null) {
            
            return;
        }

        final Class<?> cls = o.getClass();
        final String name;
        ConfigRoot configRoot = cls.getAnnotation(ConfigRoot.class);
        if (configRoot != null && !configRoot.name().equals(ConfigItem.HYPHENATED_ELEMENT_NAME)) {
            name = configRoot.name();
            if (name.startsWith("<<")) {
                throw new IllegalArgumentException("Found unsupported @ConfigRoot.name = " + name + " on " + cls);
            }
        } else {
            name = dashify(cls.getSimpleName().substring(0, cls.getSimpleName().length() - clsNameSuffix.length()));
        }
        handleObject(QUARKUS_PROPERTY_PREFIX + name, o, config, gatherQuarkusPropertyNames(config));
    }
Code review comment: I think these can be easily supported by passing in the `tableIdentifier`.

Code under review:
public String tableName() {
            throw new UnsupportedOperationException(
                    "Hive batch sink should not call PartitionCommitPolicy.Context
        }

Line commented on: "Hive batch sink should not call PartitionCommitPolicy.Context

Revised code:
public String tableName() {
            return identifier.getObjectName();
        }
Code review comment: > Why here are you endInput for the head operator and inside the task for the rest of the operators?  The reason is that we don't want `xInputProcessor` to see `OperatorChain` (or other operators). If we change this point and let `xInputProcessor` see `OperatorChain` (or other operators), we can do all works in `xInputProcessor`. What do you think?

Code under review:
private boolean checkFinished() throws Exception {
		boolean isFinished = input.isFinished();
		if (isFinished) {
			if (streamOperator instanceof BoundedOneInput) {
				synchronized (lock) {
					((BoundedOneInput) streamOperator).endInput();
				}
			}
		}
		return isFinished;
	}

Line commented on: if (streamOperator instanceof BoundedOneInput) {

Revised code:
private boolean checkFinished() throws Exception {
		boolean isFinished = input.isFinished();
		if (isFinished) {
			synchronized (lock) {
				operatorChain.endInput(1);
			}
		}
		return isFinished;
	}
Code review comment: This is a bit unrelated to this change. But it looks to me that this test is more of an integration test which could go into `flink-tests` because we use things from `flink-runtime`.

Code under review:
public void setupReporter() {
        registry =
                new MetricRegistryImpl(
                        MetricRegistryTestUtils.defaultMetricRegistryConfiguration(),
                        Collections.singletonList(
                                createReporterSetup("test1", portRangeProvider.next())));
        metricGroup =
                new FrontMetricGroup<>(
                        createReporterScopedSettings(),
                        new TaskManagerMetricGroup(registry, HOST_NAME, TASK_MANAGER));
        reporter = (PrometheusReporter) registry.getReporters().get(0);
    }

Line commented on: MetricRegistryTestUtils.defaultMetricRegistryConfiguration(),

Revised code:
public void setupReporter() {
        registry =
                new MetricRegistryImpl(
                        MetricRegistryTestUtils.defaultMetricRegistryConfiguration(),
                        Collections.singletonList(
                                createReporterSetup("test1", portRangeProvider.next())));
        metricGroup =
                new FrontMetricGroup<>(
                        createReporterScopedSettings(),
                        new TaskManagerMetricGroup(registry, HOST_NAME, TASK_MANAGER));
        reporter = (PrometheusReporter) registry.getReporters().get(0);
    }
Code review comment: `dataChangeType` should be `UPDATE` or `DELETE`

Code under review:
public static Record convertDataRecordToRecord(final String database, final String schema, final DataRecord dataRecord) {
        Map<String, Any> beforeMap = new HashMap<>();
        Map<String, Any> afterMap = new HashMap<>();
        for (Column column : dataRecord.getColumns()) {
            beforeMap.put(column.getName(), Any.pack(ColumnValueConvertUtil.convertToProtobufMessage(column.getOldValue())));
            afterMap.put(column.getName(), Any.pack(ColumnValueConvertUtil.convertToProtobufMessage(column.getValue())));
        }
        TableMetaData metaData = TableMetaData.newBuilder().setDatabase(database).setSchema(Strings.nullToEmpty(schema)).setTableName(dataRecord.getTableName()).build();
        DataChangeType dataChangeType = DataChangeType.UNKNOWN;
        if (IngestDataChangeType.INSERT.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.INSERT;
        } else if (IngestDataChangeType.UPDATE.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.INSERT;
        } else if (IngestDataChangeType.DELETE.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.INSERT;
        }
        return DataRecordResult.Record.newBuilder().setTableMetaData(metaData).putAllBefore(beforeMap).putAllAfter(afterMap).setDataChangeType(dataChangeType).build();
    }

Line commented on: dataChangeType = DataChangeType.INSERT;

Revised code:
public static Record convertDataRecordToRecord(final String database, final String schema, final DataRecord dataRecord) {
        Map<String, Any> beforeMap = new HashMap<>();
        Map<String, Any> afterMap = new HashMap<>();
        for (Column column : dataRecord.getColumns()) {
            beforeMap.put(column.getName(), Any.pack(ColumnValueConvertUtil.convertToProtobufMessage(column.getOldValue())));
            afterMap.put(column.getName(), Any.pack(ColumnValueConvertUtil.convertToProtobufMessage(column.getValue())));
        }
        TableMetaData metaData = TableMetaData.newBuilder().setDatabase(database).setSchema(Strings.nullToEmpty(schema)).setTableName(dataRecord.getTableName()).build();
        DataChangeType dataChangeType = DataChangeType.UNKNOWN;
        if (IngestDataChangeType.INSERT.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.INSERT;
        } else if (IngestDataChangeType.UPDATE.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.UPDATE;
        } else if (IngestDataChangeType.DELETE.equals(dataRecord.getType())) {
            dataChangeType = DataChangeType.DELETE;
        }
        return DataRecordResult.Record.newBuilder().setTableMetaData(metaData).putAllBefore(beforeMap).putAllAfter(afterMap).setDataChangeType(dataChangeType).build();
    }
Code review comment: Is there a difference in asserting the values here vs. within the test case itself? Maybe all `LocksWithOnFailTest`s could follow same structure for consistency.

Code under review:
public void testOnFailLockWithinLockWithoutVariable() {
        BRunUtil.invoke(compileResult, "onFailLockWithinLockWithoutVariable");
    }

Line commented on: BRunUtil.invoke(compileResult, "onFailLockWithinLockWithoutVariable");

Revised code:
public void testOnFailLockWithinLockWithoutVariable() {
        BRunUtil.invoke(compileResult, "onFailLockWithinLockWithoutVariable");
    }
Code review comment: how about add it to workerToBackend map in this case.

Code under review:
public List<Long> getWorkersByWorkerGroup(long workerGroupId) throws UserException {
        List<Long> nodeIds = new ArrayList<>();
        prepare();
        try {
            List<WorkerGroupDetailInfo> workerGroupDetailInfos = client.
                    listWorkerGroup(serviceId, Collections.singletonList(workerGroupId), true);
            for (WorkerGroupDetailInfo detailInfo : workerGroupDetailInfos) {
                List<WorkerInfo> workerInfos = detailInfo.getWorkersInfoList();
                for (WorkerInfo workerInfo : workerInfos) {
                    if (workerToBackend.containsKey(workerInfo.getWorkerId())) {
                        nodeIds.add(workerToBackend.get(workerInfo.getWorkerId()));
                    } else {
                        
                        String workerAddr = workerInfo.getIpPort();
                        String[] pair = workerAddr.split(":");
                        long nodeId = getAvailableBackendId(pair[0], Integer.parseInt(pair[1]));
                        nodeIds.add(nodeId);
                    }
                }
            }
            return nodeIds;
        } catch (StarClientException e) {
            throw new UserException("Failed to get workers by group id. error: " + e.getMessage());
        }
    }

Line commented on: nodeIds.add(nodeId);

Revised code:
public List<Long> getWorkersByWorkerGroup(long workerGroupId) throws UserException {
        List<Long> nodeIds = new ArrayList<>();
        prepare();
        try {
            List<WorkerGroupDetailInfo> workerGroupDetailInfos = client.
                    listWorkerGroup(serviceId, Collections.singletonList(workerGroupId), true);
            for (WorkerGroupDetailInfo detailInfo : workerGroupDetailInfos) {
                List<WorkerInfo> workerInfos = detailInfo.getWorkersInfoList();
                for (WorkerInfo workerInfo : workerInfos) {
                    if (workerToBackend.containsKey(workerInfo.getWorkerId())) {
                        nodeIds.add(workerToBackend.get(workerInfo.getWorkerId()));
                    } else {
                        
                        
                        long workerId = workerInfo.getWorkerId();
                        String workerAddr = workerInfo.getIpPort();
                        String[] pair = workerAddr.split(":");
                        long nodeId = getAvailableBackendId(pair[0], Integer.parseInt(pair[1]));
                        if (nodeId != -1L) {
                            nodeIds.add(nodeId);

                            
                            workerToId.put(workerAddr, workerId);
                            workerToBackend.put(workerId, nodeId);
                        }
                    }
                }
            }
            return nodeIds;
        } catch (StarClientException e) {
            throw new UserException("Failed to get workers by group id. error: " + e.getMessage());
        }
    }
Code review comment: I don't understand what is etag locking so feel free to contribute. What kind of progress reporting are you looking for?

Code under review:
private Mono<BlobRange> getFullBlobRange(BlobAccessConditions accessConditions) {
        return getProperties(accessConditions).map(rb -> new BlobRange(0, rb.value().blobSize()));
    }

Line commented on: return getProperties(accessConditions).map(rb -> new BlobRange(0, rb.value().blobSize()));

Revised code:
private Mono<BlobRange> getFullBlobRange(BlobAccessConditions accessConditions) {
        return getProperties(accessConditions).map(rb -> new BlobRange(0, rb.value().blobSize()));
    }
Code review comment: @hasithaa Here file path is correctly computate. Problem here is when we pass file path to ballerina as parameter, we need to put \\\\ in windows environment.   ex. ballerina run <balFile> -e stateFilePath=C:\\\\user\\\\tempfileDir  AFAIU above commit logic does not solve that.

Code under review:
public void setup() {
        balFilePath = new File("src" + File.separator + "test" + File.separator +
                                       "resources" + File.separator + "checkpointing" + File.separator +
                                       "interruptibleService.bal").getAbsolutePath();
        File statesStorageDir = new File("target" + File.separator + "ballerina-states");
        if (statesStorageDir.exists()) {
            statesStorageDir.delete();
        }
        String statesStoragePath = statesStorageDir.getAbsolutePath();
        String osName = Utils.getOSName();
        if (osName != null && osName.toLowerCase(Locale.ENGLISH).contains("windows")) {
            statesStoragePath = statesStoragePath.replace("\\", "\\\\");
        }
        args = new String[] { "-e", FileStorageProvider.INTERRUPTIBLE_STATES_FILE_PATH + "=" + statesStoragePath };
        ConfigRegistry.getInstance().addConfiguration(FileStorageProvider.INTERRUPTIBLE_STATES_FILE_PATH,
                                                      statesStoragePath);
        fileStorageProvider = new FileStorageProvider();
    }

Line commented on: statesStoragePath = statesStoragePath.replace("\\", "\\\\");

Revised code:
public void setup() {
        balFilePath = new File("src" + File.separator + "test" + File.separator +
                                       "resources" + File.separator + "checkpointing" + File.separator +
                                       "interruptibleService.bal").getAbsolutePath();
        File statesStorageDir = new File("target" + File.separator + "ballerina-states");
        if (statesStorageDir.exists()) {
            statesStorageDir.delete();
        }
        String statesStoragePath = statesStorageDir.getAbsolutePath();
        String osName = Utils.getOSName();
        if (osName != null && osName.toLowerCase(Locale.ENGLISH).contains("windows")) {
            statesStoragePath = statesStoragePath.replace("\\", "\\\\");
        }
        args = new String[] { "-e", FileStorageProvider.INTERRUPTIBLE_STATES_FILE_PATH + "=" + statesStoragePath };
        ConfigRegistry.getInstance().addConfiguration(FileStorageProvider.INTERRUPTIBLE_STATES_FILE_PATH,
                                                      statesStoragePath);
        fileStorageProvider = new FileStorageProvider();
    }
Code review comment: Why it's not decoded here?

Code under review:
void shouldDetermineUrlViaStorkWhenUsingTarget() throws URISyntaxException {
        String greeting = ClientBuilder.newClient().target("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target(new URI("stork:
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target(UriBuilder.fromUri("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, big%20bird");
    }

Line commented on: assertThat(greeting).isEqualTo("Hello, big%20bird");

Revised code:
void shouldDetermineUrlViaStorkWhenUsingTarget() throws URISyntaxException {
        String greeting = ClientBuilder.newClient().target("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target(new URI("stork:
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target(UriBuilder.fromUri("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, World!");

        greeting = ClientBuilder.newClient().target("stork:
                .get(String.class);
        assertThat(greeting).isEqualTo("Hello, big bird");
    }
Code review comment: Use "try-with-resource" to ensure all streams are closed if an exception is thrown during read/write.

Code under review:
private void compress(File oldFile) {
        File gzippedFile = new File(oldFile.getPath() + ".gz");
        try {
            long mtime = oldFile.lastModified();
            GZIPOutputStream compressor = new GZIPOutputStream(new FileOutputStream(gzippedFile), 0x100000);
            FileInputStream inputStream = new FileInputStream(oldFile);
            byte [] buffer = new byte[0x100000];

            for (int read = inputStream.read(buffer); read > 0; read = inputStream.read(buffer)) {
                compressor.write(buffer, 0, read);
            }
            inputStream.close();
            compressor.finish();
            compressor.flush();
            compressor.close();
            oldFile.delete();
            gzippedFile.setLastModified(mtime);
            log.info("Compressed: "+gzippedFile);
        } catch (IOException e) {
            log.warning("Got '" + e + "' while compressing '" + oldFile.getPath() + "'.");
        }
    }

Line commented on: GZIPOutputStream compressor = new GZIPOutputStream(new FileOutputStream(gzippedFile), 0x100000);

Revised code:
private void compress(File oldFile) {
        File gzippedFile = new File(oldFile.getPath() + ".gz");
        try {
            long mtime = oldFile.lastModified();
            GZIPOutputStream compressor = new GZIPOutputStream(new FileOutputStream(gzippedFile), 0x100000);
            FileInputStream inputStream = new FileInputStream(oldFile);
            byte [] buffer = new byte[0x100000];

            for (int read = inputStream.read(buffer); read > 0; read = inputStream.read(buffer)) {
                compressor.write(buffer, 0, read);
            }
            inputStream.close();
            compressor.finish();
            compressor.flush();
            compressor.close();
            oldFile.delete();
            gzippedFile.setLastModified(mtime);
            log.info("Compressed: "+gzippedFile);
        } catch (IOException e) {
            log.warning("Got '" + e + "' while compressing '" + oldFile.getPath() + "'.");
        }
    }
Code review comment: Any reason for closing here while we're using a try-with-resource block which will close this after we leave the scope?

Code under review:
protected boolean runInternal(Context span) throws IOException {
        try (CrcOutputStream outputStream = new CrcOutputStream()) {
            syncClient.downloadStreamWithResponse(outputStream, null, null, null, false, null, span);
            outputStream.close();
            return ORIGINAL_CONTENT.checkMatch(outputStream.getContentInfo(), span).block();
        }
    }

Line commented on: outputStream.close();

Revised code:
protected boolean runInternal(Context span) throws IOException {
        try (CrcOutputStream outputStream = new CrcOutputStream()) {
            syncClient.downloadStreamWithResponse(outputStream, null, null, null, false, null, span);
            outputStream.close();
            return ORIGINAL_CONTENT.checkMatch(outputStream.getContentInfo(), span).block();
        }
    }
Code review comment: >   One other reason that could explain that it works is that some execution engines write at parallelism one resulting into a single instance of the write DoFn

Code under review:
public void teardown() throws Exception {
        if (producer != null) {
          if (producer.getOutstandingRecordsCount() > 0) {
            producer.flushSync();
          }
          producer.destroy();
        }
        producer = null;
      }

Line commented on: producer = null;

Revised code:
public void teardown() throws Exception {
        teardownSharedProducer();
      }
Code review comment: Same here, make sure the expectNextCount = 0

Code under review:
public void testDeleteByIdAndPartitionKey() {
        final Mono<Void> deleteMono = repository.deleteById(DOMAIN_1.getNumber(),
            new PartitionKey(entityInformation.getPartitionKeyFieldValue(DOMAIN_1)));
        StepVerifier.create(deleteMono).verifyComplete();

        Mono<LongIdDomainPartition> findIdMono = this.repository.findById(ID_1,
            new PartitionKey(entityInformation.getPartitionKeyFieldValue(DOMAIN_1)));
        StepVerifier.create(findIdMono).verifyComplete();
    }

Line commented on: StepVerifier.create(findIdMono).verifyComplete();

Revised code:
public void testDeleteByIdAndPartitionKey() {
        final Mono<Void> deleteMono = repository.deleteById(DOMAIN_1.getNumber(),
            new PartitionKey(entityInformation.getPartitionKeyFieldValue(DOMAIN_1)));
        StepVerifier.create(deleteMono).verifyComplete();

        Mono<LongIdDomainPartition> findIdMono = this.repository.findById(ID_1,
            new PartitionKey(entityInformation.getPartitionKeyFieldValue(DOMAIN_1)));
        StepVerifier.create(findIdMono).expectNextCount(0).verifyComplete();
    }
Code review comment: this pr only focus on internalCatalog create external table, not for external catalog create table such as iceberg sink

Code under review:
public void analyze(boolean isOlap, boolean isInternalCatalog, String engineName) throws AnalysisException {
        if (isInternalCatalog) {
            if (!isOlap && !isAllowNull) {
                
                if (engineName.equalsIgnoreCase(EngineType.HIVE.name()) ||
                        engineName.equalsIgnoreCase(EngineType.ICEBERG.name()) ||
                        engineName.equalsIgnoreCase(EngineType.HUDI.name()) ||
                        engineName.equalsIgnoreCase(EngineType.ELASTICSEARCH.name()) ||
                        engineName.equalsIgnoreCase(EngineType.JDBC.name()) ||
                        engineName.equalsIgnoreCase(EngineType.FILE.name())) {
                    throw new AnalysisException(String.format("All columns must be nullable for external table. " +
                            "Column %s is not nullable, You can rebuild the external table and " +
                            "We strongly recommend that you use catalog to access external data", name));
                }
            }
        }
        analyze(isOlap);
    }

Line commented on: if (isInternalCatalog) {

Revised code:
public void analyze(boolean isOlap, boolean isInternalCatalog, String engineName) throws AnalysisException {
        if (isInternalCatalog) {
            if (!isAllowNull && !EngineType.supportNotNullColumn(engineName)) {
                
                throw new AnalysisException(String.format("All columns must be nullable for external table. " +
                        "Column %s is not nullable, You can rebuild the external table and " +
                        "We strongly recommend that you use catalog to access external data", name));
            }
        }
        analyze(isOlap);
    }
Code review comment: Unfortunately, it would depend to vendor error. There's no generic way to detect the `SQLException` is related to a deadlock. We would need to add parsing the `SQLException` depending of the vendor (for instance, detect a `ORA-00060` for Oracle, a `1205` for SQLServer).

Code under review:
public PCollection<T> expand(PCollection<T> input) {
      
      
      
      
      
      
      
      
      
      
      PCollectionView<Iterable<T>> empty =
          input
              .apply("Consume", Filter.by(SerializableFunctions.constant(false)))
              .apply(View.asIterable());
      PCollection<T> materialized =
          input.apply(
              "Identity",
              ParDo.of(
                      new DoFn<T, T>() {
                        @ProcessElement
                        public void process(ProcessContext c) {
                          c.output(c.element());
                        }
                      })
                  .withSideInputs(empty));
      return materialized.apply(Reshuffle.viaRandomKey());
    }


Revised code:
public PCollection<T> expand(PCollection<T> input) {
      
      
      
      
      
      
      
      
      
      
      PCollectionView<Iterable<T>> empty =
          input
              .apply("Consume", Filter.by(SerializableFunctions.constant(false)))
              .apply(View.asIterable());
      PCollection<T> materialized =
          input.apply(
              "Identity",
              ParDo.of(
                      new DoFn<T, T>() {
                        @ProcessElement
                        public void process(ProcessContext c) {
                          c.output(c.element());
                        }
                      })
                  .withSideInputs(empty));
      return materialized.apply(Reshuffle.viaRandomKey());
    }
Code review comment: `serverSideFeature.getConditions().getClientFilters().size() > 0` is allowed to be 0. This should result in a feature flag returning false when enabled, as no feature filter returned true.  This is how the current library works.

Code under review:
private void addServerSideFeature(Map<? extends String, ? extends Object> features, String key) {
        final Object featureValue = features.get(key);

        ServerSideFeature serverSideFeature = null;
        try {
            serverSideFeature = MAPPER.convertValue(featureValue, ServerSideFeature.class);
        } catch (IllegalArgumentException e) {
            LOGGER.error("Found invalid feature {} with value {}.", key, featureValue.toString());
        }

        if (serverSideFeature != null && serverSideFeature.getId() != null) {
            if (serverSideFeature.getConditions() != null && serverSideFeature.getConditions().getClientFilters() != null
                && serverSideFeature.getConditions().getClientFilters().size() > 0) {
                final Feature feature = new Feature();
                feature.setKey(serverSideFeature.getId());
                feature.setEvaluate(serverSideFeature.isEnabled());
                feature.setEnabledFor(serverSideFeature.getConditions().getClientFilters());
                feature.setRequirementType(serverSideFeature.getConditions().getRequirementType());
                featureManagement.put(serverSideFeature.getId(), feature);
            } else {
                onOff.put(serverSideFeature.getId(), serverSideFeature.isEnabled());
            }
        }
    }

Line commented on: if (serverSideFeature.getConditions() != null && serverSideFeature.getConditions().getClientFilters() != null

Revised code:
private void addServerSideFeature(Map<? extends String, ? extends Object> features, String key) {
        final Object featureValue = features.get(key);

        ServerSideFeature serverSideFeature = null;
        try {
            serverSideFeature = MAPPER.convertValue(featureValue, ServerSideFeature.class);
        } catch (IllegalArgumentException e) {
            LOGGER.error("Found invalid feature {} with value {}.", key, featureValue.toString());
        }

        if (serverSideFeature != null && serverSideFeature.getId() != null) {
            if (serverSideFeature.getConditions() != null && serverSideFeature.getConditions().getClientFilters() != null) {
                final Feature feature = new Feature();
                feature.setKey(serverSideFeature.getId());
                feature.setEvaluate(serverSideFeature.isEnabled());
                feature.setEnabledFor(serverSideFeature.getConditions().getClientFilters());
                feature.setRequirementType(serverSideFeature.getConditions().getRequirementType());
                featureManagement.put(serverSideFeature.getId(), feature);
            } else {
                onOff.put(serverSideFeature.getId(), serverSideFeature.isEnabled());
            }
        }
    }
Code review comment: It's not. There's an example described in the comments. ```     //   if current offset is {0:11, 1:22, 2:33, 3:55}, the parameter `kafkaPartitionOffsets` is {0: 22, 2:44}     //   in this function, current offset is modified as {0: 22, 1: 22, 2:44, 3:55}     //   so is the parameter `kafkaPartitionOffsets`. ```

Code under review:
public void modifyOffset(List<Pair<Integer, Long>> kafkaPartitionOffsets) throws DdlException {
        for (Pair<Integer, Long> pair : kafkaPartitionOffsets) {
            if (!partitionIdToOffset.containsKey(pair.first)) {
                throw new DdlException("The specified partition " + pair.first + " is not in the consumed partitions");
            }
        }
        for (Pair<Integer, Long> pair : kafkaPartitionOffsets) {
            partitionIdToOffset.put(pair.first, pair.second);
        }
        
        for (Integer partitionId : partitionIdToOffset.keySet()) {
            Pair<Integer, Long> pair = new Pair<>(partitionId, partitionIdToOffset.get(partitionId));
            if (!kafkaPartitionOffsets.contains(pair)) {
                LOG.info("add {} to kafkaPartitionOffsets {}", pair, kafkaPartitionOffsets);
                kafkaPartitionOffsets.add(pair);
            }
        }
    }


Revised code:
public void modifyOffset(List<Pair<Integer, Long>> kafkaPartitionOffsets) throws DdlException {
        for (Pair<Integer, Long> pair : kafkaPartitionOffsets) {
            if (!partitionIdToOffset.containsKey(pair.first)) {
                throw new DdlException("The specified partition " + pair.first + " is not in the consumed partitions");
            }
        }
        for (Pair<Integer, Long> pair : kafkaPartitionOffsets) {
            partitionIdToOffset.put(pair.first, pair.second);
        }
        
        for (Integer partitionId : partitionIdToOffset.keySet()) {
            Pair<Integer, Long> pair = new Pair<>(partitionId, partitionIdToOffset.get(partitionId));
            if (!kafkaPartitionOffsets.contains(pair)) {
                LOG.info("add {} to kafkaPartitionOffsets {}", pair, kafkaPartitionOffsets);
                kafkaPartitionOffsets.add(pair);
            }
        }
    }