muttakin2301/ccr2000 huggingface dataset:
Code review comment: Yes, that sounds like a good split and would exactly help with that problem. The best way to handle this PR-wise would be: - create a JIRA for splitting the interface into something like and JM/TM side. - open PR for that change. - have this PR rebased, based on the interface split-PR.  We should keep the cost/benefit ratio in mind. If you think rebasing would take you a lot of time (I think it should not) then just do it in this PR. Otherwise, I suggest the above strategy.

Code under review:
public CheckpointStorageLocation initializeLocationForCheckpoint(long checkpointId) throws IOException {
		checkArgument(checkpointId >= 0);

		
		final Path checkpointDir = createCheckpointDirectory(checkpointsDirectory, checkpointId);

		if (!areDirectoriesCreated) {
			fileSystem.mkdirs(checkpointsDirectory);
			fileSystem.mkdirs(sharedStateDirectory);
			fileSystem.mkdirs(taskOwnedStateDirectory);
			areDirectoriesCreated = true;
		}

		
		fileSystem.mkdirs(checkpointDir);

		return new FsCheckpointStorageLocation(
				fileSystem,
				checkpointDir,
				sharedStateDirectory,
				taskOwnedStateDirectory,
				CheckpointStorageLocationReference.getDefault(),
				fileSizeThreshold);
	}

Line commented on: fileSystem.mkdirs(sharedStateDirectory);

Revised code:
public CheckpointStorageLocation initializeLocationForCheckpoint(long checkpointId) throws IOException {
		checkArgument(checkpointId >= 0, "Illegal negative checkpoint id: %d.", checkpointId);
		checkArgument(baseLocationsInitialized, "The base checkpoint location has not been initialized.");

		
		final Path checkpointDir = createCheckpointDirectory(checkpointsDirectory, checkpointId);

		
		fileSystem.mkdirs(checkpointDir);

		return new FsCheckpointStorageLocation(
				fileSystem,
				checkpointDir,
				sharedStateDirectory,
				taskOwnedStateDirectory,
				CheckpointStorageLocationReference.getDefault(),
				fileSizeThreshold,
				writeBufferSize);
	}
Code review comment: Ok, so it does fail, but it looks like this method does not throw on exit code != 0, so should be ok...

Code under review:
private void pushMetricsToContainer(NodeAgentContext context, List<DimensionMetrics> metrics) {
        StringBuilder params = new StringBuilder();
        try {
            for (DimensionMetrics dimensionMetrics : metrics) {
                params.append(dimensionMetrics.toSecretAgentReport());
            }
            String wrappedMetrics = "s:" + params.toString();

            
            runPushMetricsCommand(context, wrappedMetrics, true);
            runPushMetricsCommand(context, wrappedMetrics, false);
        } catch (DockerExecTimeoutException | JsonProcessingException  e) {
            context.log(logger, LogLevel.WARNING, "Failed to push metrics to container", e);
        }
    }

Line commented on: runPushMetricsCommand(context, wrappedMetrics, true);

Revised code:
private void pushMetricsToContainer(NodeAgentContext context, List<DimensionMetrics> metrics) {
        StringBuilder params = new StringBuilder();
        try {
            for (DimensionMetrics dimensionMetrics : metrics) {
                params.append(dimensionMetrics.toSecretAgentReport());
            }
        } catch (JsonProcessingException e) {
            
            context.log(logger, LogLevel.WARNING, "Failed to wrap metrics in secret agent report", e);
            return;
        }
        String wrappedMetrics = "s:" + params.toString();

        
        runPushMetricsCommand(context, wrappedMetrics, true);
        runPushMetricsCommand(context, wrappedMetrics, false);
    }
Code review comment: The default bucket name is a concatenation of a fixed prefix, region and a project number. So for a given project, if it is running in the same region, the default bucket name would be the same.  For our integration test, it would always be the same as "gcp-options-it-us-central1-844138762903". Actually, this may cause race condition if we run this integration test twice at the same time though.

Code under review:
public void testCreateDefaultBucket() throws IOException {
    TestPipelineOptions options =
        TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);

    CloudResourceManager crmClient =
        GcpOptions.GcpTempLocationFactory.newCloudResourceManagerClient(
                options.as(CloudResourceManagerOptions.class))
            .build();

    GcsOptions gcsOptions = options.as(GcsOptions.class);
    GcsUtil gcsUtil = gcsOptions.getGcsUtil();

    String tempLocation =
        GcpOptions.GcpTempLocationFactory.tryCreateDefaultBucketWithPrefix(
            options, crmClient, "gcp-options-it-");

    GcsPath gcsPath = GcsPath.fromUri(tempLocation);
    System.out.println(gcsPath);

    Bucket bucket = gcsUtil.getBucket(gcsPath);
    assertNotNull(bucket);
    
    assertEquals(bucket.getSoftDeletePolicy().getRetentionDurationSeconds(), Long.valueOf(0L));

    gcsUtil.removeBucket(bucket);
  }

Line commented on: options, crmClient, "gcp-options-it-");

Revised code:
public void testCreateDefaultBucket() throws IOException {
    TestPipelineOptions options =
        TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);

    CloudResourceManager crmClient =
        GcpOptions.GcpTempLocationFactory.newCloudResourceManagerClient(
                options.as(CloudResourceManagerOptions.class))
            .build();

    GcsOptions gcsOptions = options.as(GcsOptions.class);
    GcsUtil gcsUtil = gcsOptions.getGcsUtil();

    Random rand = new Random();
    
    
    
    String bucketNamePrefix = "gcp-options-it-" + rand.nextInt(1000);

    String bucketName =
        String.join(
            "-",
            GcpOptions.GcpTempLocationFactory.getDefaultBucketNameStubs(
                options, crmClient, bucketNamePrefix));

    
    try {
      Bucket oldBucket = gcsUtil.getBucket(GcsPath.fromUri("gs:
      gcsUtil.removeBucket(oldBucket);
    } catch (FileNotFoundException e) {
      
    }

    String tempLocation =
        GcpOptions.GcpTempLocationFactory.tryCreateDefaultBucketWithPrefix(
            options, crmClient, bucketNamePrefix);

    GcsPath gcsPath = GcsPath.fromUri(tempLocation);
    Bucket bucket = gcsUtil.getBucket(gcsPath);
    assertNotNull(bucket);
    
    assertEquals(bucket.getSoftDeletePolicy().getRetentionDurationSeconds(), Long.valueOf(0L));

    gcsUtil.removeBucket(bucket);
    assertThrows(FileNotFoundException.class, () -> gcsUtil.getBucket(gcsPath));
  }
Code review comment: Busy sleep is generally not preferable. Consider use wait/notify which sounds exactly what we need here? That is block this thread until future1 reaches while(true)

Code under review:
public void testCloseVisibleToAwaitCompletionCallerAndProducer() throws Exception {
    BeamFnDataInboundObserver observer =
        BeamFnDataInboundObserver.forConsumers(
            Arrays.asList(DataEndpoint.create(TRANSFORM_ID, CODER, (value) -> {})),
            Collections.emptyList());

    Future<?> future =
        executor.submit(
            () -> {
              observer.accept(dataWith("ABC"));
              assertThrows(
                  BeamFnDataInboundObserver.CloseException.class,
                  () -> {
                    while (true) {
                      
                      
                      
                      observer.accept(dataWith("ABC"));
                    }
                  });
              return null;
            });
    Future<?> future2 =
        executor.submit(
            () -> {
              Thread.sleep(500); 
              observer.close();
              return null;
            });

    assertThrows(BeamFnDataInboundObserver.CloseException.class, () -> observer.awaitCompletion());
    future.get();
    future2.get();
  }

Line commented on: Thread.sleep(500);

Revised code:
public void testCloseVisibleToAwaitCompletionCallerAndProducer() throws Exception {
    BeamFnDataInboundObserver observer =
        BeamFnDataInboundObserver.forConsumers(
            Arrays.asList(DataEndpoint.create(TRANSFORM_ID, CODER, (value) -> {})),
            Collections.emptyList());

    AtomicBoolean isReady = new AtomicBoolean(false);
    Future<?> future =
        executor.submit(
            () -> {
              observer.accept(dataWith("ABC"));
              synchronized (isReady) {
                isReady.set(true);
                isReady.notify();
              }
              assertThrows(
                  BeamFnDataInboundObserver.CloseException.class,
                  () -> {
                    while (true) {
                      
                      
                      
                      observer.accept(dataWith("ABC"));
                    }
                  });
              return null;
            });
    Future<?> future2 =
        executor.submit(
            () -> {
              synchronized (isReady) {
                while (!isReady.get()) {
                  isReady.wait();
                }
              }
              observer.close();
              return null;
            });

    assertThrows(BeamFnDataInboundObserver.CloseException.class, () -> observer.awaitCompletion());
    future.get();
    future2.get();
  }
Code review comment: @geoand see this comment in HV where we try to size properly all the collections that are in the metadata: https://github.com/hibernate/hibernate-validator/blob/master/engine/src/main/java/org/hibernate/validator/internal/util/CollectionHelper.java#L130  It's not an issue with `ArrayList` as it sizes the array exactly with the size you pass. In any case, it's a detail.

Code under review:
public HashSet<T> apply(final int value) {
        return new HashSet<>(value);
    }

Line commented on: return new HashSet<>(value);

Revised code:
public HashSet<T> apply(final int value) {
        return new HashSet<>(getInitialCapacityFromExpectedSize(value));
    }
Code review comment: Ah thanks now it make sense 🙏

Code under review:
public PojoSerializer<T> duplicate() {
        TypeSerializer<Object>[] duplicateFieldSerializers = duplicateSerializers(fieldSerializers);
        TypeSerializer<Object>[] duplicateRegisteredSerializers =
                duplicateSerializers(registeredSerializers);

        
        
        return new PojoSerializer<>(
                clazz,
                fields,
                duplicateFieldSerializers,
                new LinkedHashMap<>(registeredClasses),
                duplicateRegisteredSerializers,
                subclassSerializerCache.entrySet().stream()
                        .collect(
                                Collectors.toMap(Map.Entry::getKey, e -> e.getValue().duplicate())),
                executionConfig);
    }


Revised code:
public PojoSerializer<T> duplicate() {
        TypeSerializer<Object>[] duplicateFieldSerializers = duplicateSerializers(fieldSerializers);
        TypeSerializer<Object>[] duplicateRegisteredSerializers =
                duplicateSerializers(registeredSerializers);

        return new PojoSerializer<>(
                clazz,
                fields,
                duplicateFieldSerializers,
                new LinkedHashMap<>(registeredClasses),
                duplicateRegisteredSerializers,
                subclassSerializerCache.entrySet().stream()
                        .collect(
                                Collectors.toMap(Map.Entry::getKey, e -> e.getValue().duplicate())),
                executionConfig);
    }
Code review comment: Since this is runtime validation, this is not a problem.

Code under review:
private static int getStatusCode(BMap<String, BValue> errorDetails) {
        if (!errorDetails.hasKey(STATUS_CODE)) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        BValue specifiedStatusCode = errorDetails.get(STATUS_CODE);
        if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        long specifiedIntCode = ((BInteger) specifiedStatusCode).intValue();
        if (specifiedIntCode >= DEFAULT_ERROR_RETURN_STATUS_CODE) {
            if (specifiedIntCode > Integer.MAX_VALUE) {
                return Integer.MAX_VALUE;
            }
            return (int) specifiedIntCode;
        }
        return DEFAULT_ERROR_RETURN_STATUS_CODE;
    }

Line commented on: if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {

Revised code:
private static int getStatusCode(BMap<String, BValue> errorDetails) {
        if (!errorDetails.hasKey(STATUS_CODE)) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        BValue specifiedStatusCode = errorDetails.get(STATUS_CODE);
        if (specifiedStatusCode.getType().getTag() != TypeTags.INT) {
            return DEFAULT_ERROR_RETURN_STATUS_CODE;
        }

        long specifiedIntCode = ((BInteger) specifiedStatusCode).intValue();
        if (specifiedIntCode >= DEFAULT_ERROR_RETURN_STATUS_CODE) {
            if (specifiedIntCode > Integer.MAX_VALUE) {
                return Integer.MAX_VALUE;
            }
            return (int) specifiedIntCode;
        }
        return DEFAULT_ERROR_RETURN_STATUS_CODE;
    }
Code review comment: Shouldn't we have a `finally` block here?

Code under review:
public void run() throws Exception {

        SuperstepKickoffLatch nextSuperstepLatch =
                SuperstepKickoffLatchBroker.instance().get(brokerKey());

        while (this.running && !terminationRequested()) {

            if (log.isInfoEnabled()) {
                log.info(formatLogString("starting iteration [" + currentIteration() + "]"));
            }

            super.run();

            
            verifyEndOfSuperstepState();

            if (isWorksetUpdate && isWorksetIteration) {
                long numCollected = worksetUpdateOutputCollector.getElementsCollectedAndReset();
                worksetAggregator.aggregate(numCollected);
            }

            if (log.isInfoEnabled()) {
                log.info(formatLogString("finishing iteration [" + currentIteration() + "]"));
            }

            
            sendEndOfSuperstep();

            if (isWorksetUpdate) {
                
                worksetBackChannel.notifyOfEndOfSuperstep();
            }

            boolean terminated =
                    nextSuperstepLatch.awaitStartOfSuperstepOrTermination(currentIteration() + 1);

            if (terminated) {
                requestTermination();
            } else {
                incrementIterationCounter();
            }
        }
        terminationCompleted();
    }

Line commented on: terminationCompleted();

Revised code:
public void run() throws Exception {

        try {
            SuperstepKickoffLatch nextSuperstepLatch =
                    SuperstepKickoffLatchBroker.instance().get(brokerKey());

            while (this.running && !terminationRequested()) {

                if (log.isInfoEnabled()) {
                    log.info(formatLogString("starting iteration [" + currentIteration() + "]"));
                }

                super.run();

                
                verifyEndOfSuperstepState();

                if (isWorksetUpdate && isWorksetIteration) {
                    long numCollected = worksetUpdateOutputCollector.getElementsCollectedAndReset();
                    worksetAggregator.aggregate(numCollected);
                }

                if (log.isInfoEnabled()) {
                    log.info(formatLogString("finishing iteration [" + currentIteration() + "]"));
                }

                
                sendEndOfSuperstep();

                if (isWorksetUpdate) {
                    
                    worksetBackChannel.notifyOfEndOfSuperstep();
                }

                boolean terminated =
                        nextSuperstepLatch.awaitStartOfSuperstepOrTermination(
                                currentIteration() + 1);

                if (terminated) {
                    requestTermination();
                } else {
                    incrementIterationCounter();
                }
            }
        } finally {
            terminationCompleted();
        }
    }
Code review comment: I see this is a wrapped byte array in the parent class

Code under review:
protected String deserializePartitionKeyFromStream(DataInputStream in) throws IOException {
        int partitionKeyLength = in.readInt();
        byte[] requestPartitionKeyData = new byte[(int) partitionKeyLength];
        in.read(requestPartitionKeyData);
        return new String(requestPartitionKeyData, StandardCharsets.UTF_8);
    }

Line commented on: in.read(requestPartitionKeyData);

Revised code:
protected String deserializePartitionKeyFromStream(DataInputStream in) throws IOException {
        int partitionKeyLength = in.readInt();
        byte[] requestPartitionKeyData = new byte[(int) partitionKeyLength];
        in.read(requestPartitionKeyData);
        return new String(requestPartitionKeyData, StandardCharsets.UTF_8);
    }
Code review comment: Can we do an integer division here instead of `String.valueOf(statusCode).charAt(0)` ? I think it might be better.

Code under review:
private void addHttpStatusCode(int statusCode) {
        Optional<ObserverContext> observerContext = ObserveUtils.getObserverContextOfCurrentFrame(context.getStrand());
        observerContext.ifPresent(ctx -> ctx.addTag(ObservabilityConstants.TAG_KEY_HTTP_STATUS_CODE_GROUP,
                String.valueOf(statusCode).charAt(0) + STATUS_CODE_GROUP_SUFFIX));
    }

Line commented on: String.valueOf(statusCode).charAt(0) + STATUS_CODE_GROUP_SUFFIX));

Revised code:
private void addHttpStatusCode(int statusCode) {
        Optional<ObserverContext> observerContext = ObserveUtils.getObserverContextOfCurrentFrame(context.getStrand());
        observerContext.ifPresent(ctx -> ctx.addTag(ObservabilityConstants.TAG_KEY_HTTP_STATUS_CODE_GROUP,
                statusCode / 100 + STATUS_CODE_GROUP_SUFFIX));
    }
Code review comment: will it print the reason as well?

Code under review:
private void assertFunctions(BIRNode.BIRFunction func) {
        String expectedBir = null;
        try {
            expectedBir = readFile(func.getName().getValue());
        } catch (IOException e) {
            Assert.fail();
        }
        if (!"".equals(expectedBir)) {
            String funcBir = birEmitter.emitFunction(func, 0);
            Assert.assertEquals(funcBir, expectedBir);
        }
    }

Line commented on: Assert.fail();

Revised code:
private void assertFunctions(BIRNode.BIRFunction func) {
        String expectedBir = null;
        try {
            expectedBir = readFile(func.getName().getValue());
        } catch (IOException e) {
            Assert.fail("Failed when reading file", e);
        }
        if (!"".equals(expectedBir)) {
            String funcBir = birEmitter.emitFunction(func, 0);
            Assert.assertEquals(funcBir, expectedBir);
        }
    }
Code review comment: can't we assume that the second param is always partition key?  that way the caller doesn't have to explicitly wrap pk value in PartitionKey and implementation will take care of that.  thoughts?

Code under review:
protected void performWorkload(BaseSubscriber<CosmosAsyncItemResponse> baseSubscriber, long i) throws InterruptedException {
        int index = (int) (i % docsToRead.size());
        PojoizedJson doc = docsToRead.get(index);

        String partitionKeyValue = doc.getId();
        Mono<CosmosAsyncItemResponse<PojoizedJson>> result = cosmosAsyncContainer.readItem(doc.getId(),
                                                                                           new PartitionKey(partitionKeyValue),
                                                                                           PojoizedJson.class);

        concurrencyControlSemaphore.acquire();

        if (configuration.getOperationType() == Configuration.Operation.ReadThroughput) {
            result.subscribeOn(Schedulers.parallel()).subscribe(baseSubscriber);
        } else {
            LatencySubscriber<CosmosAsyncItemResponse> latencySubscriber = new LatencySubscriber<>(baseSubscriber);
            latencySubscriber.context = latency.time();
            result.subscribeOn(Schedulers.parallel()).subscribe(latencySubscriber);
        }
    }

Line commented on: new PartitionKey(partitionKeyValue),

Revised code:
protected void performWorkload(BaseSubscriber<CosmosAsyncItemResponse> baseSubscriber, long i) throws InterruptedException {
        int index = (int) (i % docsToRead.size());
        PojoizedJson doc = docsToRead.get(index);

        String partitionKeyValue = doc.getId();
        Mono<CosmosAsyncItemResponse<PojoizedJson>> result = cosmosAsyncContainer.readItem(doc.getId(),
                                                                                           new PartitionKey(partitionKeyValue),
                                                                                           PojoizedJson.class);

        concurrencyControlSemaphore.acquire();

        if (configuration.getOperationType() == Configuration.Operation.ReadThroughput) {
            result.subscribeOn(Schedulers.parallel()).subscribe(baseSubscriber);
        } else {
            LatencySubscriber<CosmosAsyncItemResponse> latencySubscriber = new LatencySubscriber<>(baseSubscriber);
            latencySubscriber.context = latency.time();
            result.subscribeOn(Schedulers.parallel()).subscribe(latencySubscriber);
        }
    }
Code review comment: `Arrays.asList` is already a hack and not readable - the method is for converting an array to a list, but it is used here to define a new list "literal", so to speak. But programmers seem to know the pattern so it is not so bad. Removing `Arrays` from the beginning makes it less clear that this hack is being used. These should instead be `ImmutableList.of` (I'm assuming they are all used immutably).

Code under review:
public void setUp() throws TransientKinesisException {
    when(a.getShardId()).thenReturn("shard1");
    when(b.getShardId()).thenReturn("shard1");
    when(c.getShardId()).thenReturn("shard2");
    when(d.getShardId()).thenReturn("shard2");
    when(firstCheckpoint.getShardId()).thenReturn("shard1");
    when(secondCheckpoint.getShardId()).thenReturn("shard2");
    when(firstIterator.getShardId()).thenReturn("shard1");
    when(firstIterator.getCheckpoint()).thenReturn(firstCheckpoint);
    when(secondIterator.getShardId()).thenReturn("shard2");
    when(secondIterator.getCheckpoint()).thenReturn(secondCheckpoint);
    when(thirdIterator.getShardId()).thenReturn("shard3");
    when(fourthIterator.getShardId()).thenReturn("shard4");
    KinesisReaderCheckpoint checkpoint = new KinesisReaderCheckpoint(
        asList(firstCheckpoint, secondCheckpoint));
    shardReadersPool = Mockito.spy(new ShardReadersPool(kinesis, checkpoint));
    doReturn(firstIterator).when(shardReadersPool).createShardIterator(kinesis, firstCheckpoint);
    doReturn(secondIterator).when(shardReadersPool).createShardIterator(kinesis, secondCheckpoint);
  }

Line commented on: asList(firstCheckpoint, secondCheckpoint));

Revised code:
public void setUp() throws TransientKinesisException {
    when(a.getShardId()).thenReturn("shard1");
    when(b.getShardId()).thenReturn("shard1");
    when(c.getShardId()).thenReturn("shard2");
    when(d.getShardId()).thenReturn("shard2");
    when(firstCheckpoint.getShardId()).thenReturn("shard1");
    when(secondCheckpoint.getShardId()).thenReturn("shard2");
    when(firstIterator.getShardId()).thenReturn("shard1");
    when(firstIterator.getCheckpoint()).thenReturn(firstCheckpoint);
    when(secondIterator.getShardId()).thenReturn("shard2");
    when(secondIterator.getCheckpoint()).thenReturn(secondCheckpoint);
    when(thirdIterator.getShardId()).thenReturn("shard3");
    when(fourthIterator.getShardId()).thenReturn("shard4");
    KinesisReaderCheckpoint checkpoint = new KinesisReaderCheckpoint(
        ImmutableList.of(firstCheckpoint, secondCheckpoint));
    shardReadersPool = Mockito.spy(new ShardReadersPool(kinesis, checkpoint));
    doReturn(firstIterator).when(shardReadersPool).createShardIterator(kinesis, firstCheckpoint);
    doReturn(secondIterator).when(shardReadersPool).createShardIterator(kinesis, secondCheckpoint);
  }
Code review comment: Does this mean it use a expr parition?

Code under review:
public boolean hasPartitionTransformedEvolution() {
        return getNativeTable().spec().fields().stream().anyMatch(field -> field.transform().isVoid());
    }

Line commented on: return getNativeTable().spec().fields().stream().anyMatch(field -> field.transform().isVoid());

Revised code:
public boolean hasPartitionTransformedEvolution() {
        return getNativeTable().spec().fields().stream().anyMatch(field -> field.transform().isVoid());
    }
Code review comment: Should the emitted log level be configurable? I.e. to log it at debug, info or warning level? Also, since toString() on the context could become CPU intensive (if in future it uses JSON etc) one can check the log level enabled for the logger and if not enabled for info or warning then do not call toString or put the rest of the info on the stack since it is not going to be logged anyway.

Code under review:
protected void log(CosmosDiagnosticsContext ctx) {
        if (this.shouldLogDueToStatusCode(ctx.getStatusCode(), ctx.getSubStatusCode())) {
            logger.warn(
                "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{} Diagnostics: {}",
                ctx.getAccountName(),
                ctx.getDatabaseName(),
                ctx.getCollectionName(),
                ctx.getStatusCode(),
                ctx.getSubStatusCode(),
                ctx.toString());
        } else {
            logger.info(
                "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{} Diagnostics: {}",
                ctx.getAccountName(),
                ctx.getDatabaseName(),
                ctx.getCollectionName(),
                ctx.getStatusCode(),
                ctx.getSubStatusCode(),
                ctx.toString());
        }
    }

Line commented on: logger.info(

Revised code:
protected void log(CosmosDiagnosticsContext ctx) {
        if (ctx.isFailure()) {
            if (logger.isErrorEnabled()) {
                logger.error(
                    "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{} Diagnostics: {}",
                    ctx.getAccountName(),
                    ctx.getDatabaseName(),
                    ctx.getContainerName(),
                    ctx.getStatusCode(),
                    ctx.getSubStatusCode(),
                    ctx);
            }
        } else if (ctx.isThresholdViolated()) {
            if (logger.isInfoEnabled()) {
                logger.info(
                    "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{} Diagnostics: {}",
                    ctx.getAccountName(),
                    ctx.getDatabaseName(),
                    ctx.getContainerName(),
                    ctx.getStatusCode(),
                    ctx.getSubStatusCode(),
                    ctx);
            }
        } else if (logger.isTraceEnabled()) {
            logger.trace(
                "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{} Diagnostics: {}",
                ctx.getAccountName(),
                ctx.getDatabaseName(),
                ctx.getContainerName(),
                ctx.getStatusCode(),
                ctx.getSubStatusCode(),
                ctx);
        } else if (logger.isDebugEnabled()) {
            logger.debug(
                "Account: {} -> DB: {}, Col:{}, StatusCode: {}:{}, Latency: {}, Request charge: {}",
                ctx.getAccountName(),
                ctx.getDatabaseName(),
                ctx.getContainerName(),
                ctx.getStatusCode(),
                ctx.getSubStatusCode(),
                ctx.getDuration(),
                ctx.getTotalRequestCharge());
        }
    }
Code review comment: Shall we add throwable as a second parameter along with the message?

Code under review:
public void onError(Throwable throwable) {
            BMap<String, BValue> httpConnectorError =  HttpUtil.getError(dataContext.context, throwable);
            if (outboundMsgDataStreamer != null) {
                if (throwable instanceof IOException) {
                    this.dataContext.getOutboundRequest().setIoException((IOException) throwable);
                } else {
                    this.dataContext.getOutboundRequest().setIoException(new IOException(throwable.getMessage()));
                }
            }
            this.dataContext.notifyOutboundResponseStatus(httpConnectorError);
        }

Line commented on: this.dataContext.getOutboundRequest().setIoException(new IOException(throwable.getMessage()));

Revised code:
public void onError(Throwable throwable) {
            BMap<String, BValue> httpConnectorError =  HttpUtil.getError(dataContext.context, throwable);
            if (outboundMsgDataStreamer != null) {
                if (throwable instanceof IOException) {
                    this.dataContext.getOutboundRequest().setIoException((IOException) throwable);
                } else {
                    this.dataContext.getOutboundRequest()
                            .setIoException(new IOException(throwable.getMessage(), throwable));
                }
            }
            this.dataContext.notifyOutboundResponseStatus(httpConnectorError);
        }
Code review comment: Shouldn't it be completely removed?

Code under review:
protected List<ArtifactCoords> getDependencies() {
        if (dependencies == null) {
            dependencies = projectDepsSupplier.get();
            projectDepsSupplier = null;
        }
        return dependencies;
    }

Line commented on: if (dependencies == null) {

Revised code:
protected List<ArtifactCoords> getDependencies() {
        if (dependencies == null) {
            dependencies = projectDepsSupplier.get();
            projectDepsSupplier = null;
        }
        return dependencies;
    }
Code review comment: nit: the tabbing needs to be fixed here

Code under review:
public void createRelationship() {
        
        BasicRelationship buildingToFloorBasicRelationship = new BasicRelationship(
            "myRelationshipId",
            "mySourceDigitalTwinId",
            "myTargetDigitalTwinId",
            "contains")
            .addCustomProperty("Prop1", "Prop1 value")
            .addCustomProperty("Prop2", 6);

        digitalTwinsAsyncClient.createRelationship(
            "mySourceDigitalTwinId",
            "myRelationshipId",
            buildingToFloorBasicRelationship,
            BasicRelationship.class)
        .subscribe(createdRelationship -> System.out.println(
            "Created relationship with Id: " +
                createdRelationship.getId() +
                " from: " + createdRelationship.getSourceId() +
                " to: " + createdRelationship.getTargetId()));
        

        
        String relationshipPayload = getRelationshipPayload();

        digitalTwinsAsyncClient.createRelationship(
            "mySourceDigitalTwinId",
            "myRelationshipId",
            relationshipPayload,
            String.class)
        .subscribe(createRelationshipString ->
            System.out.println("Created relationship: " + createRelationshipString));
        
    }

Line commented on: "myRelationshipId",

Revised code:
public void createRelationship() {
        
        BasicRelationship buildingToFloorBasicRelationship = new BasicRelationship(
                "myRelationshipId",
                "mySourceDigitalTwinId",
                "myTargetDigitalTwinId",
                "contains")
            .addCustomProperty("Prop1", "Prop1 value")
            .addCustomProperty("Prop2", 6);

        digitalTwinsAsyncClient.createRelationship(
            "mySourceDigitalTwinId",
            "myRelationshipId",
            buildingToFloorBasicRelationship,
            BasicRelationship.class)
        .subscribe(createdRelationship -> System.out.println(
            "Created relationship with Id: " +
                createdRelationship.getId() +
                " from: " + createdRelationship.getSourceId() +
                " to: " + createdRelationship.getTargetId()));
        

        
        String relationshipPayload = getRelationshipPayload();

        digitalTwinsAsyncClient.createRelationship(
            "mySourceDigitalTwinId",
            "myRelationshipId",
            relationshipPayload,
            String.class)
        .subscribe(createRelationshipString ->
            System.out.println("Created relationship: " + createRelationshipString));
        
    }
Code review comment: I hardcoded it because some previous versions didn't work (it was impossible to create a stream using kinesis sdk) without any order. It can be changed to "latest" but then it's possible that the test will stop working after the localstack image update.

Code under review:
private static void setupLocalstack() throws Exception {
    System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY, "true");
    System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");

    
    now = Instant.ofEpochMilli(Long.divideUnsigned(Instant.now().getMillis(), 1000));

    localstackContainer =
        new LocalStackContainer("0.11.3")
            .withServices(LocalStackContainer.Service.KINESIS)
            .withEnv("USE_SSL", "true")
            .withStartupAttempts(3);
    localstackContainer.start();

    options.setAwsServiceEndpoint(
        localstackContainer
            .getEndpointConfiguration(LocalStackContainer.Service.KINESIS)
            .getServiceEndpoint()
            .replace("http", "https"));
    options.setAwsKinesisRegion(
        localstackContainer
            .getEndpointConfiguration(LocalStackContainer.Service.KINESIS)
            .getSigningRegion());
    options.setAwsAccessKey(
        localstackContainer.getDefaultCredentialsProvider().getCredentials().getAWSAccessKeyId());
    options.setAwsSecretKey(
        localstackContainer.getDefaultCredentialsProvider().getCredentials().getAWSSecretKey());
    options.setNumberOfRecords(1000);
    options.setNumberOfShards(1);
    options.setAwsKinesisStream("beam_kinesis_test");
    options.setAwsVerifyCertificate(false);
    createStream(options.getAwsKinesisStream());
  }

Line commented on: new LocalStackContainer("0.11.3")

Revised code:
private static void setupLocalstack() {
    
    now = Instant.ofEpochMilli(Long.divideUnsigned(now.getMillis(), 1000L));

    System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY, "true");
    System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");

    localstackContainer =
        new LocalStackContainer(LOCALSTACK_VERSION)
            .withServices(LocalStackContainer.Service.KINESIS)
            .withEnv("USE_SSL", "true")
            .withStartupAttempts(3);
    localstackContainer.start();

    options.setAwsServiceEndpoint(
        localstackContainer
            .getEndpointConfiguration(LocalStackContainer.Service.KINESIS)
            .getServiceEndpoint()
            .replace("http", "https"));
    options.setAwsKinesisRegion(
        localstackContainer
            .getEndpointConfiguration(LocalStackContainer.Service.KINESIS)
            .getSigningRegion());
    options.setAwsAccessKey(
        localstackContainer.getDefaultCredentialsProvider().getCredentials().getAWSAccessKeyId());
    options.setAwsSecretKey(
        localstackContainer.getDefaultCredentialsProvider().getCredentials().getAWSSecretKey());
    options.setNumberOfRecords(1000);
    options.setNumberOfShards(1);
    options.setAwsKinesisStream("beam_kinesis_test");
    options.setAwsVerifyCertificate(false);
  }
Code review comment: Does "if (index > lastIndex)" condition  not handle capacity?

Code under review:
private void unshiftArray(long index, int unshiftByN, int arrLength) {
        int lastIndex = size() + unshiftByN - 1;
        prepareForConsecutiveMultiAdd(lastIndex, arrLength);
        if (index > lastIndex) {
            throw BLangExceptionHelper.getRuntimeException(
                    getModulePrefixedReason(ARRAY_LANG_LIB, INDEX_OUT_OF_RANGE_ERROR_IDENTIFIER),
                    RuntimeErrors.INDEX_NUMBER_TOO_LARGE, index);
        }
        int i = (int) index;
        ensureCapacity(this.size + unshiftByN, this.size);
        Object arr = getArrayFromType(elementType.getTag());
        System.arraycopy(arr, i, arr, i + unshiftByN, this.size - i);
    }

Line commented on: ensureCapacity(this.size + unshiftByN, this.size);

Revised code:
private void unshiftArray(long index, int unshiftByN, int arrLength) {
        int lastIndex = size() + unshiftByN - 1;
        prepareForConsecutiveMultiAdd(lastIndex, arrLength);
        if (index > lastIndex) {
            throw BLangExceptionHelper.getRuntimeException(
                    getModulePrefixedReason(ARRAY_LANG_LIB, INDEX_OUT_OF_RANGE_ERROR_IDENTIFIER),
                    RuntimeErrors.INDEX_NUMBER_TOO_LARGE, index);
        }
        int i = (int) index;
        ensureCapacity(this.size + unshiftByN, this.size);
        Object arr = getArrayFromType(elementType.getTag());
        System.arraycopy(arr, i, arr, i + unshiftByN, this.size - i);
    }
Code review comment: OK, I will remove the Parquet protocal part. and the Parquet reader should holds a Long or Bytes vector and implement the `TimestmapColumnVector` interface.

Code under review:
public SqlTimestamp getTimestamp(int rowId, int colId, int precision) {
		if (isNullAt(rowId, colId)) {
			return null;
		}

		
		
		
		
		
		
		if (columns[colId] instanceof TimestampColumnVector) {
			return ((TimestampColumnVector) (columns[colId])).getTimestamp(rowId, precision);
		} else if (precision <= 3) {
			return SqlTimestamp.fromEpochMillis(getLong(rowId, colId));
		} else if (precision <= 6) {
			long microseconds = getLong(rowId, colId);
			return SqlTimestamp.fromEpochMillis(
				microseconds / 1000, (int) (microseconds % 1000) * 1000);
		} else {
			byte[] bytes = getBytes(rowId, colId);
			assert bytes.length == 12;
			long nanoOfDay = 0;
			for (int i = 0; i < 8; i++) {
				nanoOfDay <<= 8;
				nanoOfDay |= (bytes[i] & (0xff));
			}
			int julianDay = 0;
			for (int i = 8; i < 12; i++) {
				julianDay <<= 8;
				julianDay |= (bytes[i] & (0xff));
			}
			long millisecond =
				(julianDay - DateTimeUtils.EPOCH_JULIAN) * DateTimeUtils.MILLIS_PER_DAY + nanoOfDay / 1000000;
			int nanoOfMillisecond = (int) (nanoOfDay % 1000000);
			return SqlTimestamp.fromEpochMillis(millisecond, nanoOfMillisecond);
		}
	}

Line commented on: if (columns[colId] instanceof TimestampColumnVector) {

Revised code:
public SqlTimestamp getTimestamp(int rowId, int colId, int precision) {
		return ((TimestampColumnVector) (columns[colId])).getTimestamp(rowId, precision);
	}
Code review comment: It's best if we can ensure the configuration never ends up in a bad state (i.e. where required values are set to null).  > As for the NonNull annotations at the level of the builders: they force you to essentially create a method which accepts all mandatory fields totally negating the idea behind the builder in the first place. If you accept that you're filling in the information one "with" method at a time, you're going to temporary have fields which are null until all information is provided.  Your analysis is correct, but I think we should reconsider the assumption that we must only provide `with` methods as the public API. As far as I can tell, there are three things `with` methods accomplish here:  1. null checks 2. helper methods for StaticValueProviders 3. create a modified copy of a configuration  Item 1 is best left to the AutoValue generated code, and 2 is unneeded if we get remove ValueProviders. 3 is the only one that really provides value IMO.  If we make the Builder public, we can remove all the `@Nullable` annotations and null checks. We can still offer `with` methods to address item 3 if we want, but the builder would be the only way to construct an object initially.

Code under review:
public DriverConfiguration withUrl(ValueProvider<String> url) {
      Preconditions.checkArgument(
          url != null, "a neo4j connection URL can not be empty or null", url);
      Preconditions.checkArgument(
          StringUtils.isNotEmpty(url.get()),
          "a neo4j connection URL can not be empty or null",
          url);
      return builder().setUrl(url).build();
    }

Line commented on: "a neo4j connection URL can not be empty or null",

Revised code:
public DriverConfiguration withUrl(ValueProvider<String> url) {
      Preconditions.checkArgument(
          url != null, "a neo4j connection URL can not be empty or null", url);
      Preconditions.checkArgument(
          StringUtils.isNotEmpty(url.get()),
          "a neo4j connection URL can not be empty or null",
          url);
      return builder().setUrl(url).build();
    }
Code review comment: I now explicitly nulled the `currentProducer`. That still leaves room in the future for errors but at least not correctness issues.

Code under review:
public List<KafkaWriterState> snapshotState(long checkpointId) throws IOException {
        currentProducer = createProducer(checkpointId);
        return ImmutableList.of(kafkaWriterState);
    }

Line commented on: currentProducer = createProducer(checkpointId);

Revised code:
public List<KafkaWriterState> snapshotState(long checkpointId) throws IOException {
        if (deliveryGuarantee == DeliveryGuarantee.EXACTLY_ONCE) {
            currentProducer = getTransactionalProducer(checkpointId + 1);
            currentProducer.beginTransaction();
        }
        return ImmutableList.of(kafkaWriterState);
    }
Code review comment: ```suggestion         String lastDb = ctx.getLastDBOfCatalog(catalogName); ```

Code under review:
public void changeCatalog(ConnectContext ctx, String catalogName) throws DdlException {
        CatalogIf catalogIf = catalogMgr.getCatalog(catalogName);
        if (catalogIf == null) {
            throw new DdlException(ErrorCode.ERR_UNKNOWN_CATALOG.formatErrorMsg(catalogName),
                    ErrorCode.ERR_UNKNOWN_CATALOG);
        }

        String currentDB = ctx.getDatabase();
        if (StringUtils.isNotEmpty(currentDB)) {
            
            if (ctx.getCurrentCatalog() != null) {
                ConnectContext.get().addLastDBOfCatalog(ctx.getCurrentCatalog().getName(), currentDB);
            }
        }
        ctx.changeDefaultCatalog(catalogName);
        String lastDb = ConnectContext.get().getLastDBOfCatalog(catalogName);
        if (StringUtils.isNotEmpty(lastDb)) {
            ctx.setDatabase(lastDb);
        }
        if (catalogIf instanceof EsExternalCatalog) {
            ctx.setDatabase(EsExternalCatalog.DEFAULT_DB);
        }
    }

Line commented on: String lastDb = ConnectContext.get().getLastDBOfCatalog(catalogName);

Revised code:
public void changeCatalog(ConnectContext ctx, String catalogName) throws DdlException {
        CatalogIf catalogIf = catalogMgr.getCatalog(catalogName);
        if (catalogIf == null) {
            throw new DdlException(ErrorCode.ERR_UNKNOWN_CATALOG.formatErrorMsg(catalogName),
                    ErrorCode.ERR_UNKNOWN_CATALOG);
        }

        String currentDB = ctx.getDatabase();
        if (StringUtils.isNotEmpty(currentDB)) {
            
            if (ctx.getCurrentCatalog() != null) {
                ctx.addLastDBOfCatalog(ctx.getCurrentCatalog().getName(), currentDB);
            }
        }
        ctx.changeDefaultCatalog(catalogName);
        String lastDb = ctx.getLastDBOfCatalog(catalogName);
        if (StringUtils.isNotEmpty(lastDb)) {
            ctx.setDatabase(lastDb);
        }
        if (catalogIf instanceof EsExternalCatalog) {
            ctx.setDatabase(EsExternalCatalog.DEFAULT_DB);
        }
    }
Code review comment: I guess we can check size of array before doing [0] ```suggestion                 var channelArray = propertyName.replace(INCOMING_PREFIX, "").split("\\.")                 if( channelArray. length()>0){                      var channelName = channelArray[0];                 } ```

Code under review:
private List<String> extractChannels(Config configIn) {

        var list = new ArrayList<String>();

        for (String propertyName : configIn.getPropertyNames()) {
            if (propertyName.startsWith(INCOMING_PREFIX)) {
                var channelName = propertyName.replace(INCOMING_PREFIX, "").split("\\.")[0];
                list.add(INCOMING_PREFIX + channelName + ".");
            } else if (propertyName.startsWith(OUTGOING_PREFIX)) {
                var channelName = propertyName.replace(OUTGOING_PREFIX, "").split("\\.")[0];
                list.add(OUTGOING_PREFIX + channelName + ".");
            }
        }
        return list;
    }

Line commented on: for (String propertyName : configIn.getPropertyNames()) {

Revised code:
private List<String> extractChannels(Config configIn) {

        var list = new ArrayList<String>();

        for (String propertyName : configIn.getPropertyNames()) {
            if (propertyName.startsWith(INCOMING_PREFIX)) {
                var channelAndProp = StringUtils.substringAfter(propertyName, INCOMING_PREFIX);
                
                var channelName = StringUtils.substringBefore(channelAndProp, ".");
                if (!StringUtils.isBlank(channelName))
                    list.add(INCOMING_PREFIX + channelName + ".");
            } else if (propertyName.startsWith(OUTGOING_PREFIX)) {
                var channelAndProp = StringUtils.substringAfter(propertyName, OUTGOING_PREFIX);
                
                var channelName = StringUtils.substringBefore(channelAndProp, ".");
                if (!StringUtils.isBlank(channelName))
                    list.add(OUTGOING_PREFIX + channelName + ".");
            }
        }
        return list;
    }
Code review comment: In Arrange, Act, and Assert, this line would be "Act". And the Line 135 would be "Assert"

Code under review:
void createsMessageBatchWithSize() {
        
        int batchSize = 1024;

        final CreateBatchOptions options = new CreateBatchOptions().setMaximumSizeInBytes(batchSize);
        ServiceBusMessageBatch batch =  new ServiceBusMessageBatch(batchSize, null, null,
            null);
        when(asyncSender.createBatch(options)).thenReturn(Mono.just(batch));

        
        ServiceBusMessageBatch messageBatch = sender.createBatch(options);

        Assertions.assertEquals(batch, messageBatch);
    }

Line commented on: ServiceBusMessageBatch messageBatch = sender.createBatch(options);

Revised code:
void createsMessageBatchWithSize() {
        
        int batchSize = 1024;

        final CreateBatchOptions options = new CreateBatchOptions().setMaximumSizeInBytes(batchSize);
        final ServiceBusMessageBatch batch = new ServiceBusMessageBatch(batchSize, null, null,
            null);
        when(asyncSender.createBatch(options)).thenReturn(Mono.just(batch));

        
        ServiceBusMessageBatch messageBatch = sender.createBatch(options);

        
        Assertions.assertEquals(batch, messageBatch);
    }
Code review comment: The `numPendingPodRequests` would never be 0 from the log, so maybe it is better to make this log after below condition.

Code under review:
public void onAdded(List<KubernetesPod> pods) {
		runAsync(() -> {
			for (KubernetesPod pod : pods) {
				log.info("Received new TaskExecutor pod: {} - Remaining pending pod requests: {}",
					pod.getName(), numPendingPodRequests);

				if (numPendingPodRequests > 0) {
					numPendingPodRequests--;
					final KubernetesWorkerNode worker = new KubernetesWorkerNode(new ResourceID(pod.getName()));
					workerNodeMap.putIfAbsent(worker.getResourceID(), worker);
				}
			}
		});
	}

Line commented on: log.info("Received new TaskExecutor pod: {} - Remaining pending pod requests: {}",

Revised code:
public void onAdded(List<KubernetesPod> pods) {
		runAsync(() -> {
			for (KubernetesPod pod : pods) {
				if (numPendingPodRequests > 0) {
					numPendingPodRequests--;
					final KubernetesWorkerNode worker = new KubernetesWorkerNode(new ResourceID(pod.getName()));
					workerNodes.putIfAbsent(worker.getResourceID(), worker);
				}

				log.info("Received new TaskManager pod: {} - Remaining pending pod requests: {}",
					pod.getName(), numPendingPodRequests);
			}
		});
	}