muttakin2301/ccr2000 huggingface dataset:
Code review comment: I think for some things like this which we expect to just be started once and live for ever, it can be simpler to just prevent some behavior instead of trying to support it.  If start can be called multiple times, one caller could stop while the other doesn't but hte end state is very different start1/start2/stop1 -> stopped start1/stop1/start2 -> started So it seems safer to just prevent it since we only really expect these methods to be called by the main startup path and then in teardown for tests.

Code under review:
public void start() {
    if (isRunning.compareAndSet(false, true) && !commitSenders.isShutdown()) {
      for (int i = 0; i < numCommitSenders; i++) {
        commitSenders.submit(this::streamingCommitLoop);
      }
    }
  }

Line commented on: if (isRunning.compareAndSet(false, true) && !commitSenders.isShutdown()) {

Revised code:
public void start() {
    Preconditions.checkState(
        isRunning.compareAndSet(false, true), "Multiple calls to WorkCommitter.start().");
    for (int i = 0; i < numCommitSenders; i++) {
      commitSenders.submit(this::streamingCommitLoop);
    }
  }
Code review comment: Shall we use the Platform specific `System.lineSeparator()`? There is already a constant in `CommonUtil`

Code under review:
public String getDocumentationString(boolean newlineAtEnd) {
        StringBuilder result = new StringBuilder();
        
        
        String[] descriptionLines = this.description.trim().split("\n");
        for (String descriptionLine : descriptionLines) {
            result.append(String.format("
        }

        if (!parameters.isEmpty()) {
            StringJoiner paramJoiner = new StringJoiner(CommonUtil.MD_LINE_SEPARATOR);
            for (Map.Entry<String, String> parameter : this.parameters.entrySet()) {
                paramJoiner.add(String.format("%s
            }
            result.append(String.format("%s
        }

        if (returnDesc != null) {
            result.append(String.format("%s
        }

        if (deprecatedDesc != null) {
            result.append(String.format("%s
            String[] deprecatedDescLines = deprecatedDesc.trim().split("\n");
            for (String deprecatedLine : deprecatedDescLines) {
                result.append(String.format("%s
            }
        }

        if (newlineAtEnd) {
            return result.toString().trim() + CommonUtil.MD_LINE_SEPARATOR + padding;
        } else {
            return result.toString().trim();
        }
    }

Line commented on: String[] deprecatedDescLines = deprecatedDesc.trim().split("\n");

Revised code:
public String getDocumentationString(boolean newlineAtEnd) {
        StringBuilder result = new StringBuilder();
        String[] descriptionLines = this.description.trim().split("(\r)?\n");
        for (String descriptionLine : descriptionLines) {
            result.append(String.format("
        }

        if (!parameters.isEmpty()) {
            StringJoiner paramJoiner = new StringJoiner(CommonUtil.MD_LINE_SEPARATOR);
            for (Map.Entry<String, String> parameter : this.parameters.entrySet()) {
                paramJoiner.add(String.format("%s
            }
            result.append(String.format("%s
        }

        if (returnDesc != null) {
            result.append(String.format("%s
        }

        if (deprecatedDesc != null) {
            result.append(String.format("%s
            String[] deprecatedDescLines = deprecatedDesc.trim().split("(\r)?\n");
            for (String deprecatedLine : deprecatedDescLines) {
                result.append(String.format("%s
            }
        }

        if (newlineAtEnd) {
            return result.toString().trim() + CommonUtil.MD_LINE_SEPARATOR + padding;
        } else {
            return result.toString().trim();
        }
    }
Code review comment: Also, codepoint < MIN_SURROGATE is never true while the below is false, since the MIN_SURROGATE is 0xD800, which is less than the first thing checked below (0xFDD0).

Code under review:
public static boolean isTextCharacter(int codepoint) {
        
        

        return (codepoint < 0x80)
                ? allowedAsciiChars[codepoint]
                : (codepoint <  Character.MIN_SURROGATE) || isTextCharAboveUsAscii(codepoint);
    }

Line commented on: : (codepoint <  Character.MIN_SURROGATE) || isTextCharAboveUsAscii(codepoint);

Revised code:
public static boolean isTextCharacter(int codepoint) {
        
        

        return (codepoint < 0x80)
                ? allowedAsciiChars[codepoint]
                : (codepoint <  Character.MIN_SURROGATE) || isTextCharAboveMinSurrogate(codepoint);
    }
Code review comment: ACK - but I think we all agree that making the changes to fail fast will result in additional risk and need for validation (probably with changes to test infrastructure) - so IMO this is something we should do after releasing the hot fix. Makes sense?

Code under review:
private void runTasksInPendingAcquisitionQueue() {
        ensureInEventLoop();

        this.ensureInEventLoop();
        int channelsAvailable = this.availableChannels.size();

        
        
        
        do {

            
            final AcquireTask task = this.pendingAcquisitions.poll();


            if (task == null) {
                break;
            }

            final ScheduledFuture<?> timeoutFuture = task.timeoutFuture;

            if (timeoutFuture != null) {
                timeoutFuture.cancel(false);
            }

            task.acquired(true);
            this.acquire(task.promise);
        } while (--channelsAvailable> 0);
    }

Line commented on: this.ensureInEventLoop();

Revised code:
private void runTasksInPendingAcquisitionQueue() {
        this.ensureInEventLoop();
        int channelsAvailable = this.availableChannels.size();

        
        
        
        do {

            
            final AcquireTask task = this.pendingAcquisitions.poll();

            if (task == null) {
                break;
            }

            task.acquired(true);
            this.acquire(task.promise);
        } while (--channelsAvailable > 0);
    }
Code review comment: The only thing is that those constants doesn't actually comes from OTEL, rather the Vert.x Command Reporter, and there it comes from the OpenTracing Semantic Conventions... `db.instance` is a no go for instance =] I would rather have Vert.x enum being visible and use that as they are mapping it there instead of using OTEL conventions and it doesn't fully match... and if OTEL changes and vert.x doesn't we will have an issue...

Code under review:
public String operation() {
            return attributes.get("db.statement");
        }

Line commented on: return attributes.get("db.statement");

Revised code:
public String operation() {
            return attributes.get(DB_STATEMENT);
        }
Code review comment: No, not really. The semaphore is to ensure that we do not exit before the async operation has completed. We start all async operations and then release the semaphore only once the async call has completed. That way, the subsequent operations are executed after the previous call has completed -> similar to doing a await on a Task.

Code under review:
public static void createTwins() throws IOException, InterruptedException {
        System.out.println("CREATE DIGITAL TWINS");
        Map<String, String> twins = FileHelper.loadAllFilesInPath(TwinsPath);
        final Semaphore createTwinsSemaphore = new Semaphore(0);

        
        twins
            .forEach((twinId, twinContent) -> client.createDigitalTwinWithResponse(twinId, twinContent)
                .subscribe(
                    response -> System.out.println("Created digital twin: " + twinId + "\n\t Body: " + response.getValue()),
                    throwable -> System.err.println("Could not create digital twin " + twinId + " due to " + throwable),
                    createTwinsSemaphore::release));

        
        boolean created = createTwinsSemaphore.tryAcquire(twins.size(), MaxWaitTimeAsyncOperationsInSeconds, TimeUnit.SECONDS);
        System.out.println("Twins created: " + created);
    }

Line commented on: final Semaphore createTwinsSemaphore = new Semaphore(0);

Revised code:
public static void createTwins() throws IOException, InterruptedException {
        System.out.println("CREATE DIGITAL TWINS");
        Map<String, String> twins = FileHelper.loadAllFilesInPath(TwinsPath);
        final Semaphore createTwinsSemaphore = new Semaphore(0);

        
        twins
            .forEach((twinId, twinContent) -> client.createDigitalTwinWithResponse(twinId, twinContent)
                .subscribe(
                    response -> System.out.println("Created digital twin: " + twinId + "\n\t Body: " + response.getValue()),
                    throwable -> System.err.println("Could not create digital twin " + twinId + " due to " + throwable),
                    createTwinsSemaphore::release));

        
        boolean created = createTwinsSemaphore.tryAcquire(twins.size(), MaxWaitTimeAsyncOperationsInSeconds, TimeUnit.SECONDS);
        System.out.println("Twins created: " + created);
    }
Code review comment: ```suggestion     // Convert to double in computation to prevent overflow, which may result in loss of     // precision. ```

Code under review:
public SplitResult<OffsetRange> trySplit(double fractionOfRemainder) {
    
    if (range.getTo() != Long.MAX_VALUE || range.getTo() == range.getFrom()) {
      return super.trySplit(fractionOfRemainder);
    }
    
    if (lastAttemptedOffset != null && lastAttemptedOffset == Long.MAX_VALUE) {
      return null;
    }
    double cur =
        (lastAttemptedOffset == null) ? (double) range.getFrom() - 1 : (double) lastAttemptedOffset;

    
    
    double estimateRangeEnd = Math.max(rangeEndEstimator.estimate(), cur + 1);

    
    
    double splitPos = cur + Math.max(1L, (estimateRangeEnd - cur) * fractionOfRemainder);
    long split = Double.valueOf(splitPos).longValue();
    if (split > Double.valueOf(estimateRangeEnd).longValue()) {
      return null;
    }
    OffsetRange res = new OffsetRange(split, range.getTo());
    this.range = new OffsetRange(range.getFrom(), split);
    return SplitResult.of(range, res);
  }


Revised code:
public SplitResult<OffsetRange> trySplit(double fractionOfRemainder) {
    
    if (range.getTo() != Long.MAX_VALUE || range.getTo() == range.getFrom()) {
      return super.trySplit(fractionOfRemainder);
    }
    
    if (lastAttemptedOffset != null && lastAttemptedOffset == Long.MAX_VALUE) {
      return null;
    }
    BigDecimal cur =
        (lastAttemptedOffset == null)
            ? BigDecimal.valueOf(range.getFrom()).subtract(BigDecimal.ONE, MathContext.DECIMAL128)
            : BigDecimal.valueOf(lastAttemptedOffset);

    
    
    BigDecimal estimateRangeEnd =
        BigDecimal.valueOf(rangeEndEstimator.estimate())
            .max(cur.add(BigDecimal.ONE, MathContext.DECIMAL128));

    
    
    
    BigDecimal splitPos =
        cur.add(
            estimateRangeEnd
                .subtract(cur, MathContext.DECIMAL128)
                .multiply(BigDecimal.valueOf(fractionOfRemainder), MathContext.DECIMAL128)
                .max(BigDecimal.ONE),
            MathContext.DECIMAL128);
    long split = splitPos.longValue();
    if (split > estimateRangeEnd.longValue()) {
      return null;
    }
    OffsetRange res = new OffsetRange(split, range.getTo());
    this.range = new OffsetRange(range.getFrom(), split);
    return SplitResult.of(range, res);
  }
Code review comment: this will only trigger if users override the banner name and don't just put `default_banner.txt` into their classpath root ...which I would think most would do because then they don't need to add anything to `application.properties`  should we instead have it be that if default path can't be resolved then we load our internal resource ?

Code under review:
private String readBannerFile(BannerConfig config) {
        URL resource = Thread.currentThread().getContextClassLoader().getResource(config.path);
        if (resource != null) {
            try (InputStream is = resource.openStream()) {
                byte[] content = FileUtil.readFileContents(is);
                StringBuilder bannerTitle = new StringBuilder(new String(content, StandardCharsets.UTF_8));
                bannerTitle.append('\n');
                if (!config.isDefaultPath()) {
                    bannerTitle.append("Powered by Quarkus v").append(Version.getVersion()).append('\n');
                }
                return bannerTitle.toString();
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
        } else {
            logger.warn("Could not read banner file");
            return "";
        }
    }

Line commented on: if (!config.isDefaultPath()) {

Revised code:
private String readBannerFile(BannerConfig config) {
        try {
            Map.Entry<URL, Boolean> entry = getBanner(config);
            URL bannerResourceURL = entry.getKey();
            if (bannerResourceURL == null) {
                logger.warn("Could not locate banner file");
                return "";
            }
            try (InputStream is = bannerResourceURL.openStream()) {
                byte[] content = FileUtil.readFileContents(is);
                String bannerTitle = new String(content, StandardCharsets.UTF_8);

                int width = 0;
                Scanner scanner = new Scanner(bannerTitle);
                while (scanner.hasNextLine()) {
                    width = Math.max(width, scanner.nextLine().length());
                }

                String tagline = "\n";
                Boolean isDefaultBanner = entry.getValue();
                if (!isDefaultBanner) {
                    tagline = String.format("\n%" + width + "s\n", "Powered by Quarkus v" + Version.getVersion());
                }

                return bannerTitle + tagline;
            }
        } catch (IOException e) {
            logger.warn("Unable to read banner file");
            return "";
        }
    }
Code review comment: Shall we add a comment here saying when a connection can be closable and not?

Code under review:
public TableResourceManager(Connection conn, Statement stmt, boolean connectionClosable) {
        this.connection = conn;
        this.statement = stmt;
        this.connectionClosable = connectionClosable;
        this.resultSets = new HashSet<>(0);
    }

Line commented on: this.connectionClosable = connectionClosable;

Revised code:
public TableResourceManager(Connection conn, Statement stmt, boolean connectionClosable) {
        this.connection = conn;
        this.statement = stmt;
        this.connectionClosable = connectionClosable;
        this.resultSets = new HashSet<>(0);
    }
Code review comment: Here we may have to use a null check for `loadedKeys` because  the variable can be null if any exception is thrown while loading the keys (L106)

Code under review:
public int getChildrenCount() {
        try {
            if (!(jvmValue instanceof ObjectReference)) {
                return 0;
            }
            if (loadedKeys == null) {
                loadAllKeys();
            }
            return loadedKeys.length();
        } catch (Exception ignored) {
            return 0;
        }
    }

Line commented on: return loadedKeys.length();

Revised code:
public int getChildrenCount() {
        try {
            if (!(jvmValue instanceof ObjectReference)) {
                return 0;
            }
            loadAllKeys();
            return loadedKeys == null ? 0 : loadedKeys.length();
        } catch (Exception ignored) {
            return 0;
        }
    }
Code review comment: this uni does not seem to be used and seems to be remains of debug code

Code under review:
public Uni<String> getToken(OidcClients clients, String oidcClientId) {
        Uni<String> tokenUni = clients.getClient("MY_OIDC_CLIENT").getTokens().map(t -> "Bearer " + t.getAccessToken());
        if (oidcClientId == null) {
            return clients.getClient()
                    .getTokens()
                    .map(Tokens::getAccessToken)
                    .map(token -> "Bearer " + token);
        } else {
            return clients.getClient(oidcClientId)
                    .getTokens()
                    .map(Tokens::getAccessToken)
                    .map(token -> "Bearer " + token);
        }
    }

Line commented on: Uni<String> tokenUni = clients.getClient("MY_OIDC_CLIENT").getTokens().map(t -> "Bearer " + t.getAccessToken());

Revised code:
public Uni<String> getToken(OidcClients clients, String oidcClientId) {
        if (oidcClientId == null) {
            return clients.getClient()
                    .getTokens()
                    .map(Tokens::getAccessToken)
                    .map(token -> "Bearer " + token);
        } else {
            return clients.getClient(oidcClientId)
                    .getTokens()
                    .map(Tokens::getAccessToken)
                    .map(token -> "Bearer " + token);
        }
    }
Code review comment: Agreed, that can't happen in the quickstart as people might use it. As you already use Jackson and Vert.x JsonObject, looks like you could use that.

Code under review:
private List<Fruit> search(String term, String match) throws IOException {
        Request request = new Request(
                "GET",
                "/fruits/_search");
        String query = String.format("{\"query\": { \"match\": { \"%s\":\"%s\"}}}", term, match);
        request.setJsonEntity(query);
        Response response = restClient.performRequest(request);
        String responseBody = EntityUtils.toString(response.getEntity());

        JsonObject json = new JsonObject(responseBody);
        JsonArray hits = json.getJsonObject("hits").getJsonArray("hits");
        List<Fruit> results = new ArrayList<>(hits.size());
        for (int i = 0; i < hits.size(); i++) {
            JsonObject hit = hits.getJsonObject(i);
            Fruit fruit = hit.getJsonObject("_source").mapTo(Fruit.class);
            results.add(fruit);
        }
        return results;
    }

Line commented on: String query = String.format("{\"query\": { \"match\": { \"%s\":\"%s\"}}}", term, match);

Revised code:
private List<Fruit> search(String term, String match) throws IOException {
        Request request = new Request(
                "GET",
                "/fruits/_search");
        
        JsonObject termJson = new JsonObject().put(term, match);
        JsonObject matchJson = new JsonObject().put("match", termJson);
        JsonObject queryJson = new JsonObject().put("query", matchJson);
        request.setJsonEntity(queryJson.encode());
        Response response = restClient.performRequest(request);
        String responseBody = EntityUtils.toString(response.getEntity());

        JsonObject json = new JsonObject(responseBody);
        JsonArray hits = json.getJsonObject("hits").getJsonArray("hits");
        List<Fruit> results = new ArrayList<>(hits.size());
        for (int i = 0; i < hits.size(); i++) {
            JsonObject hit = hits.getJsonObject(i);
            Fruit fruit = hit.getJsonObject("_source").mapTo(Fruit.class);
            results.add(fruit);
        }
        return results;
    }
}
Code review comment: What are the possible `stringValue ` are we getting here?

Code under review:
private static String processStringValue(String stringValue) {
        if (stringValue.startsWith("stream <stream") && stringValue.endsWith(">>")) {
            stringValue = stringValue.replaceFirst("stream <stream", "stream").replaceFirst(">>$", ">");
        }
        if (stringValue.startsWith("stream<(") && stringValue.endsWith(")>")) {
            stringValue = stringValue.replaceFirst("stream<\\(", "stream <").replaceFirst("\\)>$", ">");
        }
        return stringValue;
    }

Line commented on: stringValue = stringValue.replaceFirst("stream <stream", "stream").replaceFirst(">>$", ">");

Revised code:
private static String processStringValue(String stringValue) {
        if (stringValue.startsWith("stream <stream") && stringValue.endsWith(">>")) {
            stringValue = stringValue.replaceFirst("^stream <stream", "stream").replaceFirst(">>$", ">");
        }
        if (stringValue.startsWith("stream<(") && stringValue.endsWith(")>")) {
            stringValue = stringValue.replaceFirst("stream<\\(", "stream <").replaceFirst("\\)>$", ">");
        }
        
        stringValue = stringValue.replaceAll(",\\s*", ", ");
        return stringValue;
    }
Code review comment: 1/ you're right, the modulus is useless here and I'll just check the number of elements  2/ It's ok as any cluster should be able to handle 10s/100s of thousands of writes per seconds (if not more). The value of 100 is a heuristic that allows to apply some backpressure per client, and it's expected that the cluster is hammered way more than this through concurrent processes.  The question is then, if many bundles are being processed at the same time, how can one limit the load on the cluster ? Is there a way to limit concurrency by lowering the number of concurrent bundles or should we make the number of concurrent async queries configurable ? I'm still inclined to avoid making this configurable as it would bring an additional knob that is specific to Cassandra, and instead deal with concurrency with "standard" Beam knobs to limit job pressure on external systems.

Code under review:
public void write(T entity) throws ExecutionException, InterruptedException {
      Mapper<T> mapper = (Mapper<T>) mappingManager.mapper(entity.getClass());
      this.writeFutures.add(mapper.saveAsync(entity));
      if (this.writeFutures.size() % CONCURRENT_ASYNC_QUERIES == 0) {
        
        LOG.info("Waiting for a batch of {} Cassandra writes to be executed...",
            CONCURRENT_ASYNC_QUERIES);
        waitForFuturesToFinish();
        this.writeFutures = Lists.newArrayList();
      }
    }

Line commented on: if (this.writeFutures.size() % CONCURRENT_ASYNC_QUERIES == 0) {

Revised code:
public void write(T entity) throws ExecutionException, InterruptedException {
      Mapper<T> mapper = (Mapper<T>) mappingManager.mapper(entity.getClass());
      this.writeFutures.add(mapper.saveAsync(entity));
      if (this.writeFutures.size() == CONCURRENT_ASYNC_QUERIES) {
        
        
        
        LOG.debug("Waiting for a batch of {} Cassandra writes to be executed...",
            CONCURRENT_ASYNC_QUERIES);
        waitForFuturesToFinish();
        this.writeFutures = Lists.newArrayList();
      }
    }
Code review comment: ```suggestion         assertThat(descriptor.getPartitionKeys).hasSize(1); ```

Code under review:
public void testBasic() {
        final Schema schema =
                Schema.newBuilder()
                        .column("f0", DataTypes.STRING())
                        .column("f1", DataTypes.BIGINT())
                        .primaryKey("f0")
                        .build();

        final TableDescriptor descriptor =
                TableDescriptor.forConnector("test-connector")
                        .schema(schema)
                        .partitionedBy("f0")
                        .comment("Test Comment")
                        .build();

        assertThat(descriptor.getSchema().isPresent()).isTrue();
        assertThat(descriptor.getSchema().get()).isEqualTo(schema);

        assertThat(descriptor.getPartitionKeys().size()).isEqualTo(1);
        assertThat(descriptor.getPartitionKeys().get(0)).isEqualTo("f0");

        assertThat(descriptor.getOptions().size()).isEqualTo(1);
        assertThat(descriptor.getOptions().get("connector")).isEqualTo("test-connector");

        assertThat(descriptor.getComment().orElse(null)).isEqualTo("Test Comment");
    }

Line commented on: assertThat(descriptor.getPartitionKeys().size()).isEqualTo(1);

Revised code:
public void testBasic() {
        final Schema schema =
                Schema.newBuilder()
                        .column("f0", DataTypes.STRING())
                        .column("f1", DataTypes.BIGINT())
                        .primaryKey("f0")
                        .build();

        final TableDescriptor descriptor =
                TableDescriptor.forConnector("test-connector")
                        .schema(schema)
                        .partitionedBy("f0")
                        .comment("Test Comment")
                        .build();

        assertThat(descriptor.getSchema()).isPresent();
        assertThat(descriptor.getSchema().get()).isEqualTo(schema);

        assertThat(descriptor.getPartitionKeys()).hasSize(1);
        assertThat(descriptor.getPartitionKeys().get(0)).isEqualTo("f0");

        assertThat(descriptor.getOptions()).hasSize(1);
        assertThat(descriptor.getOptions().get("connector")).isEqualTo("test-connector");

        assertThat(descriptor.getComment().orElse(null)).isEqualTo("Test Comment");
    }
Code review comment: Good catch, long term I think we need to add some more metadata to `BinaryDataContent` where we don't need to do all this type checking as it breaks when new types are added.

Code under review:
private RequestBody toOkHttpRequestBody(BinaryData bodyContent, HttpHeaders headers) {
        if (bodyContent == null) {
            return EMPTY_REQUEST_BODY;
        }

        String contentType = headers.getValue(HttpHeaderName.CONTENT_TYPE);
        MediaType mediaType = (contentType == null) ? null : MediaType.parse(contentType);

        BinaryDataContent content = BinaryDataHelper.getContent(bodyContent);

        if (content instanceof ByteArrayContent
            || content instanceof StringContent
            || content instanceof SerializableContent
            || content instanceof ByteBufferContent) {
            return RequestBody.create(content.toBytes(), mediaType);
        } else {
            long effectiveContentLength = getRequestContentLength(content, headers);
            if (content instanceof InputStreamContent) {
                
                return new OkHttpInputStreamRequestBody(
                    (InputStreamContent) content, effectiveContentLength, mediaType);
            } else if (content instanceof FileContent) {
                
                return new OkHttpFileRequestBody((FileContent) content, effectiveContentLength, mediaType);
            } else {
                
                return new OkHttpFluxRequestBody(
                    content, effectiveContentLength, mediaType, httpClient.callTimeoutMillis());
            }
        }
    }

Line commented on: || content instanceof ByteBufferContent) {

Revised code:
private RequestBody toOkHttpRequestBody(BinaryData bodyContent, HttpHeaders headers) {
        if (bodyContent == null) {
            return EMPTY_REQUEST_BODY;
        }

        String contentType = headers.getValue(HttpHeaderName.CONTENT_TYPE);
        MediaType mediaType = (contentType == null) ? null : MediaType.parse(contentType);

        BinaryDataContent content = BinaryDataHelper.getContent(bodyContent);

        if (content instanceof ByteArrayContent
            || content instanceof StringContent
            || content instanceof SerializableContent
            || content instanceof ByteBufferContent) {
            return RequestBody.create(content.toBytes(), mediaType);
        } else {
            long effectiveContentLength = getRequestContentLength(content, headers);
            if (content instanceof InputStreamContent) {
                
                return new OkHttpInputStreamRequestBody(
                    (InputStreamContent) content, effectiveContentLength, mediaType);
            } else if (content instanceof FileContent) {
                
                return new OkHttpFileRequestBody((FileContent) content, effectiveContentLength, mediaType);
            } else {
                
                return new OkHttpFluxRequestBody(
                    content, effectiveContentLength, mediaType, httpClient.callTimeoutMillis());
            }
        }
    }
Code review comment: The idea was that this and the other print statement would form a single line.  Test begins, console view:   `Starting test {qualified name} ({display name})`  Test completes, console view:  `Starting test {qualified name} ({display name}), completed in {millis} ms.{newline}`.

Code under review:
public void afterTestExecution(ExtensionContext context) {
        Method testMethod = context.getRequiredTestMethod();
        long start = getStore(context).remove(testMethod, long.class);
        long duration = System.currentTimeMillis() - start;

        System.out.printf("completed in %d ms.%n", duration);
    }

Line commented on: System.out.printf("completed in %d ms.%n", duration);

Revised code:
public void afterTestExecution(ExtensionContext context) {
        TestInformation testInformation = getStore(context)
            .remove(context.getRequiredTestMethod(), TestInformation.class);
        long duration = System.currentTimeMillis() - testInformation.startMillis;

        System.out.printf("%s completed in %d ms.%n", testInformation.logPrefix, duration);
    }
Code review comment: I think i might know the problem,thanks for reminding me.

Code under review:
private void addBatch(T row) {
        if (row instanceof RowData) {
            RowData rowData = (RowData) row;
            Map<String, String> valueMap = new HashMap<>();
            StringJoiner value = new StringJoiner(this.fieldDelimiter);
            for (int i = 0; i < rowData.getArity() && i < fieldGetters.length; ++i) {
                Object field = fieldGetters[i].getFieldOrNull(rowData);
                if (jsonFormat) {
                    String data = field != null ? field.toString() : null;
                    valueMap.put(this.fieldNames[i], data);
                } else {
                    String data = field != null ? field.toString() : NULL_VALUE;
                    value.add(data);
                }
            }
            Object data = jsonFormat ? valueMap : value.toString();
            batch.add(data);

        }
        if (row instanceof String) {
            batch.add(row);
        }
    }

Line commented on: batch.add(row);

Revised code:
private void addBatch(T row) {
        if (row instanceof RowData) {
            RowData rowData = (RowData) row;
            Map<String, String> valueMap = new HashMap<>();
            StringJoiner value = new StringJoiner(this.fieldDelimiter);
            for (int i = 0; i < rowData.getArity() && i < fieldGetters.length; ++i) {
                Object field = fieldGetters[i].getFieldOrNull(rowData);
                if (jsonFormat) {
                    String data = field != null ? field.toString() : null;
                    valueMap.put(this.fieldNames[i], data);
                } else {
                    String data = field != null ? field.toString() : NULL_VALUE;
                    value.add(data);
                }
            }
            Object data = jsonFormat ? valueMap : value.toString();
            batch.add(data);

        } else if (row instanceof String) {
            batch.add(row);
        } else {
            throw new RuntimeException("The type of element should be 'RowData' or 'String' only.");
        }
    }
Code review comment: Because we use the method arguments inside the function -> `createAtRuntime(beanClass, method, key)`.

Code under review:
public FaultToleranceOperation get(Class<?> beanClass, Method method) {
        String key = getCacheKey(beanClass, method);
        return operationCache.computeIfAbsent(key, new Function<String, FaultToleranceOperation>() {
            @Override
            public FaultToleranceOperation apply(String key) {
                return createAtRuntime(beanClass, method, key);
            }
        });
    }

Line commented on: @Override

Revised code:
public FaultToleranceOperation get(Class<?> beanClass, Method method) {
        CacheKey key = new CacheKey(beanClass, method);
        FaultToleranceOperation existing = operationCache.get(key);
        return existing != null ? existing : operationCache.computeIfAbsent(key, cacheFunction);
    }
Code review comment: On second thoughts is this even correct? Seems like we are throwing `KinesisDataStreamsException` in this PR

Code under review:
public KinesisDataStreamsFailFastException(final Throwable cause) {
            super(
                    "Encountered an exception while persisting records, not retrying due to either: {failOnError} being set or the exception should not be retried.",
                    cause);
        }

Line commented on: "Encountered an exception while persisting records, not retrying due to either: {failOnError} being set or the exception should not be retried.",

Revised code:
public KinesisDataStreamsFailFastException(final Throwable cause) {
            super(ERROR_MESSAGE, cause);
        }
Code review comment: add comment on current strategy - take min across chunks for each event and use that to divide up transit time  As mentioned offline it seems tricky because: - we want total time for all chunks for the work item to match real time elapsed so it aligns with other times tracked for the work item. - chunks may be delayed due to downstream pushback  Another possible way to track would be identifying the max elapsed time within each stage and scaling.

Code under review:
public void append(StreamingGetWorkResponseChunk chunk) {
        if (chunk.hasComputationMetadata()) {
          setMetadata(chunk.getComputationMetadata());
        }

        this.data = data.concat(chunk.getSerializedWorkItem());
        this.bufferedSize += chunk.getSerializedWorkItem().size();
        for (GetWorkStreamTimingInfo info : chunk.getPerWorkItemTimingInfosList()) {
          getWorkStreamTimings.compute(
              info.getEvent(),
              (event, recordedTime) -> {
                Instant newTimingForEvent = Instant.ofEpochMilli(info.getTimestampUsec() / 1000);
                if (recordedTime == null) {
                  return newTimingForEvent;
                }
                switch (event) {
                  case GET_WORK_CREATION_START:
                    return recordedTime.isBefore(newTimingForEvent)
                        ? recordedTime
                        : newTimingForEvent;
                  case GET_WORK_CREATION_END:
                  case GET_WORK_RECEIVED_BY_DISPATCHER:
                  case GET_WORK_FORWARDED_BY_DISPATCHER:
                    return recordedTime.isAfter(newTimingForEvent)
                        ? recordedTime
                        : newTimingForEvent;
                  default:
                    LOG.error("Unknown GetWorkStreamTimingInfo type: " + event.name());
                }
                return recordedTime;
              });
          if (Instant.now().isAfter(workItemReceiveTime)) {
            workItemReceiveTime = Instant.now();
          }
        }
      }

Line commented on: Instant newTimingForEvent = Instant.ofEpochMilli(info.getTimestampUsec() / 1000);

Revised code:
public void append(StreamingGetWorkResponseChunk chunk) {
        if (chunk.hasComputationMetadata()) {
          setMetadata(chunk.getComputationMetadata());
        }

        this.data = data.concat(chunk.getSerializedWorkItem());
        this.bufferedSize += chunk.getSerializedWorkItem().size();
        workTimingInfosTracker.addTimingInfo(chunk.getPerWorkItemTimingInfosList());
      }
Code review comment: Is this done to reuse the BlobBatch object? If so, I am not sure if it's obvious to the user that a single instance of BlobBatch can be reused after each submission. Maybe adding documentation to `submitBatchWithResponse` will make this clear.   The other option is to use this instance just once per submission and a new instance is created for each batch operation. We can simplify this by providing a method to clone an existing batch object (so users don't have to redefine all params).   Currently, this silently prepares for a new batch which user may be unaware of and end up creating a new BlobBatch instance anyway.

Code under review:
Mono<BlobBatchOperationInfo> prepareBlobBatchSubmission() {
        if (batchOperationQueue.isEmpty()) {
            return Mono.error(new UnsupportedOperationException("Empty batch requests aren't allowed."));
        }

        BlobBatchOperationInfo operationInfo = new BlobBatchOperationInfo();
        Deque<BlobBatchOperation<?>> operations = batchOperationQueue;

        
        batchOperationQueue = new ConcurrentLinkedDeque<>();

        return Flux.generate(sink -> {
            if (operations.isEmpty()) {
                operationInfo.finalizeBatchOperations();
                sink.complete();
            } else {
                BlobBatchOperation<?> batchOperation = operations.pop();
                sink.next(batchOperation.getResponse()
                    .subscriberContext(Context.of(BATCH_REQUEST_URL_PATH, batchOperation.getRequestUrlPath(),
                        BATCH_OPERATION_RESPONSE, batchOperation.getBatchOperationResponse(),
                        BATCH_OPERATION_INFO, operationInfo))
                    .subscribe());
            }
        }).then(Mono.just(operationInfo));
    }

Line commented on: batchOperationQueue = new ConcurrentLinkedDeque<>();

Revised code:
Mono<BlobBatchOperationInfo> prepareBlobBatchSubmission() {
        if (batchOperationQueue.isEmpty()) {
            return monoError(logger, new UnsupportedOperationException("Empty batch requests aren't allowed."));
        }

        BlobBatchOperationInfo operationInfo = new BlobBatchOperationInfo();
        Deque<BlobBatchOperation<?>> operations = batchOperationQueue;

        
        batchOperationQueue = new ConcurrentLinkedDeque<>();

        return Flux.generate(sink -> {
            if (operations.isEmpty()) {
                operationInfo.finalizeBatchOperations();
                sink.complete();
            } else {
                BlobBatchOperation<?> batchOperation = operations.pop();
                sink.next(batchOperation.getResponse()
                    .subscriberContext(Context.of(BATCH_REQUEST_URL_PATH, batchOperation.getRequestUrlPath(),
                        BATCH_OPERATION_RESPONSE, batchOperation.getBatchOperationResponse(),
                        BATCH_OPERATION_INFO, operationInfo))
                    .subscribe());
            }
        }).then(Mono.just(operationInfo));
    }
Code review comment: not use info log, i think remove this log is ok

Code under review:
public static boolean dropFromNereids(String dbName, FunctionSearchDesc function) {
        try {
            String fnName = function.getName().getFunction();
            List<DataType> argTypes = Arrays.stream(function.getArgTypes()).map(DataType::fromCatalogType)
                    .collect(Collectors.toList());
            Env.getCurrentEnv().getFunctionRegistry().dropUdf(dbName, fnName, argTypes);
        } catch (Exception e) {
            LOG.warn("Nereids drop function {}:{} failed, caused by: {}", dbName == null ? "_global_" : dbName,
                    function.getName(), e);
        }
        LOG.info(String.format("Nereids drop function: %s:%s OK!", dbName == null ? "_global_" : dbName,
                function.getName()));
        return false;
    }

Line commented on: LOG.info(String.format("Nereids drop function: %s:%s OK!", dbName == null ? "_global_" : dbName,

Revised code:
public static boolean dropFromNereids(String dbName, FunctionSearchDesc function) {
        try {
            String fnName = function.getName().getFunction();
            List<DataType> argTypes = Arrays.stream(function.getArgTypes()).map(DataType::fromCatalogType)
                    .collect(Collectors.toList());
            Env.getCurrentEnv().getFunctionRegistry().dropUdf(dbName, fnName, argTypes);
        } catch (Exception e) {
            LOG.warn("Nereids drop function {}:{} failed, caused by: {}", dbName == null ? "_global_" : dbName,
                    function.getName(), e);
        }
        return false;
    }
Code review comment: @zentol Would you mind elaborating on your suggestions ? Thanks a lot.

Code under review:
void testExternalResourceFailExceedMax() {
        assumeTrue(isExternalResourceSupported());
        assertThatThrownBy(
                        () ->
                                getAdapterWithExternalResources(
                                                SUPPORTED_EXTERNAL_RESOURCE_NAME,
                                                SUPPORTED_EXTERNAL_RESOURCE_CONFIG_KEY)
                                        .getPriorityAndResource(
                                                TASK_EXECUTOR_PROCESS_SPEC_WITH_EXTERNAL_RESOURCE_EXCEED_MAX))
                .isInstanceOf(IllegalStateException.class);
    }

Line commented on: SUPPORTED_EXTERNAL_RESOURCE_CONFIG_KEY)

Revised code:
void testExternalResourceFailExceedMax() {
        assumeThat(isExternalResourceSupported()).isTrue();

        assertThatThrownBy(
                        () ->
                                getAdapterWithExternalResources(
                                                SUPPORTED_EXTERNAL_RESOURCE_NAME,
                                                SUPPORTED_EXTERNAL_RESOURCE_CONFIG_KEY)
                                        .getPriorityAndResource(
                                                TASK_EXECUTOR_PROCESS_SPEC_WITH_EXTERNAL_RESOURCE_EXCEED_MAX))
                .isInstanceOf(IllegalStateException.class);
    }
Code review comment: This probably needs Kotlin 1.5 as well.

Code under review:
public void testGradleKotlinDSL(String language) throws Exception {
        final List<String> codestarts = getExtensionCodestarts();
        generateProjectRunTests("gradle-kotlin-dsl", language, codestarts, Collections.emptyMap());
    }

Line commented on: final List<String> codestarts = getExtensionCodestarts();

Revised code:
public void testGradleKotlinDSL(String language) throws Exception {
        final List<String> codestarts = getExtensionCodestarts();
        generateProjectRunTests("gradle-kotlin-dsl", language, codestarts);
    }
Code review comment: I have checked, and it's ok from my side. However, I prefer inviting more experts to review this PR, it will be more reliable.  Hi @pnowojski @akalash , would you mind take a look this PR in your free time? thanks.

Code under review:
private boolean hasExcessBuffers() {
        return numberOfRequestedOverdraftMemorySegments > 0;
    }

Line commented on: return numberOfRequestedOverdraftMemorySegments > 0;

Revised code:
private boolean hasExcessBuffers() {
        return numberOfRequestedOverdraftMemorySegments > 0
                || numberOfRequestedMemorySegments > currentPoolSize;
    }
Code review comment: how to represent SEMI and ANTI ?

Code under review:
public static FlinkJoinType getJoinType(boolean leftOuter, boolean rightOuter) {
        if (leftOuter && rightOuter) {
            return FULL;
        } else if (leftOuter) {
            return FlinkJoinType.LEFT;
        } else if (rightOuter) {
            return RIGHT;
        } else {
            return INNER;
        }
    }

Line commented on: if (leftOuter && rightOuter) {

Revised code:
public static FlinkJoinType getJoinType(boolean leftOuter, boolean rightOuter) {
        if (leftOuter && rightOuter) {
            return FULL;
        } else if (leftOuter) {
            return FlinkJoinType.LEFT;
        } else if (rightOuter) {
            return RIGHT;
        } else {
            return INNER;
        }
    }
Code review comment: Why not just: stateJobs = idToRoutineLoadJob.values().stream()                         .filter(entity -> entity.getState() == jobState)                         .collect(Collectors.toList());

Code under review:
public List<RoutineLoadJob> getRoutineLoadJobByState(RoutineLoadJob.JobState jobState) throws LoadException {
        List<RoutineLoadJob> jobs = new ArrayList<>();
        Collection<RoutineLoadJob> stateJobs = null;
        LOG.debug("begin to get routine load job by state {}", jobState.name());
        switch (jobState) {
            case NEED_SCHEDULER:
                stateJobs = idToRoutineLoadJob.values().stream()
                        .filter(entity -> entity.getState() == RoutineLoadJob.JobState.NEED_SCHEDULER)
                        .collect(Collectors.toList());
                break;
            case PAUSED:
                stateJobs = idToRoutineLoadJob.values().stream()
                        .filter(entity -> entity.getState() == RoutineLoadJob.JobState.PAUSED)
                        .collect(Collectors.toList());
                break;
            case RUNNING:
                stateJobs = idToRoutineLoadJob.values().stream()
                        .filter(entity -> entity.getState() == RoutineLoadJob.JobState.RUNNING)
                        .collect(Collectors.toList());
                break;
            case STOPPED:
                stateJobs = idToRoutineLoadJob.values().stream()
                        .filter(entity -> entity.getState() == RoutineLoadJob.JobState.STOPPED)
                        .collect(Collectors.toList());
                break;
            default:
                break;
        }
        if (stateJobs != null) {
            jobs.addAll(stateJobs);
            LOG.info("got {} routine load jobs by state {}", jobs.size(), jobState.name());
        }
        return jobs;
    }

Line commented on: case NEED_SCHEDULER:

Revised code:
public List<RoutineLoadJob> getRoutineLoadJobByState(RoutineLoadJob.JobState jobState) throws LoadException {
        List<RoutineLoadJob> jobs = new ArrayList<>();
        Collection<RoutineLoadJob> stateJobs = null;
        LOG.debug("begin to get routine load job by state {}", jobState.name());
        stateJobs = idToRoutineLoadJob.values().stream()
                .filter(entity -> entity.getState() == jobState).collect(Collectors.toList());
        if (stateJobs != null) {
            jobs.addAll(stateJobs);
            LOG.info("got {} routine load jobs by state {}", jobs.size(), jobState.name());
        }
        return jobs;
    }
Code review comment: It's better to put them in one line for shorter code

Code under review:
private InputStream getResourceAsStream(final String resource) throws IOException {
        ClassLoader[] classLoader = new ClassLoader[]{
                Thread.currentThread().getContextClassLoader(),
                getClass().getClassLoader(),
                ClassLoader.getSystemClassLoader(),
        };
        
        for (ClassLoader cl : classLoader) {
            if (null != cl) {
                
                
                InputStream returnValue = cl.getResourceAsStream(resource);
                
                
                if (null == returnValue) {
                    returnValue = cl.getResourceAsStream("/" + resource);
                }
                
                if (null != returnValue) {
                    return returnValue;
                }
            }
        }
        return null;
    }

Line commented on: getClass().getClassLoader(),

Revised code:
private InputStream getResourceAsStream(final String resource) throws IOException {
        ClassLoader[] classLoaders = new ClassLoader[]{
                Thread.currentThread().getContextClassLoader(), getClass().getClassLoader(), ClassLoader.getSystemClassLoader(),
        };
        for (ClassLoader each : classLoaders) {
            if (null != each) {
                InputStream result = each.getResourceAsStream(resource);
                if (null == result) {
                    result = each.getResourceAsStream("/" + resource);
                }
                if (null != result) {
                    return result;
                }
            }
        }
        return null;
    }
Code review comment: Isn't this bug what this PR is attempting to address?

Code under review:
public boolean test(RunnerApi.PTransform pTransform) {
      return false;
      
      
      
    }


Revised code:
public boolean test(RunnerApi.PTransform pTransform) {
      return false;
    }