muttakin2301/ccr2000 huggingface dataset:
Code review comment: can you add the expected formats in the error message? so that users will receive actionable errors

Code under review:
public static TableReference parseTableSpec(String tableSpec) {
    Matcher match = BigQueryIO.TABLE_SPEC.matcher(tableSpec);
    if (!match.matches()) {
      throw new IllegalArgumentException(
          "Table reference is not in the expected " + "format: " + tableSpec);
    }

    TableReference ref = new TableReference();
    ref.setProjectId(match.group("PROJECT"));

    return ref.setDatasetId(match.group("DATASET")).setTableId(match.group("TABLE"));
  }

Line commented on: "Table reference is not in the expected " + "format: " + tableSpec);

Revised code:
public static TableReference parseTableSpec(String tableSpec) {
    Matcher match = BigQueryIO.TABLE_SPEC.matcher(tableSpec);
    if (!match.matches()) {
      throw new IllegalArgumentException(
          String.format(
              "Table specification [%s] is not in one of the expected formats ("
                  + " [project_id]:[dataset_id].[table_id],"
                  + " [project_id].[dataset_id].[table_id],"
                  + " [dataset_id].[table_id])",
              tableSpec));
    }

    TableReference ref = new TableReference();
    ref.setProjectId(match.group("PROJECT"));

    return ref.setDatasetId(match.group("DATASET")).setTableId(match.group("TABLE"));
  }
Code review comment: @michalvavrik How does it look now ? I've added NPE check for `next` and ISE catch block for `remove`

Code under review:
private void removeInvalidEntries() {
        long now = now();
        for (Iterator<Map.Entry<String, CacheEntry<T>>> it = cacheMap.entrySet().iterator(); it.hasNext();) {
            Map.Entry<String, CacheEntry<T>> next = it.next();
            if (isEntryExpired(next.getValue(), now)) {
                it.remove();
                size.decrementAndGet();
            }
        }
    }

Line commented on: long now = now();

Revised code:
private void removeInvalidEntries() {
        long now = now();
        for (Iterator<Map.Entry<String, CacheEntry<T>>> it = cacheMap.entrySet().iterator(); it.hasNext();) {
            Map.Entry<String, CacheEntry<T>> next = it.next();
            if (next != null) {
                if (isEntryExpired(next.getValue(), now)) {
                    try {
                        it.remove();
                        size.decrementAndGet();
                    } catch (IllegalStateException ex) {
                        
                    }
                }
            }
        }
    }
Code review comment: I think only HiveTableSource performs parallelism inference. But this test won't use HiveTableSource, right?

Code under review:
public void init() throws IOException {
		hiveCatalog = HiveTestUtils.createHiveCatalog();
		tEnv().registerCatalog(hiveCatalog.getName(), hiveCatalog);
		tEnv().useCatalog(hiveCatalog.getName());

		
		tEnv().getConfig().getConfiguration().set(
				HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM, false);

		super.init();
	}

Line commented on: HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM, false);

Revised code:
public void init() throws IOException {
		hiveCatalog = HiveTestUtils.createHiveCatalog();
		tEnv().registerCatalog(hiveCatalog.getName(), hiveCatalog);
		tEnv().useCatalog(hiveCatalog.getName());

		
		tEnv().getConfig().getConfiguration().set(
				HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM, false);

		super.init();
	}
Code review comment: Instead of a, b, c shall we say val1, val2 and val3 because that makes more sense. Or more meaningful like hexStringArray, base64StringArray and numArray.

Code under review:
public void testByteArrayReturn() {
        byte[] a = ByteArrayUtils.hexStringToByteArray("aaabcfccadafcd341a4bdfabcd8912df");
        byte[] b = ByteArrayUtils.decodeBase64("aGVsbG8gYmFsbGVyaW5hICEhIQ==");
        byte[] c = new byte[]{3, 4, 5, 6, 7, 8, 9};
        BValue[] returns = BRunUtil.invoke(result, "testByteArrayReturn", new BValue[]{});
        Assert.assertEquals(returns.length, 3);
        Assert.assertSame(returns[0].getClass(), BByteArray.class);
        Assert.assertSame(returns[1].getClass(), BByteArray.class);
        Assert.assertSame(returns[2].getClass(), BByteArray.class);
        BByteArray byteArray1 = (BByteArray) returns[0];
        BByteArray byteArray2 = (BByteArray) returns[1];
        BByteArray byteArray3 = (BByteArray) returns[2];
        Assert.assertEquals(a.length, byteArray1.size());
        ByteArrayUtils.assertJBytesWithBBytes(a, byteArray1.getBytes());
        Assert.assertEquals(b.length, byteArray2.size());
        ByteArrayUtils.assertJBytesWithBBytes(b, byteArray2.getBytes());
        Assert.assertEquals(c.length, byteArray3.size());
        ByteArrayUtils.assertJBytesWithBBytes(c, byteArray3.getBytes());
    }

Line commented on: byte[] c = new byte[]{3, 4, 5, 6, 7, 8, 9};

Revised code:
public void testByteArrayReturn() {
        byte[] bytes1 = ByteArrayUtils.hexStringToByteArray("aaabcfccadafcd341a4bdfabcd8912df");
        byte[] bytes2 = ByteArrayUtils.decodeBase64("aGVsbG8gYmFsbGVyaW5hICEhIQ==");
        byte[] bytes3 = new byte[]{3, 4, 5, 6, 7, 8, 9};
        BValue[] returns = BRunUtil.invoke(result, "testByteArrayReturn", new BValue[]{});
        Assert.assertEquals(returns.length, 3);
        Assert.assertSame(returns[0].getClass(), BByteArray.class);
        Assert.assertSame(returns[1].getClass(), BByteArray.class);
        Assert.assertSame(returns[2].getClass(), BByteArray.class);
        BByteArray byteArray1 = (BByteArray) returns[0];
        BByteArray byteArray2 = (BByteArray) returns[1];
        BByteArray byteArray3 = (BByteArray) returns[2];
        Assert.assertEquals(bytes1.length, byteArray1.size());
        ByteArrayUtils.assertJBytesWithBBytes(bytes1, byteArray1.getBytes());
        Assert.assertEquals(bytes2.length, byteArray2.size());
        ByteArrayUtils.assertJBytesWithBBytes(bytes2, byteArray2.getBytes());
        Assert.assertEquals(bytes3.length, byteArray3.size());
        ByteArrayUtils.assertJBytesWithBBytes(bytes3, byteArray3.getBytes());
    }
Code review comment: I think `isDefaultActionUnavailable()` is not the best choice here because suspension is a temporary state; some input may come after this check.  What about using `mailboxProcessor.isMailboxLoopRunning()` instead?  It is updated on `InputStatus.END_OF_INPUT` which seems exactly what is needed here.

Code under review:
public void waitForInputProcessing() throws Exception {
		while (taskThread.isAlive()) {
			boolean allEmpty = true;
			for (int i = 0; i < numInputGates; i++) {
				if (!inputGates[i].allQueuesEmpty()) {
					allEmpty = false;
				}
			}

			if (allEmpty) {
				break;
			}
		}

		
		final AtomicBoolean allInputProcessed = new AtomicBoolean();
		final MailboxProcessor mailboxProcessor = taskThread.task.mailboxProcessor;
		final MailboxExecutor mailboxExecutor = mailboxProcessor.getMainMailboxExecutor();
		while (taskThread.isAlive()) {
			try {
				final CountDownLatch latch = new CountDownLatch(1);
				mailboxExecutor.execute(() -> {
					allInputProcessed.set(mailboxProcessor.isDefaultActionUnavailable());
					latch.countDown();
				}, "query-whether-processInput-has-suspend-itself");
				
				latch.await(1, TimeUnit.SECONDS);
			} catch (RejectedExecutionException ex) {
				
			}

			if (allInputProcessed.get()) {
				break;
			}

			try {
				Thread.sleep(1);
			} catch (InterruptedException ignored) {}
		}

		Throwable error = taskThread.getError();
		if (error != null) {
			throw new Exception("Exception in the task thread", error);
		}
	}

Line commented on: allInputProcessed.set(mailboxProcessor.isDefaultActionUnavailable());

Revised code:
public void waitForInputProcessing() throws Exception {
		while (taskThread.isAlive()) {
			boolean allEmpty = true;
			for (int i = 0; i < numInputGates; i++) {
				if (!inputGates[i].allQueuesEmpty()) {
					allEmpty = false;
				}
			}

			if (allEmpty) {
				break;
			}
		}

		
		final AtomicBoolean allInputProcessed = new AtomicBoolean();
		final MailboxProcessor mailboxProcessor = taskThread.task.mailboxProcessor;
		final MailboxExecutor mailboxExecutor = mailboxProcessor.getMainMailboxExecutor();
		while (taskThread.isAlive()) {
			try {
				final CountDownLatch latch = new CountDownLatch(1);
				mailboxExecutor.execute(() -> {
					allInputProcessed.set(mailboxProcessor.isDefaultActionUnavailable());
					latch.countDown();
				}, "query-whether-processInput-has-suspend-itself");
				
				latch.await(1, TimeUnit.SECONDS);
			} catch (RejectedExecutionException ex) {
				
			}

			if (allInputProcessed.get()) {
				break;
			}

			try {
				Thread.sleep(1);
			} catch (InterruptedException ignored) {}
		}

		Throwable error = taskThread.getError();
		if (error != null) {
			throw new Exception("Exception in the task thread", error);
		}
	}
Code review comment: Ok, I add unit test now

Code under review:
public List<Record> fetchRecords(final int batchSize, final int timeout, final TimeUnit timeUnit) {
        List<Record> result = new LinkedList<>();
        long start = System.currentTimeMillis();
        int recordsCount = 0;
        while (batchSize > recordsCount) {
            List<Record> records = queue.poll();
            if (null == records || records.isEmpty()) {
                TimeUnit.MILLISECONDS.sleep(100L);
            } else {
                recordsCount += records.size();
                result.addAll(records);
            }
            if (timeUnit.toMillis(timeout) <= System.currentTimeMillis() - start) {
                break;
            }
        }
        return result;
    }

Line commented on: TimeUnit.MILLISECONDS.sleep(100L);

Revised code:
public List<Record> fetchRecords(final int batchSize, final int timeout, final TimeUnit timeUnit) {
        List<Record> result = new LinkedList<>();
        long start = System.currentTimeMillis();
        int recordsCount = 0;
        while (batchSize > recordsCount) {
            List<Record> records = queue.poll();
            if (null == records || records.isEmpty()) {
                TimeUnit.MILLISECONDS.sleep(Math.min(100, timeUnit.toMillis(timeout)));
            } else {
                recordsCount += records.size();
                result.addAll(records);
            }
            if (timeUnit.toMillis(timeout) <= System.currentTimeMillis() - start) {
                break;
            }
        }
        return result;
    }
Code review comment: Empty list means the solution needs no moves. null means there are no solutions.  I don't understand the second paragraph - if it's null we continue because we found no solution. Otherwise we compare it to the current best, if any.

Code under review:
private Move findMitigatingMove(CapacityChecker.HostFailurePath failurePath) {
        Optional<Node> nodeWhichCantMove = failurePath.failureReason.tenant;
        if (nodeWhichCantMove.isEmpty()) return Move.empty();

        Node node = nodeWhichCantMove.get();
        NodeList allNodes = nodeRepository().list();
        
        
        HostCapacity hostCapacity = new HostCapacity(allNodes, nodeRepository().resourcesCalculator());
        Set<Node> spareHosts = hostCapacity.findSpareHosts(allNodes.hosts().satisfies(node.resources()).asList(), 2);
        List<Node> hosts = allNodes.hosts().except(spareHosts).asList();

        CapacitySolver capacitySolver = new CapacitySolver(hostCapacity, maxIterations);
        List<Move> shortestMitigation = null;
        for (Node spareHost : spareHosts) {
            List<Move> mitigation = capacitySolver.makeRoomFor(node, spareHost, hosts, List.of(), List.of());
            if (mitigation == null) continue;
            if (shortestMitigation == null || shortestMitigation.size() > mitigation.size())
                shortestMitigation = mitigation;
        }
        if (shortestMitigation == null || shortestMitigation.isEmpty()) return Move.empty();
        return shortestMitigation.get(0);
    }

Line commented on: if (shortestMitigation == null || shortestMitigation.size() > mitigation.size())

Revised code:
private Move findMitigatingMove(CapacityChecker.HostFailurePath failurePath) {
        Optional<Node> nodeWhichCantMove = failurePath.failureReason.tenant;
        if (nodeWhichCantMove.isEmpty()) return Move.empty();

        Node node = nodeWhichCantMove.get();
        NodeList allNodes = nodeRepository().list();
        
        
        HostCapacity hostCapacity = new HostCapacity(allNodes, nodeRepository().resourcesCalculator());
        Set<Node> spareHosts = hostCapacity.findSpareHosts(allNodes.hosts().satisfies(node.resources()).asList(), 2);
        List<Node> hosts = allNodes.hosts().except(spareHosts).asList();

        CapacitySolver capacitySolver = new CapacitySolver(hostCapacity, maxIterations);
        List<Move> shortestMitigation = null;
        for (Node spareHost : spareHosts) {
            List<Move> mitigation = capacitySolver.makeRoomFor(node, spareHost, hosts, List.of(), List.of());
            if (mitigation == null) continue;
            if (shortestMitigation == null || shortestMitigation.size() > mitigation.size())
                shortestMitigation = mitigation;
        }
        if (shortestMitigation == null || shortestMitigation.isEmpty()) return Move.empty();
        return shortestMitigation.get(0);
    }
Code review comment: 5 is the default value, turns out we can go with lower than default, I'll change this

Code under review:
public int getPriority() {
        return 9;
    }

Line commented on: return 9;

Revised code:
public int getPriority() {
        return 4; 
    }
Code review comment: Using `StringWriter` because of https://github.com/codejive/java-properties/issues/23

Code under review:
public void writeToDisk() throws IOException {
        if (rootProjectPath != null) {
            Files.write(rootProjectPath.resolve(getSettingsGradlePath()), getModel().getRootSettingsContent().getBytes());
            if (hasRootProjectFile(GRADLE_PROPERTIES_PATH)) {
                try (StringWriter sw = new StringWriter()) {
                    getModel().getRootPropertiesContent().store(sw, "Gradle properties");
                    Files.writeString(rootProjectPath.resolve(GRADLE_PROPERTIES_PATH),
                            sw.toString());
                }
            }
        } else {
            writeToProjectFile(getSettingsGradlePath(), getModel().getSettingsContent().getBytes());
            if (hasProjectFile(GRADLE_PROPERTIES_PATH)) {
                try (StringWriter sw = new StringWriter()) {
                    getModel().getPropertiesContent().store(sw, "Gradle properties");
                    writeToProjectFile(GRADLE_PROPERTIES_PATH, sw.toString());
                }
            }
        }
        writeToProjectFile(getBuildGradlePath(), getModel().getBuildContent());
    }

Line commented on: sw.toString());

Revised code:
public void writeToDisk() throws IOException {
        if (rootProjectPath != null) {
            Files.write(rootProjectPath.resolve(getSettingsGradlePath()), getModel().getRootSettingsContent().getBytes());
            if (hasRootProjectFile(GRADLE_PROPERTIES_PATH)) {
                try (StringWriter sw = new StringWriter()) {
                    getModel().getRootPropertiesContent().store(sw, "Gradle properties");
                    Files.writeString(rootProjectPath.resolve(GRADLE_PROPERTIES_PATH),
                            sw.toString());
                }
            }
        } else {
            writeToProjectFile(getSettingsGradlePath(), getModel().getSettingsContent().getBytes());
            if (hasProjectFile(GRADLE_PROPERTIES_PATH)) {
                try (StringWriter sw = new StringWriter()) {
                    getModel().getPropertiesContent().store(sw, "Gradle properties");
                    writeToProjectFile(GRADLE_PROPERTIES_PATH, sw.toString());
                }
            }
        }
        writeToProjectFile(getBuildGradlePath(), getModel().getBuildContent());
    }
Code review comment: That was the other option I pondered. I decided to go with this one.

Code under review:
public QuarkusCommandOutcome execute() throws QuarkusCommandException {
        
        Matcher matcher = JAVA_VERSION_PATTERN
                .matcher(this.javaTarget != null ? this.javaTarget : System.getProperty("java.version", ""));
        if (matcher.matches() && Integer.parseInt(matcher.group(1)) < 11) {
            invocation.setProperty(JAVA_TARGET, invocation.getBuildTool() == BuildTool.MAVEN ? "1.8" : "1_8");
        } else {
            invocation.setProperty(JAVA_TARGET, "11");
        }

        return new CreateProjectCommandHandler().execute(invocation);
    }

Line commented on: invocation.setProperty(JAVA_TARGET, invocation.getBuildTool() == BuildTool.MAVEN ? "1.8" : "1_8");

Revised code:
public QuarkusCommandOutcome execute() throws QuarkusCommandException {
        
        Matcher matcher = JAVA_VERSION_PATTERN
                .matcher(this.javaTarget != null ? this.javaTarget : System.getProperty("java.version", ""));
        if (matcher.matches() && Integer.parseInt(matcher.group(1)) < 11) {
            invocation.setProperty(JAVA_TARGET, "8");
        } else {
            invocation.setProperty(JAVA_TARGET, "11");
        }

        return new CreateProjectCommandHandler().execute(invocation);
    }
Code review comment: it is a replay code path, no need to print the detailed stack trace.

Code under review:
public void replayTo(long journalId) throws StarException {
        JournalCursor cursor = null;
        try {
            cursor = bdbjeJournal.read(replayedJournalId.get() + 1, journalId);
            replayJournal(cursor);
        } catch (InterruptedException | JournalInconsistentException e) {
            LOG.warn("got interrupt exception or inconsistent exception when replay star mgr journal, {}.",
                    e.getMessage());
            Util.stdoutWithTime(e.getMessage());
            System.exit(-1);
        } catch (Exception e) {
            LOG.warn("got exception when replay star mgr journal", e);
            throw new StarException(ExceptionCode.JOURNAL, e.getMessage());
        } finally {
            if (cursor != null) {
                cursor.close();
            }
        }
    }

Line commented on: LOG.warn("got exception when replay star mgr journal", e);

Revised code:
public void replayTo(long journalId) throws StarException {
        JournalCursor cursor = null;
        try {
            cursor = bdbjeJournal.read(replayedJournalId.get() + 1, journalId);
            replayJournal(cursor);
        } catch (InterruptedException | JournalInconsistentException e) {
            LOG.warn("got interrupt exception or inconsistent exception when replay star mgr journal, {}.",
                    e.getMessage());
            Util.stdoutWithTime(e.getMessage());
            System.exit(-1);
        } catch (Exception e) {
            LOG.warn("got exception when replay star mgr journal", e);
            throw new StarException(ExceptionCode.JOURNAL, e.getMessage());
        } finally {
            if (cursor != null) {
                cursor.close();
            }
        }
    }
Code review comment: Should we add a comment here? IINM, I had the same suggestion, right? May not be immediately obvious.

Code under review:
private static boolean shouldWidenExpressionTypeWithNil(BLangAssignment assignNode) {
        if (!assignNode.expr.getBType().isNullable() || !isAssignmentToOptionalField(assignNode)) {
            return false;
        }
        
        BLangFieldBasedAccess fieldAccessNode = (BLangFieldBasedAccess) assignNode.varRef;
        BRecordType recordType = (BRecordType) Types.getImpliedType(fieldAccessNode.expr.getBType());
        BField field = recordType.fields.get(fieldAccessNode.field.value);
        BType fieldType = Types.getImpliedType(field.getType());
        return TypeTags.isSimpleBasicType(fieldType.tag);
    }

Line commented on: return false;

Revised code:
private static boolean shouldWidenExpressionTypeWithNil(BLangAssignment assignNode) {
        if (!assignNode.expr.getBType().isNullable() || !isAssignmentToOptionalField(assignNode)) {
            return false;
        }
        
        BLangFieldBasedAccess fieldAccessNode = (BLangFieldBasedAccess) assignNode.varRef;
        BRecordType recordType = (BRecordType) Types.getImpliedType(fieldAccessNode.expr.getBType());
        BField field = recordType.fields.get(fieldAccessNode.field.value);
        BType fieldType = Types.getImpliedType(field.getType());
        return TypeTags.isSimpleBasicType(fieldType.tag);
    }
Code review comment: Don't you want to sample the value with the windowing information attached?  The runner would be responsible for pulling out the attributes that are being sampled that can be introspected and/or saved.

Code under review:
public void accept(WindowedValue<T> input) throws Exception {
      
      this.elementCountCounter.inc(input.getWindows().size());
      
      
      this.sampledByteSizeDistribution.tryUpdate(input.getValue(), coder);

      if (outputSampler != null) {
        outputSampler.sample(input.getValue());
      }

      
      
      
      for (ConsumerAndMetadata consumerAndMetadata : consumerAndMetadatas) {
        ExecutionState state = consumerAndMetadata.getExecutionState();
        state.activate();
        try {
          consumerAndMetadata.getConsumer().accept(input);
        } finally {
          state.deactivate();
        }
        this.sampledByteSizeDistribution.finishLazyUpdate();
      }
    }

Line commented on: outputSampler.sample(input.getValue());

Revised code:
public void accept(WindowedValue<T> input) throws Exception {
      
      this.elementCountCounter.inc(input.getWindows().size());
      
      
      this.sampledByteSizeDistribution.tryUpdate(input.getValue(), this.coder);

      if (outputSampler != null) {
        outputSampler.sample(input.getValue());
      }

      
      
      
      executionState.activate();
      try {
        this.delegate.accept(input);
      } finally {
        executionState.deactivate();
      }
      this.sampledByteSizeDistribution.finishLazyUpdate();
    }
Code review comment: Thanks for your reminder.  But I also compile `flink-sql-connector-hive-connector-3.1.2`, and the same exception throw. Then, I decompile and check the class file `org.apache.hadoop.hive.ql.metadata.VirtualColumn` in the `flink-sql-connector-hive-connector-3.1.2.jar`. I found the the line `public static final org.apache.flink.hive.shaded.com.google.common.collect.ImmutableSet<java.lang.String>`.  However it's different from the type `VirtualColumn.VIRTUAL_COLUMN_NAMES` in this method `checkColumnName`.  So, the excpetion throw. When replace with it with a method instead of field reference, we can solve such problem.

Code under review:
private static void checkColumnName(String columnName) throws SemanticException {
        ColumnInfo columnInfo = new ColumnInfo();
        columnInfo.setAlias(columnName);
        if (VirtualColumn.isVirtualColumnBasedOnAlias(columnInfo)) {
            throw new SemanticException("Invalid column name " + columnName);
        }
    }

Line commented on: ColumnInfo columnInfo = new ColumnInfo();

Revised code:
private static void checkColumnName(String columnName) throws SemanticException {
        ColumnInfo columnInfo = new ColumnInfo();
        columnInfo.setAlias(columnName);
        if (VirtualColumn.isVirtualColumnBasedOnAlias(columnInfo)) {
            throw new SemanticException("Invalid column name " + columnName);
        }
    }
Code review comment: Its not clear from this code, but the `HealthChecker` is shared, so cannot be closed here.

Code under review:
public void stop() {
        filebeatRestarter.shutdown();
        if (!terminated.compareAndSet(false, true)) {
            throw new RuntimeException("Can not re-stop a node agent.");
        }
        signalWorkToBeDone();

        do {
            try {
                loopThread.join();
                filebeatRestarter.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
            } catch (InterruptedException e) {
                context.log(logger, LogLevel.ERROR,
                        "Interrupted while waiting for converge thread and filebeatRestarter scheduler to shutdown");
            }
        } while (loopThread.isAlive() || !filebeatRestarter.isTerminated());

        healthChecker.ifPresent(HealthChecker::close);

        context.log(logger, "Stopped");
    }

Line commented on: healthChecker.ifPresent(HealthChecker::close);

Revised code:
public void stop() {
        filebeatRestarter.shutdown();
        if (!terminated.compareAndSet(false, true)) {
            throw new RuntimeException("Can not re-stop a node agent.");
        }
        signalWorkToBeDone();

        do {
            try {
                loopThread.join();
                filebeatRestarter.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
            } catch (InterruptedException e) {
                context.log(logger, LogLevel.ERROR,
                        "Interrupted while waiting for converge thread and filebeatRestarter scheduler to shutdown");
            }
        } while (loopThread.isAlive() || !filebeatRestarter.isTerminated());

        context.log(logger, "Stopped");
    }
Code review comment: I think I got it now. Thanks

Code under review:
private Mono<Void> addIntendedCollectionRidAndSessionToken(RxDocumentServiceRequest request) {
        return applySessionToken(request).then(addIntendedCollectionRid(request));
    }

Line commented on: return applySessionToken(request).then(addIntendedCollectionRid(request));

Revised code:
private Mono<Void> addIntendedCollectionRidAndSessionToken(RxDocumentServiceRequest request) {
        return applySessionToken(request).then(addIntendedCollectionRid(request));
    }
Code review comment: Actually, it was the same. This method declare a dependency on the compile classpath which will make sure that gradle resolve all dependency even included build (which was not always guaranteed). This has no effect on the compile classpath of the task.

Code under review:
public FileCollection getClasspath() {
        return QuarkusGradleUtils.getSourceSet(getProject(), SourceSet.MAIN_SOURCE_SET_NAME).getCompileClasspath();
    }

Line commented on: return QuarkusGradleUtils.getSourceSet(getProject(), SourceSet.MAIN_SOURCE_SET_NAME).getCompileClasspath();

Revised code:
public FileCollection getClasspath() {
        return QuarkusGradleUtils.getSourceSet(getProject(), SourceSet.MAIN_SOURCE_SET_NAME).getCompileClasspath();
    }
Code review comment: why UniqueKey and PrimaryKey are different

Code under review:
public void testBitmapWithPrimaryKey() throws Exception {
        ColumnDef col3 = new ColumnDef("col3", new TypeDef(ScalarType.createType(PrimitiveType.BITMAP)));
        cols.add(col3);
        CreateTableStmt stmt = new CreateTableStmt(false, false, tblNameNoDb, cols, "olap",
                new KeysDesc(KeysType.PRIMARY_KEYS, colsName), null,
                hashDistributioin, null, null, "");
        expectedEx.expect(AnalysisException.class);
        stmt.analyze(analyzer);
    }

Line commented on: expectedEx.expect(AnalysisException.class);

Revised code:
public void testBitmapWithPrimaryKey() throws Exception {
        ColumnDef col3 = new ColumnDef("col3", new TypeDef(ScalarType.createType(PrimitiveType.BITMAP)));
        cols.add(col3);
        CreateTableStmt stmt = new CreateTableStmt(false, false, tblNameNoDb, cols, "olap",
                new KeysDesc(KeysType.PRIMARY_KEYS, colsName), null,
                hashDistributioin, null, null, "");
        stmt.analyze(analyzer);
    }
Code review comment: Ideal way is to keep a default configuration and return values from it.

Code under review:
public String getProperty(String key) {
        String result = null;
        if (key !=null) {
            if (this.properties.containsKey(key)) {
                result = this.properties.getString(key);
            } else {
                return key;
            }
            result = decodeTokenText(result);
        }
        return result;
    }

Line commented on: return key;

Revised code:
public String getProperty(String key) {
        if (this.properties.containsKey(key)) {
            return this.properties.getString(key);
        }
        assert false;
        return this.defaultProperties.getString(key);
    }
Code review comment: `s/Failed reload/Failed to reload/` or `s/Failed reload/Failed reload of/`

Code under review:
public void run() {
            try {
                reloadCryptoMaterial(TransportSecurityOptions.fromJsonFile(tlsOptionsConfigFile), trustManager, keyManager);
            } catch (Throwable t) {
                log.log(Level.SEVERE, String.format("Failed reload crypto material (path='%s'): %s", tlsOptionsConfigFile, t.getMessage()), t);
            }
        }

Line commented on: log.log(Level.SEVERE, String.format("Failed reload crypto material (path='%s'): %s", tlsOptionsConfigFile, t.getMessage()), t);

Revised code:
public void run() {
            try {
                reloadCryptoMaterial(TransportSecurityOptions.fromJsonFile(tlsOptionsConfigFile), trustManager, keyManager);
            } catch (Throwable t) {
                log.log(Level.SEVERE, String.format("Failed to reload crypto material (path='%s'): %s", tlsOptionsConfigFile, t.getMessage()), t);
            }
        }
Code review comment: This no longer works, after parallell steps. It is also superseded by the validateSteps which is called after this method in the constructor.

Code under review:
private static List<Step> completeSteps(List<Step> steps) {
        
        if (steps.stream().anyMatch(step -> step.deploysTo(Environment.prod)) &&
            steps.stream().noneMatch(step -> step.deploysTo(Environment.staging))) {
            steps.add(new DeclaredZone(Environment.staging));
        }
        
        
        if (steps.stream().anyMatch(step -> step.deploysTo(Environment.staging)) &&
            steps.stream().noneMatch(step -> step.deploysTo(Environment.test))) {
            steps.add(new DeclaredZone(Environment.test));
        }
        
        
        DeclaredZone testStep = remove(Environment.test, steps);
        if (testStep != null)
            steps.add(0, testStep);
        DeclaredZone stagingStep = remove(Environment.staging, steps);
        if (stagingStep != null)
            steps.add(1, stagingStep);
        
        return steps;
    }

Line commented on: steps.stream().noneMatch(step -> step.deploysTo(Environment.staging))) {

Revised code:
private static List<Step> completeSteps(List<Step> steps) {
        
        if (steps.stream().anyMatch(step -> step.deploysTo(Environment.prod)) &&
            steps.stream().noneMatch(step -> step.deploysTo(Environment.staging))) {
            steps.add(new DeclaredZone(Environment.staging));
        }
        
        
        if (steps.stream().anyMatch(step -> step.deploysTo(Environment.staging)) &&
            steps.stream().noneMatch(step -> step.deploysTo(Environment.test))) {
            steps.add(new DeclaredZone(Environment.test));
        }
        
        
        DeclaredZone testStep = remove(Environment.test, steps);
        if (testStep != null)
            steps.add(0, testStep);
        DeclaredZone stagingStep = remove(Environment.staging, steps);
        if (stagingStep != null)
            steps.add(1, stagingStep);
        
        return steps;
    }
Code review comment: IMO, that copied code could be much simpler for Quarkus, but I rather copied it 1:1. Not sure what the general policy is here.

Code under review:
private static String getBuildInfo(String propertyId) {
        
        
        
        if (liquibaseBuildProperties == null) {
            try {
                liquibaseBuildProperties = new Properties();
                final Enumeration<URL> propertiesUrls = Scope.getCurrentScope().getClassLoader()
                        .getResources("liquibase.build.properties");
                while (propertiesUrls.hasMoreElements()) {
                    final URL url = propertiesUrls.nextElement();
                    try (InputStream buildProperties = url.openStream()) {
                        if (buildProperties != null) {
                            liquibaseBuildProperties.load(buildProperties);
                        }
                    }
                }
            } catch (IOException e) {
                Scope.getCurrentScope().getLog(LiquibaseUtil.class).severe("Cannot read liquibase.build.properties", e);
            }
        }

        String value;
        value = liquibaseBuildProperties.getProperty(propertyId);
        if (value == null) {
            value = "UNKNOWN";
        }
        return value;
    }


Revised code:
private static String getBuildInfo(String propertyId) {
        
        
        
        if (liquibaseBuildProperties == null) {
            try {
                liquibaseBuildProperties = new Properties();
                final Enumeration<URL> propertiesUrls = Scope.getCurrentScope().getClassLoader()
                        .getResources("liquibase.build.properties");
                while (propertiesUrls.hasMoreElements()) {
                    final URL url = propertiesUrls.nextElement();
                    try (InputStream buildProperties = url.openStream()) {
                        if (buildProperties != null) {
                            liquibaseBuildProperties.load(buildProperties);
                        }
                    }
                }
            } catch (IOException e) {
                Scope.getCurrentScope().getLog(LiquibaseUtil.class).severe("Cannot read liquibase.build.properties", e);
            }
        }

        String value;
        value = liquibaseBuildProperties.getProperty(propertyId);
        if (value == null) {
            value = "UNKNOWN";
        }
        return value;
    }
Code review comment: we can directly return instruction.pos

Code under review:
private Location getDesugaredPosition(BIRBasicBlock basicBlock) {
        
        
        Location desugaredPos = basicBlock.terminator.pos;
        for (BIRNonTerminator instruction : basicBlock.instructions) {
            if (instruction.pos != null) {
                desugaredPos = instruction.pos;
                break;
            }
        }
        return desugaredPos;
    }

Line commented on: desugaredPos = instruction.pos;

Revised code:
private Location getDesugaredPosition(BIRBasicBlock basicBlock) {
        
        
        for (BIRNonTerminator instruction : basicBlock.instructions) {
            if (instruction.pos != null) {
                return instruction.pos;
            }
        }
        return basicBlock.terminator.pos;
    }
Code review comment: Yes it is. `connection.action()` returns BuildActionExecuter and if you check docs: for forTasks method:  `Specifies the tasks to execute before executing the BuildAction.`  https://docs.gradle.org/current/javadoc/org/gradle/tooling/BuildActionExecuter.html

Code under review:
public static QuarkusModel create(File projectDir, String mode, List<String> jvmArgs, String... tasks) {
        try (ProjectConnection connection = GradleConnector.newConnector()
                .forProjectDirectory(projectDir)
                .connect()) {
            return connection.action(new QuarkusModelBuildAction(mode)).forTasks(tasks).addJvmArguments(jvmArgs).run();
        }
    }

Line commented on: return connection.action(new QuarkusModelBuildAction(mode)).forTasks(tasks).addJvmArguments(jvmArgs).run();

Revised code:
public static QuarkusModel create(File projectDir, String mode, List<String> jvmArgs, String... tasks) {
        try (ProjectConnection connection = GradleConnector.newConnector()
                .forProjectDirectory(projectDir)
                .connect()) {
            return connection.action(new QuarkusModelBuildAction(mode)).forTasks(tasks).addJvmArguments(jvmArgs).run();
        }
    }
Code review comment: I was in doubt whether this exception type is appropriate here or whether an `Optional` should be returned. But at the upper level, it ultimately should be used, and the reason is only known here (so it would be harder to analyze the exception/empty buffers later).

Code under review:
private List<Buffer> getInflightBuffersUnsafe(long checkpointId) throws CheckpointException {
		assert Thread.holdsLock(receivedBuffers);

		if (checkpointId < lastBarrierId) {
			throw new CheckpointException(
				String.format("Sequence number for checkpoint %d is not known (it was likely been overwritten by a newer checkpoint %d)", checkpointId, lastBarrierId),
				CheckpointFailureReason.CHECKPOINT_SUBSUMED); 
		}

		final List<Buffer> inflightBuffers = new ArrayList<>();
		Iterator<SequenceBuffer> iterator = receivedBuffers.iterator();
		
		Iterators.advance(iterator, receivedBuffers.getNumPriorityElements());

		while (iterator.hasNext()) {
			SequenceBuffer sequenceBuffer = iterator.next();
			if (sequenceBuffer.buffer.isBuffer()) {
				if (shouldBeSpilled(sequenceBuffer.sequenceNumber)) {
					inflightBuffers.add(sequenceBuffer.buffer.retainBuffer());
				} else {
					break;
				}
			}
		}

		return inflightBuffers;
	}

Line commented on: throw new CheckpointException(

Revised code:
private List<Buffer> getInflightBuffersUnsafe(long checkpointId) throws CheckpointException {
		assert Thread.holdsLock(receivedBuffers);

		if (checkpointId < lastBarrierId) {
			throw new CheckpointException(
				String.format("Sequence number for checkpoint %d is not known (it was likely been overwritten by a newer checkpoint %d)", checkpointId, lastBarrierId),
				CheckpointFailureReason.CHECKPOINT_SUBSUMED); 
		}

		final List<Buffer> inflightBuffers = new ArrayList<>();
		Iterator<SequenceBuffer> iterator = receivedBuffers.iterator();
		
		Iterators.advance(iterator, receivedBuffers.getNumPriorityElements());

		while (iterator.hasNext()) {
			SequenceBuffer sequenceBuffer = iterator.next();
			if (sequenceBuffer.buffer.isBuffer()) {
				if (shouldBeSpilled(sequenceBuffer.sequenceNumber)) {
					inflightBuffers.add(sequenceBuffer.buffer.retainBuffer());
				} else {
					break;
				}
			}
		}

		return inflightBuffers;
	}
Code review comment: Also, it can't really break anything, can it? In my eyes, this is how `checkPermission` method should look like at the first place.

Code under review:
public Uni<Boolean> checkPermission(Permission permission) {
        return association.getDeferredIdentity()
                .flatMap(new Function<>() {
                    @Override
                    public Uni<? extends Boolean> apply(SecurityIdentity identity) {
                        return identity.checkPermission(permission);
                    }
                });
    }

Line commented on: return association.getDeferredIdentity()

Revised code:
public Uni<Boolean> checkPermission(Permission permission) {
        return association.getDeferredIdentity()
                .flatMap(new Function<>() {
                    @Override
                    public Uni<? extends Boolean> apply(SecurityIdentity identity) {
                        return identity.checkPermission(permission);
                    }
                });
    }
Code review comment: Should the get always return a new instance?

Code under review:
public SimpleVersionedSerializer<KafkaCommittable> getCommittableSerializer() {
        return new KafkaCommittableSerializer();
    }

Line commented on: return new KafkaCommittableSerializer();

Revised code:
public SimpleVersionedSerializer<KafkaCommittable> getCommittableSerializer() {
        return new KafkaCommittableSerializer();
    }
Code review comment: With `AssertJ` it would look more natural:  ``` assertThat(logManager).isInstanceOf(org.jboss.logmanager.LogManager.class); ```

Code under review:
public void consoleOutputTest() {
        LogManager logManager = LogManager.getLogManager();
        Assertions.assertTrue(logManager instanceof org.jboss.logmanager.LogManager);
        DelayedHandler delayedHandler = InitialConfigurator.DELAYED_HANDLER;
        boolean loggerContainsDelayedHandler = Arrays.asList(Logger.getLogger("").getHandlers()).stream()
                .anyMatch(h -> h.equals(delayedHandler));
        Assertions.assertTrue(loggerContainsDelayedHandler);
        Handler[] handlers = delayedHandler.getHandlers();
        Handler handler = Arrays.asList(handlers).stream().filter(h -> (h instanceof ConsoleHandler)).findFirst().get();
        Assertions.assertNotNull(handler);
        Assertions.assertEquals(Level.ALL, handler.getLevel());
        Formatter formatter = handler.getFormatter();
        Assertions.assertTrue(formatter instanceof PatternFormatter);
        PatternFormatter patternFormatter = (PatternFormatter) formatter;
        Assertions.assertEquals(patternFormatter.getPattern(), "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n");
    }

Line commented on: Assertions.assertTrue(logManager instanceof org.jboss.logmanager.LogManager);

Revised code:
public void consoleOutputTest() {
        LogManager logManager = LogManager.getLogManager();
        assertThat(logManager).isInstanceOf(org.jboss.logmanager.LogManager.class);

        DelayedHandler delayedHandler = InitialConfigurator.DELAYED_HANDLER;
        assertThat(Logger.getLogger("").getHandlers()).contains(delayedHandler);

        Handler handler = Arrays.asList(delayedHandler.getHandlers()).stream().filter(h -> (h instanceof ConsoleHandler))
                .findFirst().get();
        assertThat(handler).isNotNull();
        assertThat(handler.getLevel()).isEqualTo(Level.ALL);

        Formatter formatter = handler.getFormatter();
        assertThat(formatter).isInstanceOf(PatternFormatter.class);
        PatternFormatter patternFormatter = (PatternFormatter) formatter;
        assertThat(patternFormatter.getPattern()).isEqualTo("%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n");
    }
Code review comment: Same comment about the iterator here

Code under review:
private static int countMatchingMediaTypes(List<MediaType> produces, List<MediaType> mediaTypes) {
        int count = 0;
        for (MediaType mediaType : mediaTypes) {
            for (MediaType produce : produces) {
                if (mediaType.isCompatible(produce)) {
                    count++;
                    break;
                }
            }
        }

        return count;
    }

Line commented on: for (MediaType mediaType : mediaTypes) {

Revised code:
private static int countMatchingMediaTypes(List<MediaType> produces, List<MediaType> mediaTypes) {
        int count = 0;
        for (int i = 0; i < mediaTypes.size(); i++) {
            MediaType mediaType = mediaTypes.get(i);
            for (int j = 0; j < produces.size(); j++) {
                MediaType produce = produces.get(j);
                if (mediaType.isCompatible(produce)) {
                    count++;
                    break;
                }
            }
        }

        return count;
    }
Code review comment: Not all tuples should fail right? Shouldn't we be able to remove all from a tuple with only a rest field?  ``` [int...] i = [2, 3]; i.removeAll(); // this is valid? ```

Code under review:
private void checkFixedLength(long length) {
        if (arrayType != null && arrayType.getTag() == TypeTags.TUPLE_TAG) {
            throw BLangExceptionHelper.getRuntimeException(BallerinaErrorReasons.INHERENT_TYPE_VIOLATION_ERROR,
                    RuntimeErrors.CANNOT_CHANGE_TUPLE_SIZE);
        }
        if (((BArrayType) this.arrayType).getState() == ArrayState.CLOSED_SEALED) {
            throw BLangExceptionHelper.getRuntimeException(BallerinaErrorReasons.INHERENT_TYPE_VIOLATION_ERROR,
                    RuntimeErrors.ILLEGAL_ARRAY_SIZE, length);
        }
    }

Line commented on: if (arrayType != null && arrayType.getTag() == TypeTags.TUPLE_TAG) {

Revised code:
private void checkFixedLength(long length) {
        if (arrayType != null && arrayType.getTag() == TypeTags.TUPLE_TAG) {
            throw BLangExceptionHelper.getRuntimeException(BallerinaErrorReasons.INHERENT_TYPE_VIOLATION_ERROR,
                    RuntimeErrors.CANNOT_CHANGE_TUPLE_SIZE);
        }
        if (((BArrayType) this.arrayType).getState() == ArrayState.CLOSED_SEALED) {
            throw BLangExceptionHelper.getRuntimeException(BallerinaErrorReasons.INHERENT_TYPE_VIOLATION_ERROR,
                    RuntimeErrors.ILLEGAL_ARRAY_SIZE, length);
        }
    }
Code review comment: I find the way we check the incompatibility confusing atm. There are at least two places that throw a similar exception: 1. `VersionedIOReadeableWritable` from `getIncompatibleVersionError` 2. Here from the `read` method.  I think 1) is actually a dead code now, as we tell in `TypeSerializerSnapshotSerializationProxy` that we support version 1.  Moreover I find the method `KeyedBackendSerializationProxy#getIncompatibleVersionError` misleading. I think at least the `Ops, this should not happen...` is unnecessary. You already get a message from `VersionedIOReadableWritable` that is an unsupported version. This comment does not give us any additional context.   Lastly if you leave the `getIncompatibleVersionError` method I'd rename it something along the lines of "extra information", e.g. `getAdditionalDetailsForIncompatibleVersion`.

Code under review:
public void read(DataInputView in) throws IOException {
            
            super.read(in);
            final int version = getReadVersion();

            switch (version) {
                case 2:
                    serializerSnapshot = deserializeV2(in, userCodeClassLoader);
                    break;
                case 1:
                    throw new IOException(
                            String.format(
                                    "No longer supported version [%d] for TypeSerializerSnapshot. "
                                            + "Please migrate away from the old TypeSerializerConfigSnapshot "
                                            + "and use Flink 1.16 for the migration",
                                    version));
                default:
                    throw new IOException(
                            "Unrecognized version for TypeSerializerSnapshot format: " + version);
            }
        }

Line commented on: version));

Revised code:
public void read(DataInputView in) throws IOException {
            
            super.read(in);
            final int version = getReadVersion();

            switch (version) {
                case 2:
                    serializerSnapshot = deserializeV2(in, userCodeClassLoader);
                    break;
                default:
                    throw new IOException(
                            "Unrecognized version for TypeSerializerSnapshot format: " + version);
            }
        }
Code review comment: Checkpoints are always correct because of following properties: 1) shardIteratorsMap is always in a consistent state, i.e. it is guaranteed to not contain any two shards that are in a parent-child relation. This is a requirement for current logic to properly traverse through splits and merges. 2) All records read from particular shard are removed from the queue in advance() method before shardIterator is removed from shardIteratorsMap. This guarantees that checkpoints will contain particular shardIterator at least as long as there are any unread records from this shard.  So even if shardIteratorsMap is modified an unlimited number of times during any other action, it will always be consistent. The only risk of outdated checkpoint are potential duplicates during restoration.

Code under review:
boolean allShardsUpToDate() {
    boolean shardsUpToDate = true;
    ImmutableMap<String, ShardRecordsIterator> currentShardIterators = shardIteratorsMap.get();
    for (ShardRecordsIterator shardRecordsIterator : currentShardIterators.values()) {
      shardsUpToDate &= shardRecordsIterator.isUpToDate();
    }
    return shardsUpToDate;
  }

Line commented on: shardsUpToDate &= shardRecordsIterator.isUpToDate();

Revised code:
boolean allShardsUpToDate() {
    boolean shardsUpToDate = true;
    ImmutableMap<String, ShardRecordsIterator> currentShardIterators = shardIteratorsMap.get();
    for (ShardRecordsIterator shardRecordsIterator : currentShardIterators.values()) {
      shardsUpToDate &= shardRecordsIterator.isUpToDate();
    }
    return shardsUpToDate;
  }