muttakin2301/ccr2000 huggingface dataset:
Code review comment: `writeCallMetric` and `readCallMetric` look almost identical except `METHOD` field. Is it possible to refactor the common part into a separate method?

Code under review:
public static ServiceCallMetric writeCallMetric(TableReference tableReference) {
    if (tableReference != null) {
      
      
      HashMap<String, String> baseLabels = new HashMap<String, String>();
      baseLabels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "");
      baseLabels.put(MonitoringInfoConstants.Labels.SERVICE, "BigQuery");
      baseLabels.put(MonitoringInfoConstants.Labels.METHOD, "BigQueryBatchWrite");
      baseLabels.put(
          MonitoringInfoConstants.Labels.RESOURCE,
          GcpResourceIdentifiers.bigQueryTable(
              tableReference.getProjectId(),
              tableReference.getDatasetId(),
              tableReference.getTableId()));
      baseLabels.put(
          MonitoringInfoConstants.Labels.BIGQUERY_PROJECT_ID, tableReference.getProjectId());
      baseLabels.put(
          MonitoringInfoConstants.Labels.BIGQUERY_DATASET, tableReference.getDatasetId());
      baseLabels.put(MonitoringInfoConstants.Labels.BIGQUERY_TABLE, tableReference.getTableId());

      return new ServiceCallMetric(MonitoringInfoConstants.Urns.API_REQUEST_COUNT, baseLabels);
    }
    return null;
  }

Line commented on: baseLabels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "");

Revised code:
public static ServiceCallMetric writeCallMetric(TableReference tableReference) {
    return callMetricForMethod(tableReference, "BigQueryBatchWrite");
  }
Code review comment: I'm still not convinced about this test. It's reimplementing production code in many aspects. For example exception handling (this exception ignoring during cancelation/closing) and threading model from `Task`/`StreamTask`, so at any time we modify those, this test has a chance to fail.   Also I'm pretty sure there are `ITCase`s testing for that and the logic there is simpler and more stable manner.

Code under review:
private Callable<Void> readRecoveredStateTask(RecoveredInputChannel inputChannel, ChannelStateReader reader, boolean verifyRelease) {
		return () -> {
			try {
				inputChannel.readRecoveredState(reader);
			} catch (Throwable t) {
				assertTrue("The expected exception should only happen in the case of released channel.", verifyRelease && inputChannel.isReleased());
			}

			return null;
		};
	}

Line commented on: assertTrue("The expected exception should only happen in the case of released channel.", verifyRelease && inputChannel.isReleased());

Revised code:
private Callable<Void> readRecoveredStateTask(RecoveredInputChannel inputChannel, ChannelStateReader reader, boolean verifyRelease) {
		return () -> {
			try {
				inputChannel.readRecoveredState(reader);
			} catch (Throwable t) {
				if (!(verifyRelease && inputChannel.isReleased())) {
					throw new AssertionError("Exceptions are expected here only if the input channel was released", t);
				}
			}

			return null;
		};
	}
Code review comment: Sorry for the bad wording in the question. What I meant to ask was: If an application has no global service id and no rotations, why log here? I.e. shouldn't you only log when rotation set is empty?

Code under review:
void writeContainerEndpointsZK(Optional<String> globalServiceId) {
            if (!params.containerEndpoints().isEmpty()) { 
                containerEndpoints.write(applicationId, params.containerEndpoints());
            } else { 
                if (globalServiceId.isEmpty()) {
                    log.log(LogLevel.WARNING, "Want to write rotations " + rotationsSet + " as container endpoints, but " + applicationId + " has no global-service-id. This should not happen");
                    return;
                }
                containerEndpoints.write(applicationId, toContainerEndpoints(globalServiceId.get(), rotationsSet));
            }
            checkTimeout("write container endpoints to zookeeper");
        }

Line commented on: log.log(LogLevel.WARNING, "Want to write rotations " + rotationsSet + " as container endpoints, but " + applicationId + " has no global-service-id. This should not happen");

Revised code:
void writeContainerEndpointsZK(Optional<String> globalServiceId) {
            if (!params.containerEndpoints().isEmpty()) { 
                containerEndpoints.write(applicationId, params.containerEndpoints());
            } else { 
                if (!rotationsSet.isEmpty()) {
                    if (globalServiceId.isEmpty()) {
                        log.log(LogLevel.WARNING, "Want to write rotations " + rotationsSet + " as container endpoints, but " + applicationId + " has no global-service-id. This should not happen");
                        return;
                    }
                    containerEndpoints.write(applicationId, toContainerEndpoints(globalServiceId.get(), rotationsSet));
                }
            }
            checkTimeout("write container endpoints to zookeeper");
        }
Code review comment: shall we test when it is a `const` as in the issue?

Code under review:
public void testBitwiseUnsignedRightShiftOperator3() {
        int a = 0xff;
        long i = -23445834;
        long j = 5;
        invokeUnsignedRightShiftOperatorTestFunction(a, i, j);
    }

Line commented on: invokeUnsignedRightShiftOperatorTestFunction(a, i, j);

Revised code:
public void testBitwiseUnsignedRightShiftOperator3() {
        int a = 0xff;
        long i = -23445834;
        long j = 5;
        invokeUnsignedRightShiftOperatorTestFunction(a, i, j);
    }
Code review comment: I saw significantly lower system cpu when hardcoding it yesterday. However I did a lot of different tests then, and I am not putting more into it than it is a dimension that should be tested further. And for that to happen I need config control over it.

Code under review:
public Connection(TransportThread parent, Supervisor owner, Spec spec, Object context, boolean tcpNoDelay) {
        super(context);
        this.parent = parent;
        this.owner = owner;
        this.spec = spec;
        this.tcpNoDelay = tcpNoDelay;
        server = false;
        owner.sessionInit(this);
    }

Line commented on: this.tcpNoDelay = tcpNoDelay;

Revised code:
public Connection(TransportThread parent, Supervisor owner, Spec spec, Object context, boolean tcpNoDelay) {
        super(context);
        this.parent = parent;
        this.owner = owner;
        this.spec = spec;
        this.tcpNoDelay = tcpNoDelay;
        server = false;
        owner.sessionInit(this);
    }
Code review comment: I think this is subjective. Also, I will be adding more false statements to the method for this change. Think code is obvious anyway. I prefer to keep the code as it is if there is no objection.

Code under review:
private boolean isReadonlyType(BType sourceType) {
        if (isValueType(sourceType)) {
            return true;
        }

        switch (sourceType.tag) {
            case TypeTags.NIL:
            case TypeTags.ERROR:
            case TypeTags.INVOKABLE:
            case TypeTags.SERVICE:
            case TypeTags.TYPEDESC:
            case TypeTags.HANDLE:
                return true;
        }
        return false;
    }

Line commented on: return true;

Revised code:
private boolean isReadonlyType(BType sourceType) {
        if (isValueType(sourceType)) {
            return true;
        }

        switch (sourceType.tag) {
            case TypeTags.NIL:
            case TypeTags.ERROR:
            case TypeTags.INVOKABLE:
            case TypeTags.SERVICE:
            case TypeTags.TYPEDESC:
            case TypeTags.HANDLE:
                return true;
        }
        return false;
    }
Code review comment: I think this deserves a "final" here, and the line below too.

Code under review:
public static void main(String[] args) throws Exception {
    ServerConfiguration configuration = new ServerConfiguration();
    CmdLineParser parser = new CmdLineParser(configuration);
    try {
      parser.parseArgument(args);
      fromConfig(configuration).run();
    } catch (CmdLineException e) {
      LOG.error("Unable to parse command line arguments {}", Arrays.asList(args), e);
      throw new IllegalArgumentException("Unable to parse command line arguments.", e);
    } catch (Exception e) {
      LOG.error("Hit exception with SamzaJobServer. Exiting...", e);
      throw e;
    }
  }

Line commented on: ServerConfiguration configuration = new ServerConfiguration();

Revised code:
public static void main(String[] args) throws Exception {
    final ServerConfiguration configuration = new ServerConfiguration();
    final CmdLineParser parser = new CmdLineParser(configuration);
    try {
      parser.parseArgument(args);
      fromConfig(configuration).run();
    } catch (CmdLineException e) {
      LOG.error("Unable to parse command line arguments {}", Arrays.asList(args), e);
      throw new IllegalArgumentException("Unable to parse command line arguments.", e);
    } catch (Exception e) {
      LOG.error("Hit exception with SamzaJobServer. Exiting...", e);
      throw e;
    }
  }
Code review comment: @aoyvx Please remove final modifier in method domain.

Code under review:
public void assertRemoveSchemaMetadata() {
        final Map<String, ShardingSphereSchema> map = Maps.of(
                "foo_db_1", mock(ShardingSphereSchema.class),
                "foo_db_2", mock(ShardingSphereSchema.class));
        final FederationDatabaseMetaData federationDatabaseMetaData = new FederationDatabaseMetaData("foo", map);
        assertTrue(federationDatabaseMetaData.getSchemas().containsKey("foo_db_1"));
        federationDatabaseMetaData.removeSchemaMetadata("foo_db_1");
        assertFalse(federationDatabaseMetaData.getSchemas().containsKey("foo_db_1"));
        assertTrue(federationDatabaseMetaData.getSchemas().containsKey("foo_db_2"));
    }

Line commented on: final Map<String, ShardingSphereSchema> map = Maps.of(

Revised code:
public void assertRemoveSchemaMetadata() {
        Map<String, ShardingSphereSchema> map = new HashMap<>();
        map.put("foo_db_1", mock(ShardingSphereSchema.class));
        map.put("foo_db_2", mock(ShardingSphereSchema.class));
        FederationDatabaseMetaData federationDatabaseMetaData = new FederationDatabaseMetaData("foo", map);
        assertTrue(federationDatabaseMetaData.getSchemas().containsKey("foo_db_1"));
        federationDatabaseMetaData.removeSchemaMetadata("foo_db_1");
        assertFalse(federationDatabaseMetaData.getSchemas().containsKey("foo_db_1"));
        assertTrue(federationDatabaseMetaData.getSchemas().containsKey("foo_db_2"));
    }
Code review comment: should use global var for WS and NL

Code under review:
public OnFailClauseNode transform(OnFailClauseNode onFailClauseNode) {
        Token onKeyword = formatToken(onFailClauseNode.onKeyword(), 1, 0);
        Token failKeyword = formatToken(onFailClauseNode.failKeyword(), 1, 0);
        TypeDescriptorNode typeDescriptor = formatNode(onFailClauseNode.typeDescriptor(), 1, 0);
        IdentifierToken failErrorName = formatToken(onFailClauseNode.failErrorName(), 1, 0);
        BlockStatementNode blockStatement = formatNode(onFailClauseNode.blockStatement(), 0, 1);

        return onFailClauseNode.modify()
                .withOnKeyword(onKeyword)
                .withFailKeyword(failKeyword)
                .withTypeDescriptor(typeDescriptor)
                .withFailErrorName(failErrorName)
                .withBlockStatement(blockStatement)
                .apply();
    }

Line commented on: BlockStatementNode blockStatement = formatNode(onFailClauseNode.blockStatement(), 0, 1);

Revised code:
public OnFailClauseNode transform(OnFailClauseNode onFailClauseNode) {
        Token onKeyword = formatToken(onFailClauseNode.onKeyword(), 1, 0);
        Token failKeyword = formatToken(onFailClauseNode.failKeyword(), 1, 0);
        TypeDescriptorNode typeDescriptor = formatNode(onFailClauseNode.typeDescriptor(), 1, 0);
        IdentifierToken failErrorName = formatToken(onFailClauseNode.failErrorName(), 1, 0);
        BlockStatementNode blockStatement = formatNode(onFailClauseNode.blockStatement(),
                this.trailingWS, this.trailingNL);

        return onFailClauseNode.modify()
                .withOnKeyword(onKeyword)
                .withFailKeyword(failKeyword)
                .withTypeDescriptor(typeDescriptor)
                .withFailErrorName(failErrorName)
                .withBlockStatement(blockStatement)
                .apply();
    }
Code review comment: ```suggestion                 tomlKeyEntryNode.location()), tableArrayChild.location(), tableArrayChild.children()); ```

Code under review:
private void addChildParentArrayToParent(TomlTableNode rootTable, TomlTableArrayNode tableArrayChild) {
        TomlTableNode parentTable = getParentTable(rootTable, tableArrayChild);
        List<TomlKeyEntryNode> keys = tableArrayChild.key().keys();
        TomlKeyEntryNode tomlKeyEntryNode = keys.get(keys.size() - 1);
        List<TomlKeyEntryNode> list = new ArrayList<>();
        list.add(tomlKeyEntryNode);

        TomlTableArrayNode newTomlTableArray = new TomlTableArrayNode(new TomlKeyNode(list,
                getLocationOfKeyEntryList(list)), tableArrayChild.location(), tableArrayChild.children());
        TopLevelNode topLevelNode = parentTable.entries().get(newTomlTableArray.key().name());
        if (topLevelNode == null) {
            addChildToTableAST(parentTable, newTomlTableArray);
        } else {
            
            if (topLevelNode instanceof TomlTableArrayNode) {
                ((TomlTableArrayNode) topLevelNode).addChild(newTomlTableArray.children().get(0));
            } else if (topLevelNode instanceof TomlKeyValueNode) {
                TomlDiagnostic nodeExists = dlog.error(newTomlTableArray.location(),
                        DiagnosticErrorCode.ERROR_EXISTING_NODE, newTomlTableArray.key().name());
                parentTable.addDiagnostic(nodeExists);

            } else {
                throw new UnsupportedOperationException();
            }
        }
    }

Line commented on: getLocationOfKeyEntryList(list)), tableArrayChild.location(), tableArrayChild.children());

Revised code:
private void addChildParentArrayToParent(TomlTableNode rootTable, TomlTableArrayNode tableArrayChild) {
        TomlTableNode parentTable = getParentTable(rootTable, tableArrayChild);
        List<TomlKeyEntryNode> keys = tableArrayChild.key().keys();
        TomlKeyEntryNode tomlKeyEntryNode = keys.get(keys.size() - 1);
        List<TomlKeyEntryNode> list = new ArrayList<>();
        list.add(tomlKeyEntryNode);

        TomlTableArrayNode newTomlTableArray = new TomlTableArrayNode(new TomlKeyNode(list,
                tomlKeyEntryNode.location()), tableArrayChild.location(), tableArrayChild.children());
        TopLevelNode topLevelNode = parentTable.entries().get(newTomlTableArray.key().name());
        if (topLevelNode == null) {
            addChildToTableAST(parentTable, newTomlTableArray);
        } else {
            
            if (topLevelNode instanceof TomlTableArrayNode) {
                ((TomlTableArrayNode) topLevelNode).addChild(newTomlTableArray.children().get(0));
            } else if (topLevelNode instanceof TomlKeyValueNode) {
                TomlDiagnostic nodeExists = dlog.error(newTomlTableArray.location(),
                        DiagnosticErrorCode.ERROR_EXISTING_NODE, newTomlTableArray.key().name());
                parentTable.addDiagnostic(nodeExists);

            } else {
                throw new UnsupportedOperationException();
            }
        }
    }
Code review comment: ```suggestion                         "Synthetic bean does not provide a creation method, use ExtendedBeanConfigurator#creator(), ExtendedBeanConfigurator#supplier(), ExtendedBeanConfigurator#createWith()  or ExtendedBeanConfigurator#runtimeValue()"); ```

Code under review:
public SyntheticBeanBuildItem done() {
            if (supplier == null && runtimeValue == null && fun == null && creatorConsumer == null) {
                throw new IllegalStateException(
                        "Synthetic bean does not provide a creation method, use ExtendedBeanConfigurator
            }
            return new SyntheticBeanBuildItem(this);
        }

Line commented on: "Synthetic bean does not provide a creation method, use ExtendedBeanConfigurator

Revised code:
public SyntheticBeanBuildItem done() {
            if (supplier == null && runtimeValue == null && fun == null && creatorConsumer == null) {
                throw new IllegalStateException(
                        "Synthetic bean does not provide a creation method, use ExtendedBeanConfigurator
            }
            return new SyntheticBeanBuildItem(this);
        }
Code review comment: Hello, I am curious if some work is done on  https://issues.apache.org/jira/browse/FLINK-29267 ? I require this solution, I tried to check the files but not sure how can i access connector configurations in PostgresRowConverter file.

Code under review:
protected JdbcSerializationConverter createExternalConverter(LogicalType type) {
        switch (type.getTypeRoot()) {
            case CHAR:
            case VARCHAR:
                return (val, index, statement) -> {
                    String valString = val.getString(index).toString();

                    if (UUID_REGEX_PATTERN.matcher(valString).matches()) {
                        statement.setObject(index, UUID.fromString(valString));
                    } else {
                        statement.setString(index, valString);
                    }
                };
        }

        return super.createExternalConverter(type);
    }

Line commented on: if (UUID_REGEX_PATTERN.matcher(valString).matches()) {

Revised code:
protected JdbcSerializationConverter createExternalConverter(LogicalType type) {
        switch (type.getTypeRoot()) {
            case CHAR:
            case VARCHAR:
                return (val, index, statement) -> {
                    String valString = val.getString(index).toString();

                    if (UUID_REGEX_PATTERN.matcher(valString).matches()) {
                        statement.setObject(index, UUID.fromString(valString));
                    } else {
                        statement.setString(index, valString);
                    }
                };
        }

        return super.createExternalConverter(type);
    }
Code review comment: This method is only protected by `readLock`, but you do write operation here. Why not just save the splitted rules in `commonProperties`?

Code under review:
public String[] getSqlBlockRules() {
        if (this.sqlBlockRulesSplit.length != 0) {
            return this.sqlBlockRulesSplit;
        }
        String sqlBlockRules = commonProperties.getSqlBlockRules();
        if (StringUtils.isNotEmpty(sqlBlockRules)) {
            this.sqlBlockRulesSplit = sqlBlockRules.replace(" ", "").split(",");
        }
        return this.sqlBlockRulesSplit;
    }

Line commented on: String sqlBlockRules = commonProperties.getSqlBlockRules();

Revised code:
public String[] getSqlBlockRules() {
        return commonProperties.getSqlBlockRulesSplit();
    }
Code review comment: Following is from the test case we have added.  `BAssertUtil.validateError(resultNeg, i++, "invalid escape sequence '\\\u0000'", 4, 9);`  That string is displayed as below in IntelliJ ![Screenshot from 2021-09-13 14-43-53](https://user-images.githubusercontent.com/39232462/133057806-a9dcd175-e46a-4cb7-8d77-616bbc690f1d.png)

Code under review:
private void processIdentifierEnd() {
        while (!reader.isEOF()) {
            int nextChar = reader.peek();
            if (isIdentifierFollowingChar(nextChar)) {
                reader.advance();
                continue;
            }

            if (nextChar != LexerTerminals.BACKSLASH) {
                break;
            }

            

            nextChar = reader.peek(1);
            switch (nextChar) {
                case LexerTerminals.NEWLINE:
                case LexerTerminals.CARRIAGE_RETURN:
                case LexerTerminals.TAB:
                    reader.advance();
                    reportInvalidEscapeSequence((char) 0);
                    break;
                case 'u':
                    
                    if (reader.peek(2) == LexerTerminals.OPEN_BRACE) {
                        processNumericEscape();
                    } else {
                        reader.advance(2);
                    }
                    continue;
                default:
                    if (!isValidQuotedIdentifierEscapeChar(nextChar)) {
                        reportInvalidEscapeSequence((char) nextChar);
                    }

                    reader.advance(2);
                    continue;
            }
            break;
        }
    }

Line commented on: reportInvalidEscapeSequence((char) 0);

Revised code:
private void processIdentifierEnd() {
        while (!reader.isEOF()) {
            int nextChar = reader.peek();
            if (isIdentifierFollowingChar(nextChar)) {
                reader.advance();
                continue;
            }

            if (nextChar != LexerTerminals.BACKSLASH) {
                break;
            }

            

            nextChar = reader.peek(1);
            switch (nextChar) {
                case LexerTerminals.NEWLINE:
                case LexerTerminals.CARRIAGE_RETURN:
                case LexerTerminals.TAB:
                    reader.advance();
                    reportLexerError(DiagnosticErrorCode.ERROR_INVALID_ESCAPE_SEQUENCE, "");
                    break;
                case 'u':
                    
                    if (reader.peek(2) == LexerTerminals.OPEN_BRACE) {
                        processNumericEscape();
                    } else {
                        reader.advance(2);
                    }
                    continue;
                default:
                    if (!isValidQuotedIdentifierEscapeChar(nextChar)) {
                        reportInvalidEscapeSequence((char) nextChar);
                    }

                    reader.advance(2);
                    continue;
            }
            break;
        }
    }
Code review comment: I think leaving it here is the cleanest way now that I've looked at the usage again.  It needs to eventually be copied to a byte[] in this case so it can be passed over to the native reader.

Code under review:
public static byte[] getPosition(@Nullable ShufflePosition shufflePosition) {
    if (shufflePosition == null) {
      return null;
    }
    Preconditions.checkArgument(shufflePosition instanceof ByteArrayShufflePosition);
    ByteArrayShufflePosition adapter = (ByteArrayShufflePosition) shufflePosition;
    return adapter.getPosition().toByteArray();
  }

Line commented on: return adapter.getPosition().toByteArray();

Revised code:
public static byte[] getPosition(@Nullable ShufflePosition shufflePosition) {
    if (shufflePosition == null) {
      return null;
    }
    Preconditions.checkArgument(shufflePosition instanceof ByteArrayShufflePosition);
    ByteArrayShufflePosition adapter = (ByteArrayShufflePosition) shufflePosition;
    return adapter.getPosition().toByteArray();
  }
Code review comment: Ah ah I thought about it too but decided to not bother you with this but if we are two, let's be picky :). Yeah, it would sure be a nice improvement. Maybe, use a `Set` instead of a `List`. I suppose a `HashSet` would be good enough.

Code under review:
public static RolesAllowedCheck of(String[] allowedRoles) {
        return CACHE.computeIfAbsent(Arrays.asList(allowedRoles), new Function<List<String>, RolesAllowedCheck>() {
            @Override
            public RolesAllowedCheck apply(List<String> allowedRolesList) {
                return new RolesAllowedCheck(allowedRolesList.toArray(new String[0]));
            }
        });
    }

Line commented on: return CACHE.computeIfAbsent(Arrays.asList(allowedRoles), new Function<List<String>, RolesAllowedCheck>() {

Revised code:
public static RolesAllowedCheck of(String[] allowedRoles) {
        return CACHE.computeIfAbsent(Arrays.asList(allowedRoles), new Function<List<String>, RolesAllowedCheck>() {
            @Override
            public RolesAllowedCheck apply(List<String> allowedRolesList) {
                return new RolesAllowedCheck(allowedRolesList.toArray(new String[0]));
            }
        });
    }
Code review comment: ```suggestion                     "JDBC Store configured but '%s' datasource is missing. You can configure your datasource by following the guide available at: https://quarkus.io/guides/datasource", ```

Code under review:
public QuarkusQuartzConnectionPoolProvider() {
        InstanceHandle<AgroalDataSource> instanceHandle;
        ArcContainer container = Arc.container();
        boolean useDefaultDataSource = "QUARKUS_QUARTZ_DEFAULT_DATASOURCE".equals(dataSourceName);
        if (useDefaultDataSource) {
            instanceHandle = container.instance(AgroalDataSource.class);
        } else {
            instanceHandle = container.instance(AgroalDataSource.class, new DataSourceLiteral(dataSourceName));
        }
        if (instanceHandle.isAvailable()) {
            this.dataSource = instanceHandle.get();
        } else {
            String message = String.format(
                    "JDBC Store configured but '%s' datasource is missing. You can configure your datasource by following the guide available at: https:
                    useDefaultDataSource ? "default" : dataSourceName);
            throw new IllegalStateException(message);
        }
    }

Line commented on: "JDBC Store configured but '%s' datasource is missing. You can configure your datasource by following the guide available at: https:

Revised code:
public QuarkusQuartzConnectionPoolProvider() {
        final ArcContainer container = Arc.container();
        final InstanceHandle<AgroalDataSource> instanceHandle;
        final boolean useDefaultDataSource = "QUARKUS_QUARTZ_DEFAULT_DATASOURCE".equals(dataSourceName);
        if (useDefaultDataSource) {
            instanceHandle = container.instance(AgroalDataSource.class);
        } else {
            instanceHandle = container.instance(AgroalDataSource.class, new DataSourceLiteral(dataSourceName));
        }
        if (instanceHandle.isAvailable()) {
            this.dataSource = instanceHandle.get();
        } else {
            String message = String.format(
                    "JDBC Store configured but '%s' datasource is missing. You can configure your datasource by following the guide available at: https:
                    useDefaultDataSource ? "default" : dataSourceName);
            throw new IllegalStateException(message);
        }
    }
Code review comment: I don't think this it's  reasonable.

Code under review:
public Operator(OperatorType opType) {
        this.opType = opType;
    }

Line commented on: this.opType = opType;

Revised code:
public Operator(OperatorType opType) {
        this.opType = opType;
    }
Code review comment: Is it correct that this error is getting logged? Only `Context` is an unknown type here, right? Even though there are errors when attempting to resolve `FunctionEntry`'s members, it is not unknown?

Code under review:
public void testFunctionPointerAsVariable() {
        CompileResult result =
                BCompileUtil.compile("test-src/expressions/lambda/negative/fp-type-mismatch1-negative.bal");
        Assert.assertEquals(result.getErrorCount(), 3);
        BAssertUtil.validateError(result, 0, "incompatible types: expected 'function (string,int) returns " +
                "(boolean)', found 'function (string,float) returns (boolean)'", 2, 53);
        BAssertUtil.validateError(result, 1, "unknown type 'Context'", 10, 29);
        BAssertUtil.validateError(result, 2, "unknown type 'FunctionEntry'", 12, 5);
    }

Line commented on: BAssertUtil.validateError(result, 2, "unknown type 'FunctionEntry'", 12, 5);

Revised code:
public void testFunctionPointerAsVariable() {
        CompileResult result =
                BCompileUtil.compile("test-src/expressions/lambda/negative/fp-type-mismatch1-negative.bal");
        Assert.assertEquals(result.getErrorCount(), 3);
        BAssertUtil.validateError(result, 0, "incompatible types: expected 'function (string,int) returns " +
                "(boolean)', found 'function (string,float) returns (boolean)'", 2, 53);
        BAssertUtil.validateError(result, 1, "unknown type 'Context'", 10, 29);
        BAssertUtil.validateError(result, 2, "unknown type 'FunctionEntry'", 12, 5);
    }
Code review comment: Could we preserve this existing line case. just add one more line to test `*||**|`, like ``` // complete delimiter "Whether 'tis nobler in the mind to suffer |*", // edge case: partial delimiter then complete delimiter "The slings and arrows of outrageous fortune,*||**|" // truncated delimiter "Or to take arms against a sea of troubles,|" ```

Code under review:
public void testReadStringsWithCustomDelimiter() throws Exception {
      final String[] inputStrings =
          new String[] {
            
            "To be, or not to be: that |is the question: ",
            
            "To be, or not to be: that *is the question: ",
            
            "Whether 'tis nobler in the mind to suffer *||**|",
            
            "The slings and arrows of outrageous fortune,|"
          };

      File tmpFile = tempFolder.newFile("tmpfile.txt");
      String filename = tmpFile.getPath();

      try (Writer writer = Files.newBufferedWriter(tmpFile.toPath(), UTF_8)) {
        writer.write(Joiner.on("").join(inputStrings));
      }

      PAssert.that(p.apply(TextIO.read().from(filename).withDelimiter(new byte[] {'|', '*'})))
          .containsInAnyOrder(
              "To be, or not to be: that |is the question: To be, or not to be: "
                  + "that *is the question: Whether 'tis nobler in the mind to suffer *|",
              "*|The slings and arrows of outrageous fortune,|");
      p.run();
    }

Line commented on: "Whether 'tis nobler in the mind to suffer *||**|",

Revised code:
public void testReadStringsWithCustomDelimiter() throws Exception {
      final String[] inputStrings =
          new String[] {
            
            "To be, or not to be: that |is the question: ",
            
            "To be, or not to be: that *is the question: ",
            
            "Whether 'tis nobler in the mind to suffer |*",
            
            "The slings and arrows of outrageous fortune,*||**|",
            
            "Or to take arms against a sea of troubles,|"
          };

      File tmpFile = tempFolder.newFile("tmpfile.txt");
      String filename = tmpFile.getPath();

      try (Writer writer = Files.newBufferedWriter(tmpFile.toPath(), UTF_8)) {
        writer.write(Joiner.on("").join(inputStrings));
      }

      PAssert.that(p.apply(TextIO.read().from(filename).withDelimiter(new byte[] {'|', '*'})))
          .containsInAnyOrder(
              "To be, or not to be: that |is the question: To be, or not to be: "
                  + "that *is the question: Whether 'tis nobler in the mind to suffer ",
              "The slings and arrows of outrageous fortune,*|",
              "*|Or to take arms against a sea of troubles,|");
      p.run();
    }
Code review comment: Shall we create an issue for this?

Code under review:
private boolean checkFillerValue(BUnionType type) {
        if (type.isNullable()) {
            return true;
        }
        Iterator<BType> iterator = type.getMemberTypes().iterator();
        BType firstMember = iterator.next();
        while (iterator.hasNext()) {
            if (!isSameType(firstMember, iterator.next())) {
                return false;
            }
        }
        return isValueType(firstMember) && hasFillerValue(firstMember);
    }

Line commented on: return isValueType(firstMember) && hasFillerValue(firstMember);

Revised code:
private boolean checkFillerValue(BUnionType type) {
        if (type.isNullable()) {
            return true;
        }
        Iterator<BType> iterator = type.getMemberTypes().iterator();
        BType firstMember = iterator.next();
        while (iterator.hasNext()) {
            if (!isSameType(firstMember, iterator.next())) {
                return false;
            }
        }
        return isValueType(firstMember) && hasFillerValue(firstMember);
    }
Code review comment: It feels inconsistent that the OperatorContext and the internal of taskInfo are given to the `DefaultRuntimeContext` at the same time. I think we either pass the operatorContext that contains all the info or we just pass the required info from the operator context. Same for the other operators.

Code under review:
public void open() throws Exception {
        super.open();
        StreamingRuntimeContext operatorContext = getRuntimeContext();
        TaskInfo taskInfo = operatorContext.getTaskInfo();
        context =
                new DefaultRuntimeContext(
                        operatorContext,
                        taskInfo.getNumberOfParallelSubtasks(),
                        taskInfo.getMaxNumberOfParallelSubtasks(),
                        taskInfo.getTaskName());
        partitionedContext = new DefaultPartitionedContext(context);
        nonPartitionedContext = new DefaultNonPartitionedContext<>(context);
        outputCollector = getOutputCollector();
    }

Line commented on: operatorContext,

Revised code:
public void open() throws Exception {
        super.open();
        StreamingRuntimeContext operatorContext = getRuntimeContext();
        TaskInfo taskInfo = operatorContext.getTaskInfo();
        context =
                new DefaultRuntimeContext(
                        operatorContext.getJobInfo().getJobName(),
                        operatorContext.getJobType(),
                        taskInfo.getNumberOfParallelSubtasks(),
                        taskInfo.getMaxNumberOfParallelSubtasks(),
                        taskInfo.getTaskName(),
                        operatorContext.getMetricGroup());
        partitionedContext =
                new DefaultPartitionedContext(
                        context, this::currentKey, this::setCurrentKey, getProcessingTimeManager());
        outputCollector = getOutputCollector();
        nonPartitionedContext = getNonPartitionedContext();
    }
Code review comment: Where will the key ID come from when this has rolled out?

Code under review:
private String corePublicKeyFlagValue(NodeAgentContext context) {
        return coreEncryptionPublicKeyIdFlag.with(FetchVector.Dimension.NODE_TYPE, context.nodeType().name()).value();
    }

Line commented on: return coreEncryptionPublicKeyIdFlag.with(FetchVector.Dimension.NODE_TYPE, context.nodeType().name()).value();

Revised code:
private String corePublicKeyFlagValue(NodeAgentContext context) {
        return coreEncryptionPublicKeyIdFlag.with(FetchVector.Dimension.NODE_TYPE, context.nodeType().name()).value();
    }
Code review comment: No, this method is simply checking whether the list of media items provided by the user via the ACCEPT header is supported.  The method that was using reflection was removed from this util.

Code under review:
private Response buildViolationReportResponse(ConstraintViolationException cve) {
        Status status = Status.BAD_REQUEST;
        Response.ResponseBuilder builder = Response.status(status);
        builder.header(Validation.VALIDATION_HEADER, "true");

        
        MediaType mediaType = ValidatorMediaTypeUtil.getAcceptMediaTypeFromSupported(headers.getAcceptableMediaTypes());
        if (mediaType == null) {
            mediaType = MediaType.APPLICATION_JSON_TYPE;
        }

        List<ViolationReport.Violation> violationsInReport = new ArrayList<>(cve.getConstraintViolations().size());
        for (ConstraintViolation<?> cv : cve.getConstraintViolations()) {
            violationsInReport.add(new ViolationReport.Violation(cv.getPropertyPath().toString(), cv.getMessage()));
        }
        builder.entity(new ViolationReport("Constraint Violation", status, violationsInReport));
        builder.type(mediaType);

        return builder.build();
    }

Line commented on: MediaType mediaType = ValidatorMediaTypeUtil.getAcceptMediaTypeFromSupported(headers.getAcceptableMediaTypes());

Revised code:
private Response buildViolationReportResponse(ConstraintViolationException cve) {
        Status status = Status.BAD_REQUEST;
        Response.ResponseBuilder builder = Response.status(status);
        builder.header(Validation.VALIDATION_HEADER, "true");

        
        MediaType mediaType = ValidatorMediaTypeUtil.getAcceptMediaTypeFromSupported(headers.getAcceptableMediaTypes());
        if (mediaType == null) {
            mediaType = MediaType.APPLICATION_JSON_TYPE;
        }

        List<ViolationReport.Violation> violationsInReport = new ArrayList<>(cve.getConstraintViolations().size());
        for (ConstraintViolation<?> cv : cve.getConstraintViolations()) {
            violationsInReport.add(new ViolationReport.Violation(cv.getPropertyPath().toString(), cv.getMessage()));
        }
        builder.entity(new ViolationReport("Constraint Violation", status, violationsInReport));
        builder.type(mediaType);

        return builder.build();
    }
Code review comment: `testFile` starts context with `TOP_LEVEL_NODE`. Let's add the codes inside a function

Code under review:
public void testIntermediateClauseStartRecovery() {
        testFile("query-expr/query_expr_source_68.bal", "query-expr/query_expr_assert_68.json");
    }

Line commented on: testFile("query-expr/query_expr_source_68.bal", "query-expr/query_expr_assert_68.json");

Revised code:
public void testIntermediateClauseStartRecovery() {
        testFile("query-expr/query_expr_source_68.bal", "query-expr/query_expr_assert_68.json");
    }
Code review comment: I think I see why this was done, because you need the metric when the KV states are registered. But can't we just create the metric in this class if required, after the DB was opened? then the builder can get the metrics with a getter before passing it into the `RocksDBKeyedStateBackend`.

Code under review:
public void openDB() throws IOException {
		db = RocksDBOperationUtils.openDB(dbPath, columnFamilyDescriptors,
			columnFamilyHandles, columnOptions, dbOptions);
		
		defaultColumnFamilyHandle = columnFamilyHandles.remove(0);
		if (nativeMetricMonitor != null) {
			nativeMetricMonitor.setRocksDB(db);
		}
	}

Line commented on: nativeMetricMonitor.setRocksDB(db);

Revised code:
public void openDB() throws IOException {
		db = RocksDBOperationUtils.openDB(
			dbPath,
			columnFamilyDescriptors,
			columnFamilyHandles,
			RocksDBOperationUtils.createColumnFamilyOptions(columnFamilyOptionsFactory, "default"),
			dbOptions);
		
		defaultColumnFamilyHandle = columnFamilyHandles.remove(0);
		
		nativeMetricMonitor = nativeMetricOptions.isEnabled() ?
			new RocksDBNativeMetricMonitor(nativeMetricOptions, metricGroup, db) : null;
	}
Code review comment: Fixed and a test case added.

Code under review:
private STNode parseTableConstructorExpr(SyntaxKind nextTokenKind, STNode tableKeyword, STNode keySpecifier) {
        STNode openBracket;
        STNode rowList;
        STNode closeBracket;

        
        if (nextTokenKind == SyntaxKind.KEY_KEYWORD) {
            keySpecifier = parseKeySpecifier();
            nextTokenKind = peek().kind;
        }

        switch (nextTokenKind) {
            case OPEN_BRACKET_TOKEN:
                openBracket = parseOpenBracket();
                rowList = parseRowList();
                closeBracket = parseCloseBracket();
                endContext();
                break;
            default:
                Solution solution = recover(peek(), ParserRuleContext.TABLE_KEYWORD_RHS, tableKeyword, keySpecifier);

                
                
                
                if (solution.action == Action.REMOVE) {
                    endContext();
                    return solution.recoveredNode;
                }
                return parseTableConstructorExpr(solution.tokenKind, tableKeyword, keySpecifier);
        }
        return STNodeFactory.createTableConstructorExpressionNode(tableKeyword,
                                                                  keySpecifier,
                                                                  openBracket,
                                                                  rowList,
                                                                  closeBracket);
    }

Line commented on: switch (nextTokenKind) {

Revised code:
private STNode parseTableConstructorExpr(SyntaxKind nextTokenKind, STNode tableKeyword, STNode keySpecifier) {
        STNode openBracket;
        STNode rowList;
        STNode closeBracket;

        switch (nextTokenKind) {
            case KEY_KEYWORD:
                keySpecifier = parseKeySpecifier();
                break;
            case OPEN_BRACKET_TOKEN:
                break;
            default:
                Solution solution = recover(peek(), ParserRuleContext.TABLE_KEYWORD_RHS, tableKeyword, keySpecifier);

                
                
                
                if (solution.action == Action.REMOVE) {
                    return solution.recoveredNode;
                }

                return parseTableConstructorExpr(solution.tokenKind, tableKeyword, keySpecifier);
        }

        openBracket = parseOpenBracket();
        rowList = parseRowList();
        closeBracket = parseCloseBracket();
        endContext();
        return STNodeFactory.createTableConstructorExpressionNode(tableKeyword,
                                                                  keySpecifier,
                                                                  openBracket,
                                                                  rowList,
                                                                  closeBracket);
    }
Code review comment: ```suggestion         }                  if (symbol.restParam != null) { ``` Shouldn't there be a new line here?

Code under review:
private BInvokableSymbol resolveInvokableSymbol(BInvokableSymbol symbol, BType newInvokableType, BType boundType) {
        BInvokableSymbol newInvokableSymbol = duplicateSymbol(symbol);
        newInvokableSymbol.type = newInvokableType;

        for (BVarSymbol param : symbol.params) {
            BType newParamType = resolve(param.type, boundType);
            BVarSymbol newVarSymbol = createNewVarSymbol(param, newParamType);
            newInvokableSymbol.params.add(newVarSymbol);
        }
        if (symbol.restParam != null) {
            BType newReturnType = resolve(symbol.restParam.type, boundType);
            newInvokableSymbol.restParam = createNewVarSymbol(symbol.restParam, newReturnType);
        }
        newInvokableSymbol.retType = resolve(symbol.retType, boundType);
        return newInvokableSymbol;
    }

Line commented on: newInvokableSymbol.params.add(newVarSymbol);

Revised code:
private BInvokableSymbol resolveInvokableSymbol(BInvokableSymbol symbol, BType newInvokableType, BType boundType) {
        BInvokableSymbol newInvokableSymbol = duplicateSymbol(symbol);
        newInvokableSymbol.type = newInvokableType;

        for (BVarSymbol param : symbol.params) {
            BType newParamType = resolve(param.type, boundType);
            BVarSymbol newVarSymbol = createNewVarSymbol(param, newParamType);
            newInvokableSymbol.params.add(newVarSymbol);
        }

        if (symbol.restParam != null) {
            BType newReturnType = resolve(symbol.restParam.type, boundType);
            newInvokableSymbol.restParam = createNewVarSymbol(symbol.restParam, newReturnType);
        }

        newInvokableSymbol.retType = resolve(symbol.retType, boundType);
        return newInvokableSymbol;
    }
Code review comment: Doesn't this mean that a given deployment will send one request per host in the node repository to the orchestrator? In larger zones that is ~400.

Code under review:
public boolean canAllocateTenantNodeTo(Node host, boolean dynamicProvisioning) {
        if ( ! host.type().canRun(NodeType.tenant)) return false;
        if (host.status().wantToRetire()) return false;
        if (host.allocation().map(alloc -> alloc.membership().retired()).orElse(false)) return false;
        if (suspended(host)) return false;

        if (dynamicProvisioning)
            return EnumSet.of(Node.State.active, Node.State.ready, Node.State.provisioned).contains(host.state());
        else
            return host.state() == Node.State.active;
    }

Line commented on: if (suspended(host)) return false;

Revised code:
public boolean canAllocateTenantNodeTo(Node host, boolean dynamicProvisioning) {
        if ( ! host.type().canRun(NodeType.tenant)) return false;
        if (host.status().wantToRetire()) return false;
        if (host.allocation().map(alloc -> alloc.membership().retired()).orElse(false)) return false;
        if (suspended(host)) return false;

        if (dynamicProvisioning)
            return EnumSet.of(Node.State.active, Node.State.ready, Node.State.provisioned).contains(host.state());
        else
            return host.state() == Node.State.active;
    }
Code review comment: Please remove this useless blank line.

Code under review:
public void execute(final T key, final Runnable childStatement) {
        
    }


Revised code:
public void execute(final T key, final Runnable childStatement) {
        taskFeatures.add(getExecutorService(key).submit(childStatement));
    }
Code review comment: Thanks for your suggestion,i'll update it

Code under review:
public RowData deserialize(byte[] message) throws IOException {
		try {
			final JsonNode root = objectMapper.readTree(message);
			if (root instanceof MissingNode) {
				return null;
			}
			return (RowData) runtimeConverter.convert(root);
		} catch (Throwable t) {
			if (ignoreParseErrors) {
				return null;
			}
			throw new IOException(format("Failed to deserialize JSON '%s'.", new String(message)), t);
		}
	}

Line commented on: if (root instanceof MissingNode) {

Revised code:
public RowData deserialize(byte[] message) throws IOException {
		try {
			final JsonNode root = objectMapper.readTree(message);
			return (RowData) runtimeConverter.convert(root);
		} catch (Throwable t) {
			if (ignoreParseErrors) {
				return null;
			}
			throw new IOException(format("Failed to deserialize JSON '%s'.", new String(message)), t);
		}
	}
Code review comment: I am not sure I follow, could you explain more?

Code under review:
void writerShouldDiscardRetriedEntriesOnTimeout() throws Exception {
        TestSinkInitContextAnyThreadMailbox context = new TestSinkInitContextAnyThreadMailbox();
        TestProcessingTimeService tpts = context.getTestProcessingTimeService();
        TimeoutWriter writer = new TimeoutWriter(context, 1, 10, 100, false);
        writer.setShouldFailRequest(true);
        tpts.setCurrentTime(0);
        writer.write("1", null);
        tpts.setCurrentTime(110);
        
        
        writer.deliverMessage();
        
        writer.flush(false);
        assertThat(destination).containsExactly(1L);
    }


Revised code:
void writerShouldDiscardRetriedEntriesOnTimeout() throws Exception {
        TestSinkInitContextAnyThreadMailbox context = new TestSinkInitContextAnyThreadMailbox();
        TestProcessingTimeService tpts = context.getTestProcessingTimeService();
        TimeoutWriter writer = new TimeoutWriter(context, 1, 10, 100, false);
        writer.setShouldFailRequest(true);
        tpts.setCurrentTime(0);
        writer.write("1", null);
        tpts.setCurrentTime(110);
        
        
        writer.deliverMessage();
        
        writer.flush(false);
        assertThat(destination).containsExactly(1L);
    }
Code review comment: (and remove the dump of the stack trace maybe)

Code under review:
public Message<Integer> create() {
        try {
            Thread.sleep(200);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        threads.add(Thread.currentThread().getName());
        return Message.of(count.incrementAndGet());
    }

Line commented on: e.printStackTrace();

Revised code:
public Message<Integer> create() {
        try {
            Thread.sleep(200);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        threads.add(Thread.currentThread().getName());
        return Message.of(count.incrementAndGet());
    }