muttakin2301/ccr2000 huggingface dataset:
Code review comment: This anonymous class is used twice, can you extract it as a private static class and remove the duplication?

Code under review:
public void testWriteQueue() throws Exception {
    final int maxNumRecords = 1000;
    List<RabbitMqMessage> data =
        IntStream.range(0, maxNumRecords)
            .mapToObj(i -> new RabbitMqMessage(("Test " + i).getBytes(StandardCharsets.UTF_8)))
            .collect(Collectors.toList());
    p.apply(Create.of(data))
        .apply(
            RabbitMqIO.write().withUri("amqp:

    final List<String> received = new ArrayList<>();
    ConnectionFactory connectionFactory = new ConnectionFactory();
    connectionFactory.setUri("amqp:
    Connection connection = connectionFactory.newConnection();
    Channel channel = connection.createChannel();
    channel.queueDeclare("TEST", true, false, false, null);
    Consumer consumer =
        new DefaultConsumer(channel) {
          @Override
          public void handleDelivery(
              String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
              throws IOException {
            String message = new String(body, "UTF-8");
            received.add(message);
          }
        };
    channel.basicConsume("TEST", true, consumer);

    p.run();

    while (received.size() < maxNumRecords) {
      Thread.sleep(500);
    }

    assertEquals(maxNumRecords, received.size());
    for (int i = 0; i < maxNumRecords; i++) {
      assertTrue(received.contains("Test " + i));
    }

    channel.close();
    connection.close();
  }

Line commented on: new DefaultConsumer(channel) {

Revised code:
public void testWriteQueue() throws Exception {
    final int maxNumRecords = 1000;
    List<RabbitMqMessage> data =
        generateRecords(maxNumRecords)
            .stream()
            .map(bytes -> new RabbitMqMessage(bytes))
            .collect(Collectors.toList());
    p.apply(Create.of(data))
        .apply(
            RabbitMqIO.write().withUri("amqp:

    final List<String> received = new ArrayList<>();
    ConnectionFactory connectionFactory = new ConnectionFactory();
    connectionFactory.setUri("amqp:
    Connection connection = null;
    Channel channel = null;
    try {
      connection = connectionFactory.newConnection();
      channel = connection.createChannel();
      channel.queueDeclare("TEST", true, false, false, null);
      Consumer consumer = new TestConsumer(channel, received);
      channel.basicConsume("TEST", true, consumer);

      p.run();

      while (received.size() < maxNumRecords) {
        Thread.sleep(500);
      }

      assertEquals(maxNumRecords, received.size());
      for (int i = 0; i < maxNumRecords; i++) {
        assertTrue(received.contains("Test " + i));
      }
    } finally {
      if (channel != null) {
        channel.close();
      }
      if (connection != null) {
        connection.close();
      }
    }
  }
Code review comment: Aren't these all `validateTransform()` calls?

Code under review:
public void testReadValidationFailsMissingConfiguration() {
    HadoopInputFormatIO.Read<String, String> read = HadoopInputFormatIO.<String, String>read();
    thrown.expect(NullPointerException.class);
    read.validate(PipelineOptionsFactory.create());
  }

Line commented on: read.validate(PipelineOptionsFactory.create());

Revised code:
public void testReadValidationFailsMissingConfiguration() {
    HadoopInputFormatIO.Read<String, String> read = HadoopInputFormatIO.<String, String>read();
    thrown.expect(NullPointerException.class);
    read.validateTransform();
  }
Code review comment: What happens service side if it receives an empty `clientFilters` list? The other option would be to defer this instantiation and only perform it when `addFilter` is called and `clientFilters == null`

Code under review:
public FeatureFlagConfigurationSetting(String featureId, boolean isEnabled) {
        this.featureId = featureId;
        this.isEnabled = isEnabled;
        super.setKey(KEY_PREFIX + featureId);
        super.setContentType(FEATURE_FLAG_CONTENT_TYPE);
        clientFilters = new ArrayList<>();
    }

Line commented on: clientFilters = new ArrayList<>();

Revised code:
public FeatureFlagConfigurationSetting(String featureId, boolean isEnabled) {
        this.featureId = featureId;
        this.isEnabled = isEnabled;
        super.setKey(KEY_PREFIX + featureId);
        super.setContentType(FEATURE_FLAG_CONTENT_TYPE);
    }
Code review comment: It might also be better for the new constructor to just take a long and we calculate the share usage in bytes in the constructor.

Code under review:
private Response<ShareStatistics> mapGetStatisticsResponse(SharesGetStatisticsResponse response) {
        ShareStatistics shareStatistics =
            new ShareStatistics((int) (response.getValue().getShareUsageBytes() / (Constants.GB)),
                response.getValue().getShareUsageBytes());

        return new SimpleResponse<>(response, shareStatistics);
    }

Line commented on: new ShareStatistics((int) (response.getValue().getShareUsageBytes() / (Constants.GB)),

Revised code:
private Response<ShareStatistics> mapGetStatisticsResponse(SharesGetStatisticsResponse response) {
        ShareStatistics shareStatistics =
            new ShareStatistics(response.getValue().getShareUsageBytes());

        return new SimpleResponse<>(response, shareStatistics);
    }
Code review comment: This is because `getKeyColumns` is not stable. I've fixed this by using `fullSchema`, PTAL. ```     /**      * NOTE: The result key columns are not in the creating order because `nameToColumn`      * uses unordered hashmap to keeps name to column's mapping.      */     public List<Column> getKeyColumns() {         return getColumns().stream().filter(Column::isKey).collect(Collectors.toList());     } ```

Code under review:
public void testCreateMaterializedViewWithoutSortKeys_Partitioned_1() {
        
        String sql = "create materialized view test_mv_sort_key1 " +
                "partition by c_1_3 " +
                "distributed by hash(c_1_3, c_1_0) buckets 10 " +
                "PROPERTIES (\n" +
                "\"replication_num\" = \"1\"" +
                ") " +
                "as select c_1_3, c_1_0, c_1_4, c_1_5 from t1";
        List<Column> keyColumns = getMaterializedViewKeysChecked(sql);
        Assert.assertTrue(keyColumns.get(0).getName().equals("c_1_0"));
        Assert.assertTrue(keyColumns.get(1).getName().equals("c_1_3"));
    }

Line commented on: Assert.assertTrue(keyColumns.get(0).getName().equals("c_1_0"));

Revised code:
public void testCreateMaterializedViewWithoutSortKeys_Partitioned_1() {
        
        String sql = "create materialized view test_mv_sort_key1 " +
                "partition by c_1_3 " +
                "distributed by hash(c_1_3, c_1_0) buckets 10 " +
                "PROPERTIES (\n" +
                "\"replication_num\" = \"1\"" +
                ") " +
                "as select c_1_3, c_1_0, c_1_4, c_1_5 from t1";
        List<Column> keyColumns = getMaterializedViewKeysChecked(sql);
        Assert.assertTrue(keyColumns.get(0).getName().equals("c_1_3"));
        Assert.assertTrue(keyColumns.get(1).getName().equals("c_1_0"));
    }
Code review comment: Yes, the APIS should be changed, here are the final discussion result: ``` post: job/start/ job/stop/  get: job/progress/${id} ```

Code under review:
protected void channelRead0(final ChannelHandlerContext channelHandlerContext, final FullHttpRequest request) {
        String requestPath = request.uri();
        String requestBody = request.content().toString(CharsetUtil.UTF_8);
        HttpMethod method = request.method();
        if (!URL_PATTERN.matcher(requestPath).matches()) {
            response("not support request", channelHandlerContext, HttpResponseStatus.BAD_REQUEST, request);
            return;
        }
        if ("/shardingscaling/start".equalsIgnoreCase(requestPath) && method.equals(HttpMethod.POST)) {
            startShardingScalingJob(requestBody);
            response("start", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        if (requestPath.contains("/shardingscaling/progress/") && method.equals(HttpMethod.GET)) {
            
            response("progress", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        if (requestPath.contains("/shardingscaling/stop/") && method.equals(HttpMethod.DELETE)) {
            
            response("stop", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        response("not support request", channelHandlerContext, HttpResponseStatus.BAD_REQUEST, request);
    }

Line commented on: if (requestPath.contains("/shardingscaling/stop/") && method.equals(HttpMethod.DELETE)) {

Revised code:
protected void channelRead0(final ChannelHandlerContext channelHandlerContext, final FullHttpRequest request) {
        String requestPath = request.uri();
        String requestBody = request.content().toString(CharsetUtil.UTF_8);
        HttpMethod method = request.method();
        if (!URL_PATTERN.matcher(requestPath).matches()) {
            response("not support request", channelHandlerContext, HttpResponseStatus.BAD_REQUEST, request);
            return;
        }
        if ("/shardingscaling/job/start".equalsIgnoreCase(requestPath) && method.equals(HttpMethod.POST)) {
            startShardingScalingJob(requestBody);
            response("start", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        if (requestPath.contains("/shardingscaling/job/progress/") && method.equals(HttpMethod.GET)) {
            
            response("progress", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        if ("/shardingscaling/job/stop".equalsIgnoreCase(requestPath) && method.equals(HttpMethod.POST)) {
            
            response("stop", channelHandlerContext, HttpResponseStatus.OK, request);
            return;
        }
        response("not support request", channelHandlerContext, HttpResponseStatus.BAD_REQUEST, request);
    }
Code review comment: as above, moved refcounting into SpannerAccessor.

Code under review:
public void setup() {
      spannerAccessor = spannerAccessors.get(spannerConfig);
      if (spannerAccessor == null) {
        synchronized (spannerAccessors) {
          
          spannerAccessor = spannerAccessors.get(spannerConfig);
          if (spannerAccessor == null) {
            
            LOG.info("Connecting to {}", spannerConfig);
            spannerAccessor = SpannerAccessor.create(spannerConfig);
            spannerAccessors.put(spannerConfig, spannerAccessor);
            refcounts.putIfAbsent(spannerConfig, new AtomicInteger(0));
          }
        }
      }
      
      int refcount = refcounts.get(spannerConfig).incrementAndGet();
      LOG.debug("Setup: Spanner accessor refcount={} for {}", refcount, spannerConfig);

      bundleWriteBackoff =
          FluentBackoff.DEFAULT
              .withMaxCumulativeBackoff(spannerConfig.getMaxCumulativeBackoff().get())
              .withInitialBackoff(spannerConfig.getMaxCumulativeBackoff().get().dividedBy(60));
    }

Line commented on: synchronized (spannerAccessors) {

Revised code:
public void setup() {
      spannerAccessor = SpannerAccessor.getOrCreate(spannerConfig);
      bundleWriteBackoff =
          FluentBackoff.DEFAULT
              .withMaxCumulativeBackoff(spannerConfig.getMaxCumulativeBackoff().get())
              .withInitialBackoff(spannerConfig.getMaxCumulativeBackoff().get().dividedBy(60));
    }
Code review comment: Nice , thanks for introducing fault tolerance.

Code under review:
private void validateDataCreation(int expectedSize) {
        final String containerName = _configuration.getCollectionId();
        final CosmosAsyncDatabase database = _client.getDatabase(_configuration.getDatabaseId());
        final CosmosAsyncContainer container = database.getContainer(containerName);
        LOGGER.info("Validating {} documents were loaded into [{}:{}]",
            expectedSize, _configuration.getDatabaseId(), containerName);

        final List<FeedResponse<ObjectNode>> queryItemsResponseList = container
            .queryItems(COUNT_ALL_QUERY, ObjectNode.class)
            .byPage()
            .collectList()
            .block(BULK_LOAD_WAIT_DURATION);
        final int resultCount = Optional.ofNullable(queryItemsResponseList)
            .map(responseList -> responseList.get(0))
            .map(FeedResponse::getResults)
            .map(list -> list.get(0))
            .map(objectNode -> objectNode.get(COUNT_ALL_QUERY_RESULT_FIELD).intValue())
            .orElse(0);

        if (resultCount < (expectedSize * 0.90)) {
            throw new IllegalStateException(
                String.format("Number of documents %d in the container %s is less than the expected threshold %f ",
                    resultCount, containerName, (expectedSize * 0.90)));
        }
    }

Line commented on: if (resultCount < (expectedSize * 0.90)) {

Revised code:
private void validateDataCreation(int expectedSize) {
        final String containerName = _configuration.getCollectionId();
        final CosmosAsyncDatabase database = _client.getDatabase(_configuration.getDatabaseId());
        final CosmosAsyncContainer container = database.getContainer(containerName);
        LOGGER.info("Validating {} documents were loaded into [{}:{}]",
            expectedSize, _configuration.getDatabaseId(), containerName);

        final List<FeedResponse<ObjectNode>> queryItemsResponseList = container
            .queryItems(COUNT_ALL_QUERY, ObjectNode.class)
            .byPage()
            .collectList()
            .block(BULK_LOAD_WAIT_DURATION);
        final int resultCount = Optional.ofNullable(queryItemsResponseList)
            .map(responseList -> responseList.get(0))
            .map(FeedResponse::getResults)
            .map(list -> list.get(0))
            .map(objectNode -> objectNode.get(COUNT_ALL_QUERY_RESULT_FIELD).intValue())
            .orElse(0);

        if (resultCount < (expectedSize * 0.90)) {
            throw new IllegalStateException(
                String.format("Number of documents %d in the container %s is less than the expected threshold %f ",
                    resultCount, containerName, (expectedSize * 0.90)));
        }
    }
Code review comment: Do we need to new up `DataLakeFileInputStreamOptions` here? If it's null shouldn't `options.isUpn()` be null?  Could we just make the if check below  `if (options != null && options.isUpn() != null) {`

Code under review:
public DataLakeFileOpenInputStreamResult openInputStream(DataLakeFileInputStreamOptions options, Context context) {
        Context newContext;
        options = options == null ? new DataLakeFileInputStreamOptions() : options;
        if (options.isUpn() != null) {
            HttpHeaders headers = new HttpHeaders();
            headers.set("x-ms-upn", options.isUpn() ? "true" : "false");
            if (context == null) {
                newContext = new Context(AddHeadersFromContextPolicy.AZURE_REQUEST_HTTP_HEADERS_KEY, headers);
            } else {
                newContext = context.addData(AddHeadersFromContextPolicy.AZURE_REQUEST_HTTP_HEADERS_KEY, headers);
            }
        } else {
            newContext = null;
        }

        BlobInputStreamOptions convertedOptions = Transforms.toBlobInputStreamOptions(options);
        BlobInputStream inputStream = blockBlobClient.openInputStream(convertedOptions, newContext);
        return new InternalDataLakeFileOpenInputStreamResult(inputStream,
            Transforms.toPathProperties(inputStream.getProperties()));
    }

Line commented on: options = options == null ? new DataLakeFileInputStreamOptions() : options;

Revised code:
public DataLakeFileOpenInputStreamResult openInputStream(DataLakeFileInputStreamOptions options, Context context) {
        context = BuilderHelper.addUpnHeader(() -> (options == null) ? null : options.isUpn(), context);

        BlobInputStreamOptions convertedOptions = Transforms.toBlobInputStreamOptions(options);
        BlobInputStream inputStream = blockBlobClient.openInputStream(convertedOptions, context);
        return new InternalDataLakeFileOpenInputStreamResult(inputStream,
            Transforms.toPathProperties(inputStream.getProperties()));
    }
Code review comment: ```suggestion         service = new ScheduledThreadPoolExecutor(1, r -> new Thread(r, name() + "-worker")); ```

Code under review:
public Maintainer(Controller controller, Duration interval, JobControl jobControl, String name, Set<SystemName> activeSystems) {
        if (interval.isNegative() || interval.isZero())
            throw new IllegalArgumentException("Interval must be positive, but was " + interval);

        this.controller = controller;
        this.maintenanceInterval = interval;
        this.jobControl = jobControl;
        this.name = name;
        this.activeSystems = Set.copyOf(activeSystems);

        service = new ScheduledThreadPoolExecutor(1, r -> new Thread(r, getClass().getSimpleName() + "-worker"));
        long delay = staggeredDelay(controller.curator().cluster(), controller.hostname(), controller.clock().instant(), interval);
        service.scheduleAtFixedRate(this, delay, interval.toMillis(), TimeUnit.MILLISECONDS);
        jobControl.started(name());
    }

Line commented on: service = new ScheduledThreadPoolExecutor(1, r -> new Thread(r, getClass().getSimpleName() + "-worker"));

Revised code:
public Maintainer(Controller controller, Duration interval, JobControl jobControl, String name, Set<SystemName> activeSystems) {
        if (interval.isNegative() || interval.isZero())
            throw new IllegalArgumentException("Interval must be positive, but was " + interval);

        this.controller = controller;
        this.maintenanceInterval = interval;
        this.jobControl = jobControl;
        this.name = name;
        this.activeSystems = Set.copyOf(activeSystems);

        service = new ScheduledThreadPoolExecutor(1, r -> new Thread(r, name() + "-worker"));
        long delay = staggeredDelay(controller.curator().cluster(), controller.hostname(), controller.clock().instant(), interval);
        service.scheduleAtFixedRate(this, delay, interval.toMillis(), TimeUnit.MILLISECONDS);
        jobControl.started(name());
    }
Code review comment: Does it make sense to have this.retryPolicy  set to new Retrypolicy() in the constructor of the class. And in the method `public ChatClientBuilder retryPolicy(RetryPolicy retryPolicy)`, it just reset the retryPolicy attribute This way, we can avoid the null checks in the code.

Code under review:
private void applyRequiredPolicies(List<HttpPipelinePolicy> policies) {
        policies.add(getUserAgentPolicy());
        policies.add(this.retryPolicy == null ? new RetryPolicy() : this.retryPolicy);
        policies.add(new CookiePolicy());
        policies.add(new HttpLoggingPolicy(logOptions));
    }

Line commented on: policies.add(this.retryPolicy == null ? new RetryPolicy() : this.retryPolicy);

Revised code:
private void applyRequiredPolicies(List<HttpPipelinePolicy> policies) {
        policies.add(getUserAgentPolicy());
        policies.add(this.retryPolicy == null ? new RetryPolicy() : this.retryPolicy);
        policies.add(new CookiePolicy());
        policies.add(new HttpLoggingPolicy(logOptions));
    }
Code review comment: I don't think we need this line, since already checked in the `while`.

Code under review:
public String toString() {
        StringBuilder sb = new StringBuilder();
        sb.append(getTypeName());
        sb.append(getSizeString());
        BType element = elementType;
        while (element instanceof BArrayType) {
            sb.append(((BArrayType) element).getSizeString());
            if (!(((BArrayType) element).elementType instanceof BArrayType)) {
                break;
            }
            element = ((BArrayType) element).elementType;
        }
        return sb.toString();
    }

Line commented on: if (!(((BArrayType) element).elementType instanceof BArrayType)) {

Revised code:
public String toString() {
        StringBuilder sb = new StringBuilder();
        BType tempElementType = elementType;
        sb.append(getSizeString());
        while (tempElementType.getTag() == TypeTags.ARRAY_TAG) {
            BArrayType arrayElement = (BArrayType) tempElementType;
            sb.append(arrayElement.getSizeString());
            tempElementType = arrayElement.elementType;
        }
        if (tempElementType.getTag() == TypeTags.UNION_TAG) {
            return sb.insert(0, "(" + tempElementType.toString() + ")").toString();
        }
        return  sb.insert(0, tempElementType.toString()).toString();
    }
Code review comment: @FroMage you could also check that the bean has all the expected types, see [`ResourceBeanTypeTest`](https://github.com/quarkusio/quarkus/blob/main/extensions/resteasy-reactive/rest-client-reactive/deployment/src/test/java/io/quarkus/rest/client/reactive/beanTypes/ResourceBeanTypeTest.java#L39-L45) as an example. But I won't insist :)

Code under review:
void shouldDeployWithoutIssues() {
        
    }


Revised code:
void shouldDeployWithoutIssues() {
        
    }
Code review comment: Currently they do not have a shared parent to put the constant, and it seems not worthwhile to make a new class for constant, as currently here is only 1 "kubernetes" duplicated in 2 places.

Code under review:
private static boolean isWebApp(SiteInner inner) {
        boolean ret = false;
        if (inner.kind() == null) {
            ret = true;
        } else {
            List<String> kinds = Arrays.asList(inner.kind().split(Pattern.quote(",")));
            if ((kinds.contains("app") || kinds.contains("api")) && !kinds.contains("kubernetes")) {
                ret = true;
            }
        }
        return ret;
    }

Line commented on: if ((kinds.contains("app") || kinds.contains("api")) && !kinds.contains("kubernetes")) {

Revised code:
private static boolean isWebApp(SiteInner inner) {
        boolean ret = false;
        if (inner.kind() == null) {
            ret = true;
        } else {
            List<String> kinds = Arrays.asList(inner.kind().split(Pattern.quote(",")));
            if ((kinds.contains("app") || kinds.contains("api")) && !kinds.contains("kubernetes")) {
                ret = true;
            }
        }
        return ret;
    }
Code review comment: What I meant was something like ```java Map<Name, BPackageSymbol> modules = new HashMap<>(); modules.put(Names.ERROR, this.langErrorModuleSymbol); modules.put(Names.OBJECT, this.langObjectModuleSymbol); modules.put(Names.XML, this.langXmlModuleSymbol);  this.predeclaredModules = Collections.unmodifiableMap(modules); ```

Code under review:
public void loadPredeclaredModules() {
        this.predeclaredModules.put(Names.ERROR, this.langErrorModuleSymbol);
        this.predeclaredModules.put(Names.OBJECT, this.langObjectModuleSymbol);
        this.predeclaredModules.put(Names.XML, this.langXmlModuleSymbol);

        this.predeclaredModules = Collections.unmodifiableMap(this.predeclaredModules);
    }

Line commented on: this.predeclaredModules = Collections.unmodifiableMap(this.predeclaredModules);

Revised code:
public void loadPredeclaredModules() {
        Map<Name, BPackageSymbol> modules = new HashMap<>();
        modules.put(Names.ERROR, this.langErrorModuleSymbol);
        modules.put(Names.OBJECT, this.langObjectModuleSymbol);
        modules.put(Names.XML, this.langXmlModuleSymbol);

        this.predeclaredModules = Collections.unmodifiableMap(modules);
    }
Code review comment: New issue https://github.com/ballerina-platform/ballerina-lang/issues/36069

Code under review:
public Boolean visit(BRecordType t, BType s) {
            if (t == s) {
                return true;
            }

            if (s.tag != TypeTags.RECORD || !hasSameReadonlyFlag(s, t)) {
                return false;
            }

            BRecordType source = (BRecordType) s;

            if (source.fields.size() != t.fields.size()) {
                return false;
            }

            for (BField sourceField : source.fields.values()) {
                if (t.fields.containsKey(sourceField.name.value)) {
                    BField targetField = t.fields.get(sourceField.name.value);
                    if (isSameType(sourceField.type, targetField.type, new HashSet<>(this.unresolvedTypes)) &&
                            hasSameOptionalFlag(sourceField.symbol, targetField.symbol) &&
                            (!Symbols.isFlagOn(targetField.symbol.flags, Flags.READONLY) ||
                                     Symbols.isFlagOn(sourceField.symbol.flags, Flags.READONLY))) {
                        continue;
                    }
                }
                return false;
            }
            return isSameType(source.restFieldType, t.restFieldType, new HashSet<>(this.unresolvedTypes));
        }

Line commented on: if (isSameType(sourceField.type, targetField.type, new HashSet<>(this.unresolvedTypes)) &&

Revised code:
public Boolean visit(BRecordType t, BType s) {
            if (t == s) {
                return true;
            }

            if (s.tag != TypeTags.RECORD || !hasSameReadonlyFlag(s, t)) {
                return false;
            }

            BRecordType source = (BRecordType) s;

            if (source.fields.size() != t.fields.size()) {
                return false;
            }

            for (BField sourceField : source.fields.values()) {
                if (t.fields.containsKey(sourceField.name.value)) {
                    BField targetField = t.fields.get(sourceField.name.value);
                    if (isSameType(sourceField.type, targetField.type, new HashSet<>(this.unresolvedTypes)) &&
                            hasSameOptionalFlag(sourceField.symbol, targetField.symbol) &&
                            (!Symbols.isFlagOn(targetField.symbol.flags, Flags.READONLY) ||
                                     Symbols.isFlagOn(sourceField.symbol.flags, Flags.READONLY))) {
                        continue;
                    }
                }
                return false;
            }
            return isSameType(source.restFieldType, t.restFieldType, new HashSet<>(this.unresolvedTypes));
        }
Code review comment: > projects  @tristaZero It can be seen from the source code that when the original projects and the optimized projects are the same, projectInts will return null. Maybe the `0 == projects.length` condition is redundant.  ```java   private static TableScanNode createProjectableFilterable(Compiler compiler,       TableScan rel, ImmutableList<RexNode> filters, ImmutableIntList projects,       ProjectableFilterableTable pfTable) {     final DataContext root = compiler.getDataContext();     final ImmutableIntList originalProjects = projects;     for (;;) {       final List<RexNode> mutableFilters = Lists.newArrayList(filters);       final int[] projectInts;       if (projects == null           || projects.equals(TableScan.identity(rel.getTable()))) {         projectInts = null;       } else {         projectInts = projects.toIntArray();       }       final Enumerable<Object[]> enumerable1 =           pfTable.scan(root, mutableFilters, projectInts); ... } ```

Code under review:
public String generate(final String table) {
        
        String project = null == projects || 0 == projects.length ? "*" : Arrays.stream(projects).mapToObj(each -> fields.get(each).getName()).collect(Collectors.joining(", "));
        return String.format("SELECT %s FROM %s", project, table);
    }

Line commented on: String project = null == projects || 0 == projects.length ? "*" : Arrays.stream(projects).mapToObj(each -> fields.get(each).getName()).collect(Collectors.joining(", "));

Revised code:
public String generate(final String table) {
        
        Collection<String> actualColumnNames = null == projects ? columnNames : Arrays.stream(projects).mapToObj(columnNames::get).collect(Collectors.toList());
        return String.format("SELECT %s FROM %s", Joiner.on(", ").join(actualColumnNames), table);
    }
Code review comment: it may or may not be the CPU history logged in the diagnostics, but since it is close enough, so should be fine?

Code under review:
public String toJson() {
        String snapshot = this.cachedRequestDiagnostics;
        if (snapshot != null) {
            return snapshot;
        }

        synchronized (this.spanName) {
            snapshot = this.cachedRequestDiagnostics;
            if (snapshot != null) {
                return snapshot;
            }

            this.systemUsage = ClientSideRequestStatistics.fetchSystemInformation();
            return this.cachedRequestDiagnostics = getRequestDiagnostics();
        }
    }

Line commented on: this.systemUsage = ClientSideRequestStatistics.fetchSystemInformation();

Revised code:
public String toJson() {
        String snapshot = this.cachedRequestDiagnostics;
        if (snapshot != null) {
            return snapshot;
        }

        synchronized (this.spanName) {
            snapshot = this.cachedRequestDiagnostics;
            if (snapshot != null) {
                return snapshot;
            }

            this.systemUsage = ClientSideRequestStatistics.fetchSystemInformation();
            return this.cachedRequestDiagnostics = getRequestDiagnostics();
        }
    }
Code review comment: It's already exposed ~~for the other classes~~ but yes, it's better not to use it. I'll change it to a private logger.

Code under review:
private void cleanup() throws Exception {
		StreamTask.LOG.debug(
			"Cleanup AsyncCheckpointRunnable for checkpoint {} of {}.",
			checkpointMetaData.getCheckpointId(),
			taskName);

		Exception exception = null;

		
		for (OperatorSnapshotFutures operatorSnapshotResult : operatorSnapshotsInProgress.values()) {
			if (operatorSnapshotResult != null) {
				try {
					operatorSnapshotResult.cancel();
				} catch (Exception cancelException) {
					exception = ExceptionUtils.firstOrSuppressed(cancelException, exception);
				}
			}
		}

		if (null != exception) {
			throw exception;
		}
	}

Line commented on: StreamTask.LOG.debug(

Revised code:
private void cleanup() throws Exception {
		LOG.debug(
			"Cleanup AsyncCheckpointRunnable for checkpoint {} of {}.",
			checkpointMetaData.getCheckpointId(),
			taskName);

		Exception exception = null;

		
		for (OperatorSnapshotFutures operatorSnapshotResult : operatorSnapshotsInProgress.values()) {
			if (operatorSnapshotResult != null) {
				try {
					operatorSnapshotResult.cancel();
				} catch (Exception cancelException) {
					exception = ExceptionUtils.firstOrSuppressed(cancelException, exception);
				}
			}
		}

		if (null != exception) {
			throw exception;
		}
	}
Code review comment: I'd modify these to return a collection instead and make them static.

Code under review:
private void initRegionExecutionViewByVertex(final Set<PipelinedRegion> pipelinedRegions) {
		for (PipelinedRegion pipelinedRegion : pipelinedRegions) {
			final PipelinedRegionExecutionView regionExecutionView = new PipelinedRegionExecutionView(pipelinedRegion);
			for (ExecutionVertexID executionVertexId : pipelinedRegion) {
				regionExecutionViewByVertex.put(executionVertexId, regionExecutionView);
			}
		}
	}

Line commented on: for (ExecutionVertexID executionVertexId : pipelinedRegion) {

Revised code:
private void initRegionExecutionViewByVertex(final Set<PipelinedRegion> pipelinedRegions) {
		for (PipelinedRegion pipelinedRegion : pipelinedRegions) {
			final PipelinedRegionExecutionView regionExecutionView = new PipelinedRegionExecutionView(pipelinedRegion);
			for (ExecutionVertexID executionVertexId : pipelinedRegion) {
				regionExecutionViewByVertex.put(executionVertexId, regionExecutionView);
			}
		}
	}
Code review comment: Actually `srcRel.getTable().getQualifiedName()` returns a `List`, in Calcite there's a concept of `database` which we don't touch so far.

Code under review:
private BeamSqlSeekableTable getSeekableTableFromRelNode(BeamRelNode relNode, BeamSqlEnv sqlEnv) {
    BeamIOSourceRel srcRel = (BeamIOSourceRel) relNode;
    String tableName = Joiner.on('.').join(srcRel.getTable().getQualifiedName());
    BeamSqlTable sourceTable = sqlEnv.findTable(tableName);
    return (BeamSqlSeekableTable) sourceTable;
  }

Line commented on: String tableName = Joiner.on('.').join(srcRel.getTable().getQualifiedName());

Revised code:
private BeamSqlSeekableTable getSeekableTableFromRelNode(BeamRelNode relNode, BeamSqlEnv sqlEnv) {
    BeamIOSourceRel srcRel = (BeamIOSourceRel) relNode;
    String tableName = Joiner.on('.').join(srcRel.getTable().getQualifiedName());
    BeamSqlTable sourceTable = sqlEnv.findTable(tableName);
    return (BeamSqlSeekableTable) sourceTable;
  }
Code review comment: This isn't quite right - we're now invalidating all of the StreamWriters when any one of them fails. I think instead you want to just null out the one that failed and allow it to be recreated the next get.

Code under review:
void invalidateWriteStream() {
        if (streamAppendClient != null) {
          synchronized (APPEND_CLIENTS) {
            
            runAsyncIgnoreFailure(closeWriterExecutor, streamAppendClient::unpin);
            
            
            
            
            
            
            
            @Nullable
            List<StreamAppendClient> cachedAppendClients = APPEND_CLIENTS.getIfPresent(streamName);
            if (cachedAppendClients != null
                && System.identityHashCode(cachedAppendClients.get(clientNumber))
                    == System.identityHashCode(streamAppendClient)) {
              APPEND_CLIENTS.invalidate(streamName);
            }
          }
          streamAppendClient = null;
        }
      }

Line commented on: == System.identityHashCode(streamAppendClient)) {

Revised code:
void invalidateWriteStream() {
        if (streamAppendClient != null) {
          synchronized (APPEND_CLIENTS) {
            
            runAsyncIgnoreFailure(closeWriterExecutor, streamAppendClient::unpin);
            
            
            
            
            
            
            
            String cacheEntryKey = getStreamAppendClientCacheEntryKey();
            @Nullable
            StreamAppendClient cachedAppendClient = APPEND_CLIENTS.getIfPresent(cacheEntryKey);
            if (cachedAppendClient != null
                && System.identityHashCode(cachedAppendClient)
                    == System.identityHashCode(streamAppendClient)) {
              APPEND_CLIENTS.invalidate(cacheEntryKey);
            }
          }
          streamAppendClient = null;
        }
      }
Code review comment: This change is caused by code formatting, and have nothing to do with the business logic of this PR. I have rolled it back to make this PR more focused.

Code under review:
public String getErrorRespWhenUnauthorized(AccessDeniedException accessDeniedException) {
        if (Strings.isNullOrEmpty(accessDeniedException.getMessage())) {
            ConnectContext context = ConnectContext.get();
            if (context != null) {
                AuthorizationMgr authorizationMgr = GlobalStateMgr.getCurrentState().getAuthorizationMgr();
                UserIdentity userIdentity = context.getCurrentUserIdentity();
                List<String> activatedRoles = authorizationMgr.getRoleNamesByRoleIds(context.getCurrentRoleIds());
                List<String> inactivatedRoles =
                        authorizationMgr.getInactivatedRoleNamesByUser(userIdentity, activatedRoles);
                return "Access denied for user " + userIdentity + ". " +
                        String.format(ErrorCode.ERR_ACCESS_DENIED_HINT_MSG_FORMAT, activatedRoles, inactivatedRoles);
            }
            return "Access denied.";
        } else {
            return accessDeniedException.getMessage();
        }
    }

Line commented on: return "Access denied for user " + userIdentity + ". " +

Revised code:
public String getErrorRespWhenUnauthorized(AccessDeniedException accessDeniedException) {
        if (Strings.isNullOrEmpty(accessDeniedException.getMessage())) {
            ConnectContext context = ConnectContext.get();
            if (context != null) {
                AuthorizationMgr authorizationMgr = GlobalStateMgr.getCurrentState().getAuthorizationMgr();
                UserIdentity userIdentity = context.getCurrentUserIdentity();
                List<String> activatedRoles = authorizationMgr.getRoleNamesByRoleIds(context.getCurrentRoleIds());
                List<String> inactivatedRoles =
                        authorizationMgr.getInactivatedRoleNamesByUser(userIdentity, activatedRoles);
                return "Access denied for user " + userIdentity  + ". " +
                        String.format(ErrorCode.ERR_ACCESS_DENIED_HINT_MSG_FORMAT, activatedRoles, inactivatedRoles);
            }
            return "Access denied.";
        } else {
            return accessDeniedException.getMessage();
        }
    }
Code review comment: the path could still refer to an object instead of a directory

Code under review:
private void deleteRecursively(Path path) throws IOException {
        final FileStatus[] containingFiles =
                Preconditions.checkNotNull(
                        listStatus(path),
                        "Hadoop FileSystem.listStatus should never return null based on its contract.");

        if (containingFiles.length == 0) {
            deleteObject(path);
            return;
        }

        IOException exception = null;
        for (FileStatus fileStatus : containingFiles) {
            final Path childPath = fileStatus.getPath();

            try {
                if (fileStatus.isDir()) {
                    deleteRecursively(childPath);
                } else {
                    deleteObject(childPath);
                }
            } catch (IOException e) {
                exception = ExceptionUtils.firstOrSuppressed(e, exception);
            }
        }

        
        
        
        
        if (exception != null) {
            throw exception;
        }
    }

Line commented on: IOException exception = null;

Revised code:
private void deleteRecursively(Path path) throws IOException {
        final FileStatus[] containingFiles =
                Preconditions.checkNotNull(
                        listStatus(path),
                        "Hadoop FileSystem.listStatus should never return null based on its contract.");

        if (containingFiles.length == 0) {
            
            
            deleteObject(path);
            return;
        }

        IOException exception = null;
        for (FileStatus fileStatus : containingFiles) {
            final Path childPath = fileStatus.getPath();

            try {
                if (fileStatus.isDir()) {
                    deleteRecursively(childPath);
                } else {
                    deleteObject(childPath);
                }
            } catch (IOException e) {
                exception = ExceptionUtils.firstOrSuppressed(e, exception);
            }
        }

        
        
        
        
        if (exception != null) {
            throw exception;
        }
    }
Code review comment: ```suggestion                  private static final Map<Byte, TransactionState> BYTE_TO_STATE =                 Arrays.stream(TransactionState.values())                         .collect(Collectors.toMap(e -> e.state, e -> e));          static TransactionState fromByte(byte state) {             TransactionState transactionState = BYTE_TO_STATE.get(state);             if (transactionState == null) {                 throw new IllegalArgumentException(                         String.format("The given state %s is not supported.", state));             }             return transactionState;         } ```

Code under review:
private static byte readTransactionState(ByteBuffer buffer) {
        
        buffer.getLong();
        
        buffer.getShort();
        
        buffer.getInt();
        
        return buffer.get();
    }


Revised code:
private static byte readTransactionState(ByteBuffer buffer) {
        
        buffer.getLong();
        
        buffer.getShort();
        
        buffer.getInt();
        
        return buffer.get();
    }
Code review comment: Yes, you are right. I forgot that the code is waiting on the latch there.

Code under review:
public void taskCachedThreadPoolAllowsForSynchronousCheckpoints() throws Exception {
		final Task task = createTask(SynchronousCheckpointTestingTask.class);

		try (TaskCleaner ignored = new TaskCleaner(task)) {
			task.startTaskThread();

			executionLatch.await();

			assertEquals(ExecutionState.RUNNING, task.getExecutionState());
			assertEquals(CheckpointingState.NONE, synchronousCheckpointPhase.getState());

			
			
			
			
			
			
			task.triggerCheckpointBarrier(
					42,
					156865867234L,
					new CheckpointOptions(CheckpointType.SYNC_SAVEPOINT, CheckpointStorageLocationReference.getDefault()),
					true);
			checkpointLatch.await();

			assertNull(error.get());
			assertEquals(CheckpointingState.PERFORMING_CHECKPOINT, synchronousCheckpointPhase.getState());

			checkpointTriggered.await();

			task.notifyCheckpointComplete(42);

			notifyLatch.await();
			assertNull(error.get());
			assertEquals(CheckpointingState.EXECUTED_CALLBACK, synchronousCheckpointPhase.getState());

			checkpointCompletionLatch.trigger();
			checkpointLatch.await();

			assertNull(error.get());
			assertEquals(CheckpointingState.FINISHED_CHECKPOINT, synchronousCheckpointPhase.getState());
			assertEquals(ExecutionState.RUNNING, task.getExecutionState());
		}
	}

Line commented on: task.startTaskThread();

Revised code:
public void taskCachedThreadPoolAllowsForSynchronousCheckpoints() throws Exception {
		final Task task = createTask(SynchronousCheckpointTestingTask.class);

		try (TaskCleaner ignored = new TaskCleaner(task)) {
			task.startTaskThread();

			assertThat(eventQueue.take(), is(Event.TASK_IS_RUNNING));
			assertTrue(eventQueue.isEmpty());

			assertEquals(ExecutionState.RUNNING, task.getExecutionState());

			
			
			
			
			
			
			task.triggerCheckpointBarrier(
					42,
					156865867234L,
					new CheckpointOptions(CheckpointType.SYNC_SAVEPOINT, CheckpointStorageLocationReference.getDefault()),
					true);

			assertThat(eventQueue.take(), is(Event.PRE_TRIGGER_CHECKPOINT));
			assertTrue(eventQueue.isEmpty());

			checkpointTriggered.await();

			task.notifyCheckpointComplete(42);

			assertThat(eventQueue.take(), is(Event.PRE_NOTIFY_CHECKPOINT_COMPLETE));
			assertThat(
				Arrays.asList(eventQueue.take(), eventQueue.take()),
				containsInAnyOrder(Event.POST_NOTIFY_CHECKPOINT_COMPLETE, Event.POST_TRIGGER_CHECKPOINT));
			assertTrue(eventQueue.isEmpty());

			assertEquals(ExecutionState.RUNNING, task.getExecutionState());
		}
	}
Code review comment: It seems this is the only use for the `jpaConfig` object. Does this stricly need some lazy initialization, or could you read the field earlier and store only an immutable boolean field rather than keeping a reference to the bootstrap proxy?

Code under review:
public boolean validateExistingCurrentSessions() {
        return jpaConfig.isValidateTenantInCurrentSessions();
    }

Line commented on: return jpaConfig.isValidateTenantInCurrentSessions();

Revised code:
public boolean validateExistingCurrentSessions() {
        return false;
    }
Code review comment: The retryRate is not used in this method, probably can move this logic to reevaluateThresholds

Code under review:
private void recordOperation(boolean isRetry) {
        long totalSnapshot = this.totalOperationCount.incrementAndGet();
        CurrentIntervalThresholds currentThresholdsSnapshot = this.currentThresholds.get();
        long currentTotalCountSnapshot = currentThresholdsSnapshot.currentOperationCount.incrementAndGet();
        long currentRetryCountSnapshot;
        if (isRetry) {
            currentRetryCountSnapshot = currentThresholdsSnapshot.currentRetriedOperationCount.incrementAndGet();
        } else {
            currentRetryCountSnapshot = currentThresholdsSnapshot.currentRetriedOperationCount.get();
        }

        double retryRate = (double)currentRetryCountSnapshot / currentTotalCountSnapshot;
        Pair<Boolean, Boolean> shouldReevaluateResult =
            this.shouldReevaluateThresholds(totalSnapshot, currentTotalCountSnapshot);
        boolean shouldReevaluate = shouldReevaluateResult.getLeft();
        if (shouldReevaluate) {
            boolean onlyUpscale = shouldReevaluateResult.getRight();
            if (onlyUpscale ||
                this.currentThresholds.compareAndSet(currentThresholdsSnapshot, new CurrentIntervalThresholds())) {

                this.reevaluateThresholds(
                    totalSnapshot,
                    currentTotalCountSnapshot,
                    currentRetryCountSnapshot,
                    retryRate,
                    shouldReevaluateResult.getRight());
            }
        }
    }

Line commented on: double retryRate = (double)currentRetryCountSnapshot / currentTotalCountSnapshot;

Revised code:
private void recordOperation(boolean isRetry) {
        long totalSnapshot = this.totalOperationCount.incrementAndGet();
        CurrentIntervalThresholds currentThresholdsSnapshot = this.currentThresholds.get();
        long currentTotalCountSnapshot = currentThresholdsSnapshot.currentOperationCount.incrementAndGet();
        long currentRetryCountSnapshot;
        if (isRetry) {
            currentRetryCountSnapshot = currentThresholdsSnapshot.currentRetriedOperationCount.incrementAndGet();
        } else {
            currentRetryCountSnapshot = currentThresholdsSnapshot.currentRetriedOperationCount.get();
        }

        Pair<Boolean, Boolean> shouldReevaluateResult =
            this.shouldReevaluateThresholds(totalSnapshot, currentTotalCountSnapshot);
        boolean shouldReevaluate = shouldReevaluateResult.getLeft();
        if (shouldReevaluate) {
            boolean onlyUpscale = shouldReevaluateResult.getRight();
            if (onlyUpscale ||
                this.currentThresholds.compareAndSet(currentThresholdsSnapshot, new CurrentIntervalThresholds())) {

                this.reevaluateThresholds(
                    totalSnapshot,
                    currentTotalCountSnapshot,
                    currentRetryCountSnapshot,
                    shouldReevaluateResult.getRight());
            }
        }
    }