muttakin2301/ccr2000 huggingface dataset:
Code review comment: Remove a tab from all these lines for uniformity.

Code under review:
public void testStopOnlyWithMaxWM() throws Exception {
		JobID jid = new JobID();

		String[] parameters = { "-d", jid.toString() };
		OneShotLatch stopWithSavepointLatch = new OneShotLatch();
		TestingClusterClient<String> clusterClient = new TestingClusterClient<>();
		clusterClient.setStopWithSavepointFunction((jobID, advanceToEndOfEventTime, savepointDirectory) -> {
				assertThat(jobID, is(jid));
				assertThat(advanceToEndOfEventTime, is(true));
				assertNull(savepointDirectory);
				stopWithSavepointLatch.trigger();
				return CompletableFuture.completedFuture(savepointDirectory);
		});
		MockedCliFrontend testFrontend = new MockedCliFrontend(clusterClient);
		testFrontend.stop(parameters);

		stopWithSavepointLatch.await();
	}

Line commented on: assertThat(jobID, is(jid));

Revised code:
public void testStopOnlyWithMaxWM() throws Exception {
		JobID jid = new JobID();

		String[] parameters = { "-d", jid.toString() };
		OneShotLatch stopWithSavepointLatch = new OneShotLatch();
		TestingClusterClient<String> clusterClient = new TestingClusterClient<>();
		clusterClient.setStopWithSavepointFunction((jobID, advanceToEndOfEventTime, savepointDirectory) -> {
			assertThat(jobID, is(jid));
			assertThat(advanceToEndOfEventTime, is(true));
			assertNull(savepointDirectory);
			stopWithSavepointLatch.trigger();
			return CompletableFuture.completedFuture(savepointDirectory);
		});
		MockedCliFrontend testFrontend = new MockedCliFrontend(clusterClient);
		testFrontend.stop(parameters);

		stopWithSavepointLatch.await();
	}
Code review comment: I mean to add `catch InterruptedException` section before `catch Throwable` in `LegacySourceFunctionThread` (with `Thread.currentThread().interrupt();` in it).

Code under review:
protected void cancelTask() {
		try {
			if (headOperator != null) {
				headOperator.cancel();
			}
		}
		finally {
			sourceThread.interrupt();
		}
	}

Line commented on: sourceThread.interrupt();

Revised code:
protected void cancelTask() {
		try {
			if (headOperator != null) {
				headOperator.cancel();
			}
		}
		finally {
			sourceThread.interrupt();
		}
	}
Code review comment: nevermind, I had to revert it, FindBugs reported it:  ```EQ_GETCLASS_AND_CLASS_CONSTANT: equals method fails for subtypes This class has an equals method that will be broken if it is inherited by subclasses. It compares a class literal with the class of the argument (e.g., in class Foo it might check if Foo.class == o.getClass()). It is better to check if this.getClass() == o.getClass().```   So I changed it from the fields to the getter methods.

Code under review:
public boolean equals(Object o) {
      if (this == o) {
        return true;
      }
      if (o == null || getClass() != o.getClass()) {
        return false;
      }

      GameActionInfo gameActionInfo = (GameActionInfo) o;

      if (!user.equals(gameActionInfo.getUser())) {
        return false;
      }

      if (!team.equals(gameActionInfo.getTeam())) {
        return false;
      }

      if (!score.equals(gameActionInfo.getScore())) {
        return false;
      }

      return timestamp.equals(gameActionInfo.getTimestamp());
    }

Line commented on: if (o == null || getClass() != o.getClass()) {

Revised code:
public boolean equals(Object o) {
      if (this == o) {
        return true;
      }
      if (o == null || o.getClass() != this.getClass()) {
        return false;
      }

      GameActionInfo gameActionInfo = (GameActionInfo) o;

      if (!this.getUser().equals(gameActionInfo.getUser())) {
        return false;
      }

      if (!this.getTeam().equals(gameActionInfo.getTeam())) {
        return false;
      }

      if (!this.getScore().equals(gameActionInfo.getScore())) {
        return false;
      }

      return this.getTimestamp().equals(gameActionInfo.getTimestamp());
    }
Code review comment: I did not think this is expensive though ? And it's only been called twice ?

Code under review:
private boolean shouldScanAnnotations(Capabilities capabilities) {
        
        Config config = ConfigProvider.getConfig();
        boolean scanDisable = config.getOptionalValue(OASConfig.SCAN_DISABLE, Boolean.class).orElse(false);
        if (scanDisable) {
            return false;
        }

        
        boolean isJaxrs = capabilities.isCapabilityPresent(Capabilities.RESTEASY);
        boolean isSpring = capabilities.isCapabilityPresent(Capabilities.SPRING_WEB);
        return isJaxrs || isSpring;
    }

Line commented on: Config config = ConfigProvider.getConfig();

Revised code:
private boolean shouldScanAnnotations(Capabilities capabilities) {
        
        Config config = ConfigProvider.getConfig();
        boolean scanDisable = config.getOptionalValue(OASConfig.SCAN_DISABLE, Boolean.class).orElse(false);
        if (scanDisable) {
            return false;
        }

        
        boolean isJaxrs = capabilities.isCapabilityPresent(Capabilities.RESTEASY);
        boolean isSpring = capabilities.isCapabilityPresent(Capabilities.SPRING_WEB);
        return isJaxrs || isSpring;
    }
Code review comment: Discussed offline, agreed will leave this change out of this PR.   Meanwhile, @mbhaskar created this working item to track it : https://github.com/Azure/azure-sdk-for-java/issues/20271  Thanks

Code under review:
public boolean expire() {
        final CosmosException error;
        if (this.args.serviceRequest().isReadOnly()) {
            
            error = new GoneException(this.toString(), null, this.args.physicalAddress());
        } else if (!this.hasSendingRequestStarted()) {
            
            error = new GoneException(this.toString(), null, this.args.physicalAddress());
        } else {
            
            error = new RequestTimeoutException(this.toString(), this.args.physicalAddress());
        }

        BridgeInternal.setRequestHeaders(error, this.args.serviceRequest().getHeaders());
        return this.completeExceptionally(error);
    }

Line commented on: error = new GoneException(this.toString(), null, this.args.physicalAddress());

Revised code:
public boolean expire() {
        final CosmosException error;
        if (this.args.serviceRequest().isReadOnly() || !this.hasSendingRequestStarted()) {
            
            
            
            error = new GoneException(this.toString(), null, this.args.physicalAddress());
        } else {
            
            error = new RequestTimeoutException(this.toString(), this.args.physicalAddress());
        }

        BridgeInternal.setRequestHeaders(error, this.args.serviceRequest().getHeaders());
        return this.completeExceptionally(error);
    }
Code review comment: this description is used only for the purpose of logging if the connection is lost (see [ZooKeeperMultipleComponentLeaderElectionDriver:175](https://github.com/apache/flink/blob/23d942cb6bba947ca3844687a65e8d0451c62041/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/ZooKeeperMultipleComponentLeaderElectionDriver.java#L175). I couldn't find any code location where the `ZooKeeperMultipleComponentLeaderElectionDriver.toString` method is utilized. Either, we should add some random string here to differentiate different Dispatcher instances or we could remove it entirely (the k8s implementation doesn't make use of it at all but swallows it in `KubernetesMultipleComponentLeaderElectionDriverFactory.create`

Code under review:
private MultipleComponentLeaderElectionService getOrInitializeSingleLeaderElectionService() {
        if (multipleComponentLeaderElectionService == null) {
            try {
                multipleComponentLeaderElectionService =
                        new DefaultMultipleComponentLeaderElectionService(
                                fatalErrorHandler,
                                "Single leader election service.",
                                new ZooKeeperMultipleComponentLeaderElectionDriverFactory(
                                        leaderNamespacedCuratorFramework));
            } catch (Exception e) {
                throw new FlinkRuntimeException(
                        String.format(
                                "Could not initialize the %s",
                                DefaultMultipleComponentLeaderElectionService.class
                                        .getSimpleName()),
                        e);
            }
        }

        return multipleComponentLeaderElectionService;
    }

Line commented on: "Single leader election service.",

Revised code:
private MultipleComponentLeaderElectionService getOrInitializeSingleLeaderElectionService() {
        synchronized (lock) {
            if (multipleComponentLeaderElectionService == null) {
                try {
                    multipleComponentLeaderElectionService =
                            new DefaultMultipleComponentLeaderElectionService(
                                    fatalErrorHandler,
                                    new ZooKeeperMultipleComponentLeaderElectionDriverFactory(
                                            leaderNamespacedCuratorFramework));
                } catch (Exception e) {
                    throw new FlinkRuntimeException(
                            String.format(
                                    "Could not initialize the %s",
                                    DefaultMultipleComponentLeaderElectionService.class
                                            .getSimpleName()),
                            e);
                }
            }

            return multipleComponentLeaderElectionService;
        }
    }
Code review comment: could you please explain the scenario which this bug shows itself?  why do we need this?

Code under review:
void encode(final ByteBuf out) {

        final int expectedLength = RntbdRequestFrame.LENGTH + this.headers.computeLength();
        final int start = out.writerIndex();

        out.writeIntLE(expectedLength);
        this.frame.encode(out);
        this.headers.encode(out);

        checkState(out.writerIndex() - start == expectedLength,
            "encoding error: {\"expectedLength\": %s, \"observedLength\": %s}",
            expectedLength,
            out.writerIndex() - expectedLength);

        if (this.payload.length > 0) {
            out.writeIntLE(this.payload.length);
            out.writeBytes(this.payload);
        }
    }

Line commented on: final int start = out.writerIndex();

Revised code:
void encode(final ByteBuf out) {

        final int expectedLength = RntbdRequestFrame.LENGTH + this.headers.computeLength();
        final int start = out.writerIndex();

        out.writeIntLE(expectedLength);
        this.frame.encode(out);
        this.headers.encode(out);

        final int observedLength = out.writerIndex() - start;

        checkState(observedLength == expectedLength,
            "encoding error: {\"expectedLength\": %s, \"observedLength\": %s}",
            expectedLength,
            observedLength);

        if (this.payload.length > 0) {
            out.writeIntLE(this.payload.length);
            out.writeBytes(this.payload);
        }
    }
Code review comment: ```suggestion             assertThat(handler.requestedBuffers).isEmpty(); ``` There is `isEmpty()` method from assertJ

Code under review:
void testBuffersNotRequestedForEmptyStream() throws IOException, InterruptedException {
        ChannelStateSerializer serializer = new ChannelStateSerializerImpl();
        TestRecoveredChannelStateHandler handler = new TestRecoveredChannelStateHandler();

        try (FSDataInputStream stream = getStream(serializer, 0)) {
            new ChannelStateChunkReader(serializer)
                    .readChunk(stream, serializer.getHeaderLength(), handler, "channelInfo", 0);
        } finally {
            assertThat(handler.requestedBuffers.isEmpty()).isTrue();
        }
    }

Line commented on: assertThat(handler.requestedBuffers.isEmpty()).isTrue();

Revised code:
void testBuffersNotRequestedForEmptyStream() throws IOException, InterruptedException {
        ChannelStateSerializer serializer = new ChannelStateSerializerImpl();
        TestRecoveredChannelStateHandler handler = new TestRecoveredChannelStateHandler();

        try (FSDataInputStream stream = getStream(serializer, 0)) {
            new ChannelStateChunkReader(serializer)
                    .readChunk(stream, serializer.getHeaderLength(), handler, "channelInfo", 0);
        } finally {
            assertThat(handler.requestedBuffers).isEmpty();
        }
    }
Code review comment: I have added a new test case for that. The compressed size is actually larger for this test string.

Code under review:
void output_stream_compresses_input() throws IOException {
        byte[] inputData = "The quick brown fox jumps over the lazy dog".getBytes();
        ByteArrayOutputStream arrayOut = new ByteArrayOutputStream();
        try (ZstdOuputStream zstdOut = new ZstdOuputStream(arrayOut, 12)) {
            zstdOut.write(inputData[0]);
            zstdOut.write(inputData, 1, inputData.length - 1);
        }
        byte[] compressedData = arrayOut.toByteArray();
        ZstdCompressor compressor = new ZstdCompressor();
        byte[] decompressedData = new byte[inputData.length];
        compressor.decompress(compressedData, 0, compressedData.length, decompressedData, 0, decompressedData.length);
        assertArrayEquals(inputData, decompressedData);
    }

Line commented on: ZstdCompressor compressor = new ZstdCompressor();

Revised code:
void output_stream_compresses_input() throws IOException {
        byte[] inputData = "The quick brown fox jumps over the lazy dog".getBytes();
        ByteArrayOutputStream arrayOut = new ByteArrayOutputStream();
        try (ZstdOuputStream zstdOut = new ZstdOuputStream(arrayOut, 12)) {
            zstdOut.write(inputData[0]);
            zstdOut.write(inputData, 1, inputData.length - 1);
        }
        byte[] compressedData = arrayOut.toByteArray();
        ZstdCompressor compressor = new ZstdCompressor();
        byte[] decompressedData = new byte[inputData.length];
        compressor.decompress(compressedData, 0, compressedData.length, decompressedData, 0, decompressedData.length);
        assertArrayEquals(inputData, decompressedData);
    }
Code review comment: If it's already set to want to retire it's fine that we do it. However, as you point out it is not fine to undo it if somebody else sets it and we failed (regardless of when it happens).  PTAL - I'm going to run out of "try" clauses soon.

Code under review:
private boolean markWantToRetire(Node node, boolean wantToRetire) {
        try (Mutex lock = nodeRepository().lock(node)) {
            Optional<Node> nodeToMove = nodeRepository().getNode(node.hostname());
            if (nodeToMove.isEmpty()) return false;
            if (nodeToMove.get().state() != Node.State.active) return false;

            if (node.status().wantToRetire() != wantToRetire)
                nodeRepository().write(nodeToMove.get().withWantToRetire(wantToRetire, Agent.system, clock.instant()), lock);
            return true;
        }
    }

Line commented on: if (node.status().wantToRetire() != wantToRetire)

Revised code:
private boolean markWantToRetire(Node node, boolean wantToRetire) {
        try (Mutex lock = nodeRepository().lock(node)) {
            Optional<Node> nodeToMove = nodeRepository().getNode(node.hostname());
            if (nodeToMove.isEmpty()) return false;
            if (nodeToMove.get().state() != Node.State.active) return false;

            if (node.status().wantToRetire() == wantToRetire) return false;

            nodeRepository().write(nodeToMove.get().withWantToRetire(wantToRetire, Agent.system, clock.instant()), lock);
            return true;
        }
    }
Code review comment: This does not actually match the behavior of BigQuery. To match BigQuery you should remove the `HashSet` and instead replace anything starting with `_field_` with the correct index for that column.  (In BigQuery, this condition would be `"".equals(name) || name.startsWith("_field_")`)

Code under review:
private static List<String> getFieldNameList(List<StructField> fields) {
    ImmutableList.Builder<String> b = ImmutableList.builder();
    Set<String> usedName = new HashSet<>();
    for (int i = 0; i < fields.size(); i++) {
      String name = fields.get(i).getName();
      
      if ("".equals(name) || usedName.contains(name)) {
        name = "_field_" + (i + 1); 
        if (usedName.contains(name)) {
          throw new UnsupportedOperationException(
              name + " is not a valid field name. Please use another name for the field.");
        }
      }
      b.add(name);
      usedName.add(name);
    }
    return b.build();
  }

Line commented on: if ("".equals(name) || usedName.contains(name)) {

Revised code:
private static List<String> getFieldNameList(List<StructField> fields) {
    ImmutableList.Builder<String> b = ImmutableList.builder();
    Set<String> usedName = new HashSet<>();
    for (int i = 0; i < fields.size(); i++) {
      String name = fields.get(i).getName();
      
      if ("".equals(name) || name.startsWith("_field_") || usedName.contains(name)) {
        name = "_field_" + (i + 1); 
      }
      b.add(name);
      usedName.add(name);
    }
    return b.build();
  }
Code review comment: I know `KinesisProducerConfiguration` is not serializable that is what I said above, but I was wondering if having a transient field in `Write` set in expand could make sense as expand will always be called after a serialisation of `Write` transform

Code under review:
public PDone expand(PCollection<byte[]> input) {
      checkArgument(getStreamName() != null, "withStreamName() is required");
      checkArgument(
          (getPartitionKey() != null) || (getPartitioner() != null),
          "withPartitionKey() or withPartitioner() is required");
      checkArgument(
          getPartitionKey() == null || (getPartitioner() == null),
          "only one of either withPartitionKey() or withPartitioner() is possible");
      checkArgument(getAWSClientsProvider() != null, "withAWSClientsProvider() is required");
      producerConfiguration(); 

      input.apply(ParDo.of(new KinesisWriterFn(this)));
      return PDone.in(input.getPipeline());
    }

Line commented on: producerConfiguration();

Revised code:
public PDone expand(PCollection<byte[]> input) {
      checkArgument(getStreamName() != null, "withStreamName() is required");
      checkArgument(
          (getPartitionKey() != null) || (getPartitioner() != null),
          "withPartitionKey() or withPartitioner() is required");
      checkArgument(
          getPartitionKey() == null || (getPartitioner() == null),
          "only one of either withPartitionKey() or withPartitioner() is possible");
      checkArgument(getAWSClientsProvider() != null, "withAWSClientsProvider() is required");
      createProducerConfiguration(); 

      input.apply(ParDo.of(new KinesisWriterFn(this)));
      return PDone.in(input.getPipeline());
    }
Code review comment: Yeah, I recommend you to do as Bharat suggests. Let the library do the heavy lifting. There is URL library where you can create the URL from a String and call `getHost()` to get the host name.

Code under review:
private String getHostNameFromEndpoint() {
        return endpoint.replace("https:
    }

Line commented on: return endpoint.replace("https:

Revised code:
private String getHostNameFromEndpoint() throws MalformedURLException {
        return new URL(endpoint).getHost();
    }
Code review comment: Was this also recommended by ErrorProne?

Code under review:
private void nextBatch() {
    curBatch = Collections.emptyIterator();

    ConsumerRecords<byte[], byte[]> records;
    try {
      
      records = availableRecordsQueue.poll(RECORDS_DEQUEUE_POLL_TIMEOUT.getMillis(),
                                           TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      LOG.warn("{}: Unexpected", this, e);
      return;
    }

    if (records == null) {
      return;
    }

    partitionStates.forEach(p -> p.recordIter = records.records(p.topicPartition).iterator());

    
    curBatch = Iterators.cycle(new ArrayList<>(partitionStates));
  }

Line commented on: curBatch = Iterators.cycle(new ArrayList<>(partitionStates));

Revised code:
private void nextBatch() {
    curBatch = Collections.emptyIterator();

    ConsumerRecords<byte[], byte[]> records;
    try {
      
      records = availableRecordsQueue.poll(RECORDS_DEQUEUE_POLL_TIMEOUT.getMillis(),
                                           TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      LOG.warn("{}: Unexpected", this, e);
      return;
    }

    if (records == null) {
      return;
    }

    partitionStates.forEach(p -> p.recordIter = records.records(p.topicPartition).iterator());

    
    curBatch = Iterators.cycle(new ArrayList<>(partitionStates));
  }
Code review comment: +1, `T?` is 80% of the case, when considering unions.

Code under review:
private void reportIncompatibleMappingConstructorError(BLangRecordLiteral mappingConstructorExpr, BType expType) {
        if (expType.tag != TypeTags.UNION) {
            dlog.error(mappingConstructorExpr.pos, DiagnosticCode.MAPPING_CONSTRUCTOR_COMPATIBLE_TYPE_NOT_FOUND,
                       expType);
            return;
        }

        BUnionType unionType = (BUnionType) expType;
        Set<BType> memberTypes = unionType.getMemberTypes();

        for (BType bType : memberTypes) {
            if (isMappingConstructorCompatibleType(bType)) {
                dlog.error(mappingConstructorExpr.pos, DiagnosticCode.INCOMPATIBLE_MAPPING_CONSTRUCTOR, unionType);
                return;
            }
        }

        dlog.error(mappingConstructorExpr.pos, DiagnosticCode.MAPPING_CONSTRUCTOR_COMPATIBLE_TYPE_NOT_FOUND, unionType);
    }

Line commented on: return;

Revised code:
private void reportIncompatibleMappingConstructorError(BLangRecordLiteral mappingConstructorExpr, BType expType) {
        if (expType.tag != TypeTags.UNION) {
            dlog.error(mappingConstructorExpr.pos, DiagnosticCode.MAPPING_CONSTRUCTOR_COMPATIBLE_TYPE_NOT_FOUND,
                       expType);
            return;
        }

        BUnionType unionType = (BUnionType) expType;
        BType[] memberTypes = unionType.getMemberTypes().toArray(new BType[0]);

        
        
        if (memberTypes.length == 2) {
            if (memberTypes[0].tag == TypeTags.RECORD && memberTypes[1].tag == TypeTags.NIL) {
                reportMissingRecordFieldDiagnostics(mappingConstructorExpr.keyValuePairs, (BRecordType) memberTypes[0]);
                return;
            } else if (memberTypes[1].tag == TypeTags.RECORD && memberTypes[0].tag == TypeTags.NIL) {
                reportMissingRecordFieldDiagnostics(mappingConstructorExpr.keyValuePairs, (BRecordType) memberTypes[1]);
                return;
            }
        }

        
        
        
        for (BType bType : memberTypes) {
            if (isMappingConstructorCompatibleType(bType)) {
                dlog.error(mappingConstructorExpr.pos, DiagnosticCode.INCOMPATIBLE_MAPPING_CONSTRUCTOR, unionType);
                return;
            }
        }

        dlog.error(mappingConstructorExpr.pos, DiagnosticCode.MAPPING_CONSTRUCTOR_COMPATIBLE_TYPE_NOT_FOUND, unionType);
    }
Code review comment: True. I think I was feeling funny when I wrote this.

Code under review:
public void testInvocationCleanup() {
    final JobApi.GetJobsRequest getJobsRequest = JobApi.GetJobsRequest.newBuilder().build();

    final int maxRunningJobs = maxInvocationHistory + new Random().nextInt(50);
    
    final List<Consumer<JobApi.JobStateEvent>> stateListeners = new ArrayList<>(maxRunningJobs);

    for (int i = 0; i < maxRunningJobs; i++) {
      when(invocation.getId()).thenReturn(String.valueOf(i));
      prepareAndRunJob();

      
      ArgumentCaptor<Consumer<JobApi.JobStateEvent>> stateListener =
          ArgumentCaptor.forClass(Consumer.class);
      verify(invocation, times(i + 1)).addStateListener(stateListener.capture());
      stateListeners.add(stateListener.getValue());

      
      RecordingObserver<JobApi.GetJobsResponse> recorder = new RecordingObserver<>();
      service.getJobs(getJobsRequest, recorder);
      assertThat(recorder.isSuccessful(), is(true));

      
      assertThat(recorder.getValue().getJobInfoCount(), is(i + 1));
    }

    
    JobApi.JobStateEvent terminalEvent =
        JobApi.JobStateEvent.newBuilder().setState(JobApi.JobState.Enum.DONE).build();

    for (int i = 0; i < maxRunningJobs; i++) {
      
      stateListeners.get(i).accept(terminalEvent);

      RecordingObserver<JobApi.GetJobsResponse> recorder = new RecordingObserver<>();
      service.getJobs(getJobsRequest, recorder);

      
      
      int jobInfoCount = recorder.getValue().getJobInfoCount();
      if (i / maxInvocationHistory == 0) {
        
        assertThat(jobInfoCount, is(maxRunningJobs));
      } else {
        
        assertThat(jobInfoCount, is(maxInvocationHistory + maxRunningJobs - i - 1));
      }
    }
  }

Line commented on: if (i / maxInvocationHistory == 0) {

Revised code:
public void testInvocationCleanup() {
    final int maxInvocationHistory = 3;
    service =
        InMemoryJobService.create(
            stagingServiceDescriptor, session -> "token", null, invoker, maxInvocationHistory);

    assertThat(getNumberOfInvocations(), is(0));

    Job job1 = runJob();
    assertThat(getNumberOfInvocations(), is(1));
    Job job2 = runJob();
    assertThat(getNumberOfInvocations(), is(2));
    Job job3 = runJob();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory));

    
    
    Job job4 = runJob();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory + 1));

    
    job1.finish();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory + 1));
    job2.finish();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory + 1));
    job3.finish();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory + 1));

    
    job4.finish();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory));

    
    Job job5 = runJob();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory + 1));
    job5.finish();
    assertThat(getNumberOfInvocations(), is(maxInvocationHistory));
  }
Code review comment: please help change to `spring-cloud-azure-not-existing-xx`

Code under review:
protected void doHealthCheck(Health.Builder builder) {
        try {
            this.secretAsyncClient.getSecretWithResponse("spring-cloud-azure-none-existing-secret", "")
                .block(timeout);
            builder.up();
        } catch (Exception e) {
            if (e instanceof ResourceNotFoundException) {
                builder.up();
            } else {
                throw e;
            }
        }
    }

Line commented on: this.secretAsyncClient.getSecretWithResponse("spring-cloud-azure-none-existing-secret", "")

Revised code:
protected void doHealthCheck(Health.Builder builder) {
        try {
            this.secretAsyncClient.getSecretWithResponse("spring-cloud-azure-not-existing-secret", "")
                .block(timeout);
            builder.up();
        } catch (Exception e) {
            if (e instanceof ResourceNotFoundException) {
                builder.up();
            } else {
                throw e;
            }
        }
    }
Code review comment: Remember to use the new getter after merging #5306

Code under review:
private static DiscFilterRequest toDiscFilterRequest(Request request) {
        DiscFilterRequest r = mock(DiscFilterRequest.class);
        when(r.getMethod()).thenReturn(request.method().name());
        when(r.getUri()).thenReturn(URI.create("http:
        when(r.getRemoteAddr()).thenReturn(request.remoteAddr());
        if (request.commonName().isPresent()) {
            X509Certificate cert = certificateFor(request.commonName().get(), keyPair());
            when(r.getAttribute(ServletRequest.JDISC_REQUEST_X509CERT))
                    .thenReturn(new X509Certificate[]{cert});
        }
        return r;
    }

Line commented on: when(r.getAttribute(ServletRequest.JDISC_REQUEST_X509CERT))

Revised code:
private static DiscFilterRequest toDiscFilterRequest(Request request) {
        DiscFilterRequest r = mock(DiscFilterRequest.class);
        when(r.getMethod()).thenReturn(request.method().name());
        when(r.getUri()).thenReturn(URI.create("http:
        when(r.getRemoteAddr()).thenReturn(request.remoteAddr());
        if (request.commonName().isPresent()) {
            X509Certificate cert = certificateFor(request.commonName().get(), keyPair());
            when(r.getAttribute(ServletRequest.JDISC_REQUEST_X509CERT))
                    .thenReturn(new X509Certificate[]{cert});
        }
        return r;
    }
Code review comment: Add unit test for this as we should make sure the map is getting serialized and deserialized correctly.

Code under review:
public String getClaims() {
        final Map<String, String> resourceClaims = getResourceClaims();
        if (!resourceClaims.isEmpty()) {
            try {
                return defaultSerializerAdapter.serialize(resourceClaims, SerializerEncoding.JSON);
            } catch (IOException e) {
                throw LOGGER.logExceptionAsError(new RuntimeException(e));
            }
        }
        return null;
    }

Line commented on: return defaultSerializerAdapter.serialize(resourceClaims, SerializerEncoding.JSON);

Revised code:
public String getClaims() {
        final Map<String, String> resourceClaims = getResourceClaims();
        if (!resourceClaims.isEmpty()) {
            try {
                return DEFAULT_SERIALIZER_ADAPTER.serialize(resourceClaims, SerializerEncoding.JSON);
            } catch (IOException ex) {
                throw LOGGER.logExceptionAsError(new UncheckedIOException(ex));
            }
        }
        return null;
    }
Code review comment: what if the nested array type?

Code under review:
public static boolean argumentTypeContainDecimalV2(String fnName, Type[] argumentTypes) {
        if (!FunctionSet.ARRAY_SLICE.equals(fnName)) {
            return false;
        }

        if (Arrays.stream(argumentTypes).anyMatch(Type::isDecimalV2)) {
            return true;
        }

        
        return Arrays.stream(argumentTypes).filter(Type::isArrayType).map(t -> (ArrayType) t)
                .anyMatch(t -> t.getItemType().isDecimalV2());
    }

Line commented on: return Arrays.stream(argumentTypes).filter(Type::isArrayType).map(t -> (ArrayType) t)

Revised code:
public static boolean argumentTypeContainDecimalV2(String fnName, Type[] argumentTypes) {
        if (!FunctionSet.ARRAY_SLICE.equals(fnName)) {
            return false;
        }

        if (Arrays.stream(argumentTypes).anyMatch(Type::isDecimalV2)) {
            return true;
        }

        
        return Arrays.stream(argumentTypes).filter(Type::isArrayType).map(t -> (ArrayType) t)
                .anyMatch(t -> t.getItemType().isDecimalV2());
    }
Code review comment: I will optimize the type of statement intercepted `QueryableRALStatement`, `QueryableScalingRALStatement`, `UpdatableScalingRALStatement` they will be ignored.

Code under review:
public static TextProtocolBackendHandler newInstance(final DatabaseType databaseType, final DistSQLStatement sqlStatement, final ConnectionSession connectionSession) throws SQLException {
        if (sqlStatement instanceof RQLStatement) {
            return RQLBackendHandlerFactory.newInstance((RQLStatement) sqlStatement, connectionSession);
        }
        checkLockedSchema(connectionSession);
        if (sqlStatement instanceof RDLStatement) {
            return RDLBackendHandlerFactory.newInstance(databaseType, (RDLStatement) sqlStatement, connectionSession);
        }
        if (sqlStatement instanceof RALStatement) {
            return RALBackendHandlerFactory.newInstance(databaseType, (RALStatement) sqlStatement, connectionSession);
        }
        throw new UnsupportedOperationException(sqlStatement.getClass().getCanonicalName());
    }

Line commented on: if (sqlStatement instanceof RDLStatement) {

Revised code:
public static TextProtocolBackendHandler newInstance(final DatabaseType databaseType, final DistSQLStatement sqlStatement, final ConnectionSession connectionSession) throws SQLException {
        if (sqlStatement instanceof RQLStatement) {
            return RQLBackendHandlerFactory.newInstance((RQLStatement) sqlStatement, connectionSession);
        }
        if (sqlStatement instanceof RDLStatement) {
            checkLockedSchema(connectionSession);
            return RDLBackendHandlerFactory.newInstance(databaseType, (RDLStatement) sqlStatement, connectionSession);
        }
        if (sqlStatement instanceof RALStatement) {
            if (sqlStatement instanceof QueryableRALStatement || sqlStatement instanceof QueryableScalingRALStatement || sqlStatement instanceof UpdatableScalingRALStatement) {
                return RALBackendHandlerFactory.newInstance(databaseType, (RALStatement) sqlStatement, connectionSession);
            }
            checkLockedSchema(connectionSession);
            return RALBackendHandlerFactory.newInstance(databaseType, (RALStatement) sqlStatement, connectionSession);
        }
        throw new UnsupportedOperationException(sqlStatement.getClass().getCanonicalName());
    }
Code review comment: Could you test a case with `select * from tbl where parition_key is null` when there exists null values in the first parition and normal values in other paritions? The expect prune result should be one partition.

Code under review:
private BitSet evaluateNullRangeHitSet(ScalarOperator scalarOperator, Type columnType) {
            PartitionKey nullKey = new PartitionKey();
            nullKey.pushColumn(NullLiteral.create(columnType), columnType.getPrimitiveType());
            Range<PartitionKey> nullRange = Range.closed(nullKey, nullKey);
            return evaluateRangeHitSet(scalarOperator, nullRange);
        }

Line commented on: return evaluateRangeHitSet(scalarOperator, nullRange);

Revised code:
private BitSet evaluateNullRangeHitSet(ScalarOperator scalarOperator, Type columnType) {
            PartitionKey nullKey = new PartitionKey();
            nullKey.pushColumn(NullLiteral.create(columnType), columnType.getPrimitiveType());
            Range<PartitionKey> nullRange = Range.closed(nullKey, nullKey);
            return evaluateRangeHitSet(scalarOperator, nullRange);
        }
Code review comment: nit: One could introduce variables for `foo` and `bar` so that we don't have two places where they are defined.

Code under review:
public void testVariableExclusionParsing() throws Exception {
		final Configuration config = new Configuration();
		config.setString(ConfigConstants.METRICS_REPORTER_PREFIX + "test." + ConfigConstants.METRICS_REPORTER_FACTORY_CLASS_SUFFIX, TestReporterFactory.class.getName());
		config.setString(ConfigConstants.METRICS_REPORTER_PREFIX + "test." + ConfigConstants.METRICS_REPORTER_EXCLUDED_VARIABLES, "foo;bar");

		final List<ReporterSetup> reporterSetups = ReporterSetup.fromConfiguration(config);

		assertEquals(1, reporterSetups.size());

		final ReporterSetup reporterSetup = reporterSetups.get(0);

		assertThat(reporterSetup.getExcludedVariables(), hasItems(ScopeFormat.asVariable("foo"), ScopeFormat.asVariable("bar")));
	}

Line commented on: assertThat(reporterSetup.getExcludedVariables(), hasItems(ScopeFormat.asVariable("foo"), ScopeFormat.asVariable("bar")));

Revised code:
public void testVariableExclusionParsing() throws Exception {
		final String excludedVariable1 = "foo";
		final String excludedVariable2 = "foo";
		final Configuration config = new Configuration();
		config.setString(ConfigConstants.METRICS_REPORTER_PREFIX + "test." + ConfigConstants.METRICS_REPORTER_FACTORY_CLASS_SUFFIX, TestReporterFactory.class.getName());
		config.setString(ConfigConstants.METRICS_REPORTER_PREFIX + "test." + ConfigConstants.METRICS_REPORTER_EXCLUDED_VARIABLES, excludedVariable1 + ";" + excludedVariable2);

		final List<ReporterSetup> reporterSetups = ReporterSetup.fromConfiguration(config);

		assertEquals(1, reporterSetups.size());

		final ReporterSetup reporterSetup = reporterSetups.get(0);

		assertThat(reporterSetup.getExcludedVariables(), hasItems(ScopeFormat.asVariable(excludedVariable1), ScopeFormat.asVariable(excludedVariable2)));
	}
Code review comment: This and the next are from an (unsuccessful) attempt to use in process channel. Would still prefer to do that, if someone knows how?

Code under review:
private DockerJobBundleFactory createJobBundleFactory(JobInfo jobInfo) throws Exception {
    return new DockerJobBundleFactory(jobInfo) {

      @Override
      protected ServerFactory getServerFactory() {
        return ServerFactory.createWithUrlFactory(
            (host, port) -> HostAndPort.fromParts("localhost", port).toString());
        
      }

      @Override
      protected EnvironmentFactory getEnvironmentFactory(
          GrpcFnServer<FnApiControlClientPoolService> controlServer,
          GrpcFnServer<GrpcLoggingService> loggingServer,
          GrpcFnServer<ArtifactRetrievalService> retrievalServer,
          GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
          Source clientSource,
          IdGenerator idGenerator) {
        String workerId = "fakeWorkerId";
        ThreadFactory threadFactory = new ThreadFactoryBuilder().setDaemon(true).build();
        
        sdkHarnessExecutor = Executors.newSingleThreadExecutor(threadFactory);
        sdkHarnessExecutorFuture =
            sdkHarnessExecutor.submit(
                () -> {
                  try {
                    FnHarness.main(
                        workerId,
                        PipelineOptionsFactory.create(),
                        loggingServer.getApiServiceDescriptor(),
                        controlServer.getApiServiceDescriptor(),
                        ManagedChannelFactory.createDefault(),
                        
                        HarnessStreamObserverFactories.fromOptions(
                            PipelineOptionsFactory.create()));
                  } catch (Exception e) {
                    throw new RuntimeException(e);
                  }
                });
        try {
          InstructionRequestHandler controlClient =
              clientSource.take(workerId, Duration.ofSeconds(2));
          return (container) -> RemoteEnvironment.forHandler(container, controlClient);
        } catch (Exception ex) {
          throw new RuntimeException(ex);
        }
      }
    };
  }


Revised code:
private DockerJobBundleFactory createJobBundleFactory(JobInfo jobInfo) throws Exception {
    return new DockerJobBundleFactory(jobInfo) {

      @Override
      protected ServerFactory getServerFactory() {
        return InProcessServerFactory.create();
      }

      @Override
      protected EnvironmentFactory getEnvironmentFactory(
          GrpcFnServer<FnApiControlClientPoolService> controlServer,
          GrpcFnServer<GrpcLoggingService> loggingServer,
          GrpcFnServer<ArtifactRetrievalService> retrievalServer,
          GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
          Source clientSource,
          IdGenerator idGenerator) {
        return InProcessEnvironmentFactory.create(
            PipelineOptionsFactory.create(), loggingServer, controlServer, clientSource);
      }
    };
  }
Code review comment: Is there a reason why `FindBlobsOptions` can no longer be null? Or did this throw an NPE if null previously and is being triggered sooner?

Code under review:
PagedFlux<TaggedBlobItem> findBlobsByTags(FindBlobsOptions options, Duration timeout) {
        throwOnAnonymousAccess();
        StorageImplUtils.assertNotNull("options", options);
        BiFunction<String, Integer, Mono<PagedResponse<TaggedBlobItem>>> func =
            (marker, pageSize) -> {
                options.setMaxResultsPerPage(pageSize);
                return withContext(context -> this.findBlobsByTags(options, marker, timeout, context));
            };
        return StoragePagedFlux.create(pageSize -> func.apply(null, pageSize), func);
    }

Line commented on: StorageImplUtils.assertNotNull("options", options);

Revised code:
PagedFlux<TaggedBlobItem> findBlobsByTags(FindBlobsOptions options, Duration timeout) {
        throwOnAnonymousAccess();
        StorageImplUtils.assertNotNull("options", options);
        BiFunction<String, Integer, Mono<PagedResponse<TaggedBlobItem>>> func =
            (marker, pageSize) -> withContext(context -> this.findBlobsByTags(
                new FindBlobsOptions(options.getQuery()).setMaxResultsPerPage(pageSize), marker, timeout, context));
        return new PagedFlux<>(pageSize -> func.apply(null, pageSize), func);
    }
Code review comment: I have no idea what you're talking about ;-). Chars in Java are unicode, or? I would be more than grateful if you could check whatever needs to be checked.

Code under review:
private String doEscape(CharSequence value, int index, StringBuilder builder) {
        int length = value.length();
        while (++index < length) {
            char c = value.charAt(index);
            String replacement = replacements.get(c);
            if (replacement != null) {
                builder.append(replacement);
            } else {
                builder.append(c);
            }
        }
        return builder.toString();
    }

Line commented on: char c = value.charAt(index);

Revised code:
private String doEscape(CharSequence value, int index, StringBuilder builder) {
        int length = value.length();
        while (++index < length) {
            char c = value.charAt(index);
            String replacement = replacements.get(c);
            if (replacement != null) {
                builder.append(replacement);
            } else {
                builder.append(c);
            }
        }
        return builder.toString();
    }
Code review comment: nit: It actually seems irrelevant to set this option. I don't really mind either way, but I would suggest using the same approach in `ZooKeeperLeaderElectionConnectionHandlingTest` (I got confused why it was not required there...)

Code under review:
public void before() throws Exception {
        testingServer = new TestingServer();

        config = new Configuration();
        config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
        config.setString(
                HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM, testingServer.getConnectString());

        zooKeeperClient = ZooKeeperUtils.startCuratorFramework(config);
        zooKeeperClient.blockUntilConnected();
    }

Line commented on: config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");

Revised code:
public void before() throws Exception {
        testingServer = new TestingServer();

        final Configuration config = new Configuration();
        config.setString(
                HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM, testingServer.getConnectString());

        zooKeeperClient = ZooKeeperUtils.startCuratorFramework(config);
        zooKeeperClient.blockUntilConnected();
    }
Code review comment: Please, add more detailed error message, like `SparkReceiver was interrupted because of <>`.

Code under review:
public void start(Receiver<V> sparkReceiver) {
      this.sparkReceiver = sparkReceiver;
      try {
        new WrappedSupervisor(
            sparkReceiver,
            new SparkConf(),
            objects -> {
              V record = (V) objects[0];
              recordsQueue.offer(record);
              return null;
            });
      } catch (Exception e) {
        LOG.error("Can not init Spark Receiver!", e);
      }
      ((HasOffset) sparkReceiver).setStartOffset(startOffset);
      sparkReceiver.supervisor().startReceiver();
      try {
        TimeUnit.MILLISECONDS.sleep(START_POLL_TIMEOUT_MS);
      } catch (InterruptedException e) {
        LOG.error("Interrupted", e);
      }
    }

Line commented on: LOG.error("Interrupted", e);

Revised code:
public void start(Receiver<V> sparkReceiver) {
      this.sparkReceiver = sparkReceiver;
      try {
        new WrappedSupervisor(
            sparkReceiver,
            new SparkConf(),
            objects -> {
              V record = (V) objects[0];
              recordsQueue.offer(record);
              return null;
            });
      } catch (Exception e) {
        LOG.error("Can not init Spark Receiver!", e);
        throw new IllegalStateException("Spark Receiver was not initialized");
      }
      ((HasOffset) sparkReceiver).setStartOffset(startOffset);
      sparkReceiver.supervisor().startReceiver();
      try {
        TimeUnit.MILLISECONDS.sleep(START_POLL_TIMEOUT_MS);
      } catch (InterruptedException e) {
        LOG.error("SparkReceiver was interrupted before polling started", e);
        throw new IllegalStateException("Spark Receiver was interrupted before polling started");
      }
    }
Code review comment: It's possible that the URL is "file://"

Code under review:
String resolve() throws IOException {

    String from =
        String.format("%s/v%s/%s.zip", GITHUB_DOWNLOAD_PREFIX, getSDKVersion(), buildFileName());

    if (!Strings.isNullOrEmpty(options.getPrismLocation())) {
      checkArgument(
          !options.getPrismLocation().startsWith(GITHUB_TAG_PREFIX),
          "Provided --prismLocation URL is not an Apache Beam Github "
              + "Release page URL or download URL: ",
          from);

      from = options.getPrismLocation();
    }

    String fromFileName = getNameWithoutExtension(from);
    Path to = Paths.get(userHome(), PRISM_BIN_PATH, fromFileName);

    if (Files.exists(to)) {
      return to.toString();
    }

    createDirectoryIfNeeded(to);

    if (from.startsWith("http")) {
      String result = resolve(new URL(from), to);
      checkState(Files.exists(to), "Resolved location does not exist: %s", result);
      return result;
    }

    String result = resolve(Paths.get(from), to);
    checkState(Files.exists(to), "Resolved location does not exist: %s", result);
    return result;
  }

Line commented on: if (from.startsWith("http")) {

Revised code:
String resolve() throws IOException {

    String from =
        String.format("%s/v%s/%s.zip", GITHUB_DOWNLOAD_PREFIX, getSDKVersion(), buildFileName());

    if (!Strings.isNullOrEmpty(options.getPrismLocation())) {
      checkArgument(
          !options.getPrismLocation().startsWith(GITHUB_TAG_PREFIX),
          "Provided --prismLocation URL is not an Apache Beam Github "
              + "Release page URL or download URL: ",
          from);

      from = options.getPrismLocation();
    }

    String fromFileName = getNameWithoutExtension(from);
    Path to = Paths.get(userHome(), PRISM_BIN_PATH, fromFileName);

    if (Files.exists(to)) {
      return to.toString();
    }

    createDirectoryIfNeeded(to);

    if (from.startsWith("http")) {
      String result = resolve(new URL(from), to);
      checkState(Files.exists(to), "Resolved location does not exist: %s", result);
      return result;
    }

    String result = resolve(Paths.get(from), to);
    checkState(Files.exists(to), "Resolved location does not exist: %s", result);
    return result;
  }