muttakin2301/ccr2000 huggingface dataset:
Code review comment: Do we already have positive tests for this?

Code under review:
public void setup() {
        compileResult = BCompileUtil.compile("test-src/annotations/annot_attachments_negative.bal");
        Assert.assertEquals(compileResult.getErrorCount(), 266);
    }

Line commented on: Assert.assertEquals(compileResult.getErrorCount(), 266);

Revised code:
public void setup() {
        compileResult = BCompileUtil.compile("test-src/annotations/annot_attachments_negative.bal");
        Assert.assertEquals(compileResult.getErrorCount(), 266);
    }
Code review comment: I think this is no longer valid.

Code under review:
private ClassLoader getClassLoader() {
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        if (cl == null) {
            cl = getClass().getClassLoader();
        }
        
        
        
        
        if (cl == null) {
            cl = Object.class.getClassLoader();
        }
        return cl;
    }


Revised code:
private ClassLoader getClassLoader() {
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        if (cl == null) {
            cl = getClass().getClassLoader();
        }
        if (cl == null) {
            cl = Object.class.getClassLoader();
        }
        return cl;
    }
Code review comment: Okay got it :) So we can remove the check for the os and have only the check for the file name

Code under review:
private static boolean isDirEmpty(final Path directory) throws IOException {

        try (DirectoryStream<Path> dirStream = Files.newDirectoryStream(directory)) {
            
            Iterator pathIterator = dirStream.iterator();
            if (pathIterator.hasNext()) {

                Path path = (Path) pathIterator.next();
                Path filePath = path.getFileName();
                if (filePath != null && filePath.toString().equals(ProjectDirConstants.MAC_OS_DS_STORE_FILE)) {

                    return System.getProperty("os.name").toLowerCase(Locale.ENGLISH).contains("mac") &&
                            !pathIterator.hasNext();
                } else {
                    return false;
                }

            } else {
                return true;
            }
        }
    }

Line commented on: !pathIterator.hasNext();

Revised code:
private static boolean isDirEmpty(final Path directory) throws IOException {

        try (DirectoryStream<Path> dirStream = Files.newDirectoryStream(directory)) {
            
            Iterator pathIterator = dirStream.iterator();
            if (!pathIterator.hasNext()) {
                return true;
            }
            Path path = (Path) pathIterator.next();
            Path fileName = path.getFileName();
            return fileName != null && fileName.toString().equals(ProjectDirConstants.DS_STORE_FILE) &&
                    !pathIterator.hasNext();
        }
    }
Code review comment: If you need to support functions not in the form of xx(), but fun(A, B) in the future, how do you need to be compatible?

Code under review:
public Expr obtainExpr() {
        if (SUPPORTED_DEFAULT_FNS.contains(expr)) {
            String functionName = expr.replace("()", "");
            FunctionCallExpr functionCallExpr = new FunctionCallExpr(new FunctionName(functionName), Lists.newArrayList());
            Function fn = Expr.getBuiltinFunction(functionName, new Type[] {}, Function.CompareMode.IS_IDENTICAL);
            functionCallExpr.setFn(fn);
            functionCallExpr.setType(fn.getReturnType());
            return functionCallExpr;
        }
        return null;
    }

Line commented on: String functionName = expr.replace("()", "");

Revised code:
public Expr obtainExpr() {
        if (SUPPORTED_DEFAULT_FNS.contains(expr)) {
            String functionName = expr.replace("()", "");
            FunctionCallExpr functionCallExpr = new FunctionCallExpr(new FunctionName(functionName), Lists.newArrayList());
            Function fn = Expr.getBuiltinFunction(functionName, new Type[] {}, Function.CompareMode.IS_IDENTICAL);
            functionCallExpr.setFn(fn);
            functionCallExpr.setType(fn.getReturnType());
            return functionCallExpr;
        }
        return null;
    }
Code review comment: Why are we setting a default page size here? Shouldn't this be handled by the creator of this type?

Code under review:
public ContinuablePagedIterable(ContinuablePagedFlux<C, T, P> pagedFlux, int batchSize) {
        super(pagedFlux);
        this.pagedFlux = pagedFlux;
        this.batchSize = batchSize;
        this.defaultPageSize = 1;
        this.continuationPredicate = null;
        this.syncPageRetrieverProvider = null;
    }

Line commented on: this.defaultPageSize = 1;

Revised code:
public ContinuablePagedIterable(ContinuablePagedFlux<C, T, P> pagedFlux, int batchSize) {
        super(pagedFlux);
        this.pagedFlux = pagedFlux;
        this.batchSize = batchSize;
        this.defaultPageSize = null;
        this.continuationPredicate = null;
        this.pageRetrieverSyncProvider = null;
    }
Code review comment: @pubudu538 ATM T1 values are all capital. eg: `ARRAY`. Shall we make it like `Array`?

Code under review:
private String getTypeErrorMessage(Type found) {
        Map<String, String> message = this.schema.message();
        String typeCustomMessage = message.get(SchemaDeserializer.TYPE);
        if (typeCustomMessage == null) {
            return String.format("key '%s' expects %s . found %s", this.key, schema.type(), found);
        }
        return typeCustomMessage;
    }

Line commented on: return String.format("key '%s' expects %s . found %s", this.key, schema.type(), found);

Revised code:
private String getTypeErrorMessage(Type found) {
        Map<String, String> message = this.schema.message();
        String typeCustomMessage = message.get(SchemaDeserializer.TYPE);
        if (typeCustomMessage == null) {
            return String.format("incompatible type for key '%s': expected '%s', found '%s'", this.key, schema.type(),
                    found);
        }
        return typeCustomMessage;
    }
Code review comment: Imo it just shouldnt print anythiNg unless debug.   Why would we want those behind firewalls to get annoying message on every build?

Code under review:
public void close() {
        try {
            
            CompletableFuture.allOf(postFutures.toArray(new CompletableFuture[0])).get(
                    PropertyUtils.getProperty("quarkus.analytics.timeout", DEFAULT_TIMEOUT),
                    TimeUnit.MILLISECONDS);
            if (log.isDebugEnabled() && !postFutures.isEmpty()) {
                log.debug("[Quarkus build analytics] Build analytics sent successfully. Sent event can be seen at .../target/" +
                        fileLocations.lastTrackFileName());
            }
        } catch (ExecutionException | TimeoutException e) {
            if (log.isDebugEnabled()) {
                log.debug("[Quarkus build analytics] Failed to send build analytics to Segment: " +
                        e.getClass().getName() + ": " +
                        (e.getMessage() == null ? "(no message)" : e.getMessage()));
            }
            log.info("[Quarkus build analytics] Failed to send build analytics to Segment. " +
                    "Connection might not be available or is too slow.");
        } catch (Exception e) {
            log.warn("[Quarkus build analytics] Failed to send build analytics to Segment: " +
                    e.getClass().getName() + ": " +
                    (e.getMessage() == null ? "(no message)" : e.getMessage()));
        }
    }

Line commented on: log.info("[Quarkus build analytics] Failed to send build analytics to Segment. " +

Revised code:
public void close() {
        try {
            
            CompletableFuture.allOf(postFutures.toArray(new CompletableFuture[0])).get(
                    PropertyUtils.getProperty("quarkus.analytics.timeout", DEFAULT_TIMEOUT),
                    TimeUnit.MILLISECONDS);
            if (log.isDebugEnabled() && !postFutures.isEmpty()) {
                log.debug("[Quarkus build analytics] Build analytics sent successfully. Sent event can be seen at .../target/" +
                        fileLocations.lastTrackFileName());
            }
        } catch (ExecutionException | TimeoutException e) {
            if (log.isDebugEnabled()) {
                log.debug("[Quarkus build analytics] Failed to send build analytics to Segment. " +
                        "Connection might not be available or is too slow: " +
                        e.getClass().getName() + ": " +
                        (e.getMessage() == null ? "(no message)" : e.getMessage()));
            }
        } catch (Exception e) {
            if (log.isDebugEnabled()) {
                log.debug("[Quarkus build analytics] Failed to send build analytics to Segment: " +
                        e.getClass().getName() + ": " +
                        (e.getMessage() == null ? "(no message)" : e.getMessage()));
            }
        }
    }
Code review comment: Should we also add this change to the changelog?

Code under review:
Mono<PiiEntityCollection> recognizePiiEntities(String document, String language) {
        try {
            Objects.requireNonNull(document, "'document' cannot be null.");
            return recognizePiiEntitiesBatch(
                Collections.singletonList(new TextDocumentInput("0", document).setLanguage(language)), null)
                .map(resultCollectionResponse -> {
                    PiiEntityCollection entityCollection = null;
                    
                    for (RecognizePiiEntitiesResult entitiesResult : resultCollectionResponse.getValue()) {
                        if (entitiesResult.isError()) {
                            throw logger.logExceptionAsError(toTextAnalyticsException(entitiesResult.getError()));
                        }
                        entityCollection = new PiiEntityCollection(entitiesResult.getEntities(),
                            entitiesResult.getEntities().getRedactedText(),
                            entitiesResult.getEntities().getWarnings());
                    }
                    return entityCollection;
                });
        } catch (RuntimeException ex) {
            return monoError(logger, ex);
        }
    }

Line commented on: throw logger.logExceptionAsError(toTextAnalyticsException(entitiesResult.getError()));

Revised code:
Mono<PiiEntityCollection> recognizePiiEntities(String document, String language) {
        try {
            Objects.requireNonNull(document, "'document' cannot be null.");
            return recognizePiiEntitiesBatch(
                Collections.singletonList(new TextDocumentInput("0", document).setLanguage(language)), null)
                .map(resultCollectionResponse -> {
                    PiiEntityCollection entityCollection = null;
                    
                    for (RecognizePiiEntitiesResult entitiesResult : resultCollectionResponse.getValue()) {
                        if (entitiesResult.isError()) {
                            throw logger.logExceptionAsError(toTextAnalyticsException(entitiesResult.getError()));
                        }
                        entityCollection = new PiiEntityCollection(entitiesResult.getEntities(),
                            entitiesResult.getEntities().getRedactedText(),
                            entitiesResult.getEntities().getWarnings());
                    }
                    return entityCollection;
                });
        } catch (RuntimeException ex) {
            return monoError(logger, ex);
        }
    }
Code review comment: Let's open a separate issue for that, it's a bigger piece of work

Code under review:
public String getId() {
        String id = this.title.toLowerCase().replaceAll(SPACE, DASH);
        try {
            id = URLEncoder.encode(id, StandardCharsets.UTF_8.toString());
        } catch (UnsupportedEncodingException ex) {
            throw new RuntimeException(ex);
        }

        if (!this.isInternal() && this.namespace != null) {
            
            id = this.namespace.toLowerCase() + SLASH + id;
        } else if (this.isInternal() && this.namespace != null) {
            
            String d = "devui-" + id;
            if (d.equals(this.namespace)) {
                return id;
            } else {
                int i = this.namespace.indexOf(DASH) + 1;
                String stripDevui = this.namespace.substring(i);
                return stripDevui + DASH + id;
            }
        }
        return id;
    }

Line commented on: id = URLEncoder.encode(id, StandardCharsets.UTF_8.toString());

Revised code:
public String getId() {
        String id = this.title.toLowerCase().replaceAll(SPACE, DASH);
        try {
            id = URLEncoder.encode(id, StandardCharsets.UTF_8.toString());
        } catch (UnsupportedEncodingException ex) {
            throw new RuntimeException(ex);
        }

        if (!this.isInternal() && this.namespace != null) {
            
            id = this.namespace.toLowerCase() + SLASH + id;
        } else if (this.isInternal() && this.namespace != null) {
            
            String d = "devui-" + id;
            if (d.equals(this.namespace)) {
                return id;
            } else {
                int i = this.namespace.indexOf(DASH) + 1;
                String stripDevui = this.namespace.substring(i);
                return stripDevui + DASH + id;
            }
        }
        return id;
    }
Code review comment: Also, if we can, don't forget to update `AtomicCtasITCase`.

Code under review:
protected TableEnvironment getTableEnvironment() {
        EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build();
        return StreamTableEnvironment.create(
                StreamExecutionEnvironment.getExecutionEnvironment(), settings);
    }

Line commented on: return StreamTableEnvironment.create(

Revised code:
protected TableEnvironment getTableEnvironment() {
        EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build();
        return StreamTableEnvironment.create(
                StreamExecutionEnvironment.getExecutionEnvironment(), settings);
    }
Code review comment: @maxandersen No, This PR will affect neither `http://127.0.0.1:8080/q/dev/` nor `0.0.0.0:8080/q/dev/` since DevUI pages are not implemented as SPA using XHR or Fetch scripts, so `Origin` will not be even produced by the browser. For the next version of DevUI the Origin check may need to be adjusted.

Code under review:
public void handle(RoutingContext event) {
        HttpServerRequest request = event.request();
        HttpServerResponse response = event.response();
        String origin = request.getHeader(HttpHeaders.ORIGIN);
        if (origin == null) {
            event.next();
        } else {
            if (!origin.contains(LOCAL_HOST)) {
                response.end();
            } else {
                super.handle(event);
            }
        }
    }

Line commented on: response.end();

Revised code:
public void handle(RoutingContext event) {
        HttpServerRequest request = event.request();
        HttpServerResponse response = event.response();
        String origin = request.getHeader(HttpHeaders.ORIGIN);
        if (origin == null) {
            corsFilter().handle(event);
        } else {
            if (origin.startsWith(HTTP_LOCAL_HOST) || origin.startsWith(HTTPS_LOCAL_HOST)
                    || origin.startsWith(HTTP_LOCAL_HOST_IP) || origin.startsWith(HTTPS_LOCAL_HOST_IP)) {
                corsFilter().handle(event);
            } else {
                LOG.errorf("Only localhost origin is allowed, but Origin header value is: %s", origin);
                response.setStatusCode(403);
                response.setStatusMessage("CORS Rejected - Invalid origin");
                response.end();
            }
        }
    }
Code review comment: stopServices may throw an exception on failure.  But we want to ignore such a failure, I think. @hmusum   Do we also want to ignore stopServices() if active? I think so.  Taking down the container may be considered equivalent to a reboot of a node, in case stopping the services are first tried, then forced (KILL), as part of reboot.

Code under review:
private Optional<Container> removeContainerIfNeeded(ContainerNodeSpec nodeSpec) {
        Optional<Container> existingContainer = getContainer();
        if (!existingContainer.isPresent()) return Optional.empty();

        Optional<String> removeReason = shouldRemoveContainer(nodeSpec, existingContainer.get());
        if (removeReason.isPresent()) {
            logger.info("Will remove container " + existingContainer.get() + ": " + removeReason.get());

            if (existingContainer.get().state.isRunning()) {
                if (nodeSpec.nodeState == Node.State.active) {
                    orchestratorSuspendNode();
                }
                stopServices();
            }
            vespaVersion = Optional.empty();
            dockerOperations.removeContainer(existingContainer.get());
            metricReceiver.unsetMetricsForContainer(hostname);
            containerState = ABSENT;
            return Optional.empty();
        }
        return existingContainer;
    }

Line commented on: stopServices();

Revised code:
private Optional<Container> removeContainerIfNeeded(ContainerNodeSpec nodeSpec) {
        Optional<Container> existingContainer = getContainer();
        if (!existingContainer.isPresent()) return Optional.empty();

        Optional<String> removeReason = shouldRemoveContainer(nodeSpec, existingContainer.get());
        if (removeReason.isPresent()) {
            logger.info("Will remove container " + existingContainer.get() + ": " + removeReason.get());

            if (existingContainer.get().state.isRunning()) {
                if (nodeSpec.nodeState == Node.State.active) {
                    orchestratorSuspendNode();
                }

                try {
                    stopServices();
                } catch (Exception e) {
                    logger.info("Failed stopping services, ignoring", e);
                }
            }
            vespaVersion = Optional.empty();
            dockerOperations.removeContainer(existingContainer.get());
            metricReceiver.unsetMetricsForContainer(hostname);
            containerState = ABSENT;
            return Optional.empty();
        }
        return existingContainer;
    }
Code review comment: this is the most difficult model I have ever seen :laughing:

Code under review:
public JsonWriter toJson(JsonWriter jsonWriter) {
        return null;
    }

Line commented on: return null;

Revised code:
public JsonWriter toJson(JsonWriter jsonWriter) {
        return toJsonInternal(jsonWriter, "foo");
    }
Code review comment: I wanted to keep the declaration of a class and its features encapsulated in single class. Agree that this doesn't play well with the rest of the `DoFnSignaturesTest`. But because there is effort in the direction of superseding all this with "pipeline features", I think it is fine to keep is as is for now and drop it as part of the effort later.

Code under review:
public void testAllDoFnFeatures() {
    tests.forEach(FeatureTest::test);
  }

Line commented on: tests.forEach(FeatureTest::test);

Revised code:
public void testAllDoFnFeatures() {
    tests.forEach(FeatureTest::test);
  }
Code review comment: hostNames is supposed to contain parentHostName already (if applicable)

Code under review:
public void suspend(String parentHostName, List<String> hostNames) {
        final BatchOperationResult batchOperationResult;
        try {
            String params = String.join("&hostname=", hostNames);
            String url = String.format("%s/%s?hostname=%s", ORCHESTRATOR_PATH_PREFIX_HOST_SUSPENSION_API,
                                       parentHostName, params);
            batchOperationResult = configServerApi.put(url, Optional.empty(), BatchOperationResult.class);
        } catch (HttpException e) {
            throw new OrchestratorException("Failed to batch suspend for " +
                    parentHostName + ": " + e.toString());
        } catch (Exception e) {
            throw new RuntimeException("Got error on batch suspend for " + parentHostName + ", with nodes " + hostNames, e);
        }

        batchOperationResult.getFailureReason().ifPresent(reason -> {
            throw new OrchestratorException(reason);
        });
    }

Line commented on: String url = String.format("%s/%s?hostname=%s", ORCHESTRATOR_PATH_PREFIX_HOST_SUSPENSION_API,

Revised code:
public void suspend(String parentHostName, List<String> hostNames) {
        final BatchOperationResult batchOperationResult;
        try {
            String params = String.join("&hostname=", hostNames);
            String url = String.format("%s/%s?hostname=%s", ORCHESTRATOR_PATH_PREFIX_HOST_SUSPENSION_API,
                                       parentHostName, params);
            batchOperationResult = configServerApi.put(url, Optional.empty(), BatchOperationResult.class);
        } catch (HttpException e) {
            throw new OrchestratorException("Failed to batch suspend for " +
                    parentHostName + ": " + e.toString());
        } catch (Exception e) {
            throw new RuntimeException("Got error on batch suspend for " + parentHostName + ", with nodes " + hostNames, e);
        }

        batchOperationResult.getFailureReason().ifPresent(reason -> {
            throw new OrchestratorException(reason);
        });
    }
Code review comment: ```suggestion       // Java uses BigDecimal so 0.2 * 170 = 63.9...       // BigDecimal.longValue() will round down to 63 instead of the expected 64 ```

Code under review:
public void testTrySplitForProcessSplitOnMiddleWindow() throws Exception {
      List<BoundedWindow> windows = ImmutableList.copyOf(currentElement.getWindows());
      OffsetRangeTracker tracker = new OffsetRangeTracker(currentRestriction);
      tracker.tryClaim(30L);
      KV<WindowedSplitResult, Integer> result =
          FnApiDoFnRunner.trySplitForProcess(
              currentElement,
              currentRestriction,
              window2,
              windows,
              currentWatermarkEstimatorState,
              0.2,
              tracker,
              watermarkAndState,
              1,
              3);
      assertEquals(2, (int) result.getValue());
      KV<WindowedValue, WindowedValue> expectedElementSplit =
          createSplitInWindow(new OffsetRange(0, 63), new OffsetRange(63, 100), window2);
      KV<WindowedValue, WindowedValue> expectedWindowSplit =
          createSplitAcrossWindows(ImmutableList.of(window1), ImmutableList.of(window3));
      assertEquals(expectedElementSplit.getKey(), result.getKey().getPrimarySplitRoot());
      assertEquals(expectedElementSplit.getValue(), result.getKey().getResidualSplitRoot());
      assertEquals(
          expectedWindowSplit.getKey(), result.getKey().getPrimaryInFullyProcessedWindowsRoot());
      assertEquals(
          expectedWindowSplit.getValue(), result.getKey().getResidualInUnprocessedWindowsRoot());
    }

Line commented on: createSplitInWindow(new OffsetRange(0, 63), new OffsetRange(63, 100), window2);

Revised code:
public void testTrySplitForProcessSplitOnMiddleWindow() throws Exception {
      List<BoundedWindow> windows = ImmutableList.copyOf(currentElement.getWindows());
      OffsetRangeTracker tracker = new OffsetRangeTracker(currentRestriction);
      tracker.tryClaim(30L);
      KV<WindowedSplitResult, Integer> result =
          FnApiDoFnRunner.trySplitForProcess(
              currentElement,
              currentRestriction,
              window2,
              windows,
              currentWatermarkEstimatorState,
              0.2,
              tracker,
              watermarkAndState,
              1,
              3);
      assertEquals(2, (int) result.getValue());
      
      
      KV<WindowedValue, WindowedValue> expectedElementSplit =
          createSplitInWindow(new OffsetRange(0, 63), new OffsetRange(63, 100), window2);
      KV<WindowedValue, WindowedValue> expectedWindowSplit =
          createSplitAcrossWindows(ImmutableList.of(window1), ImmutableList.of(window3));
      assertEquals(expectedElementSplit.getKey(), result.getKey().getPrimarySplitRoot());
      assertEquals(expectedElementSplit.getValue(), result.getKey().getResidualSplitRoot());
      assertEquals(
          expectedWindowSplit.getKey(), result.getKey().getPrimaryInFullyProcessedWindowsRoot());
      assertEquals(
          expectedWindowSplit.getValue(), result.getKey().getResidualInUnprocessedWindowsRoot());
    }
Code review comment: I'm not 100% why the requirements are being documented in this log.

Code under review:
public void onNext(T value) {
    numberOfMessagesBeforeReadyCheck += 1;
    if (numberOfMessagesBeforeReadyCheck >= maxMessagesBeforeCheck) {
      numberOfMessagesBeforeReadyCheck = 0;
      int waitTime = 1;
      int totalTimeWaited = 0;
      int phase = phaser.getPhase();
      while (!outboundObserver.isReady()) {
        try {
          phaser.awaitAdvanceInterruptibly(phase, waitTime, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
          totalTimeWaited += waitTime;
          waitTime = waitTime * 2;
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
          throw new RuntimeException(e);
        }
      }
      if (totalTimeWaited > 0) {
        
        
        if (phase == phaser.getPhase()) {
          LOGGER.info(
              "Output channel stalled for {}s, outbound thread {}. gRPC requires that outbound "
              + "channel be on a non gRPC inbound channel thread for the channel readiness "
              + "callback to function.",
              totalTimeWaited,
              Thread.currentThread().getName());
        } else {
          LOGGER.debug(
              "Output channel stalled for {}s, outbound thread {}.",
              totalTimeWaited,
              Thread.currentThread().getName());
        }
      }
    }
    synchronized (outboundObserver) {
      outboundObserver.onNext(value);
    }
  }

Line commented on: "Output channel stalled for {}s, outbound thread {}. gRPC requires that outbound "

Revised code:
public void onNext(T value) {
    numberOfMessagesBeforeReadyCheck += 1;
    if (numberOfMessagesBeforeReadyCheck >= maxMessagesBeforeCheck) {
      numberOfMessagesBeforeReadyCheck = 0;
      int waitTime = 1;
      int totalTimeWaited = 0;
      int phase = phaser.getPhase();
      while (!outboundObserver.isReady()) {
        try {
          phaser.awaitAdvanceInterruptibly(phase, waitTime, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
          totalTimeWaited += waitTime;
          waitTime = waitTime * 2;
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
          throw new RuntimeException(e);
        }
      }
      if (totalTimeWaited > 0) {
        
        
        if (phase == phaser.getPhase()) {
          LOGGER.info(
              "Output channel stalled for {}s, outbound thread {}. See: "
              + "https:
              + "this issue.",
              totalTimeWaited,
              Thread.currentThread().getName());
        } else {
          LOGGER.debug(
              "Output channel stalled for {}s, outbound thread {}.",
              totalTimeWaited,
              Thread.currentThread().getName());
        }
      }
    }
    synchronized (outboundObserver) {
      outboundObserver.onNext(value);
    }
  }
Code review comment: Consider rewriting this using `Files` instead of `/bin/ln`

Code under review:
private void createSymlinkToCurrentFile() {
            if (symlinkName == null) return;
            File f = new File(fileName);
            File f2 = new File(f.getParent(), symlinkName);
            String[] cmd = new String[]{"/bin/ln", "-sf", f.getName(), f2.getPath()};
            try {
                int retval = new ProcessExecuter().exec(cmd).getFirst();
                
                
                if (retval != 0) {
                    logger.warning("Command '" + Arrays.toString(cmd) + "' + failed with exitcode=" + retval);
                }
            } catch (IOException e) {
                logger.warning("Got '" + e + "' while doing'" + Arrays.toString(cmd) + "'.");
            }
        }

Line commented on: int retval = new ProcessExecuter().exec(cmd).getFirst();

Revised code:
private void createSymlinkToCurrentFile() {
            if (symlinkName == null) return;
            File f = new File(fileName);
            File f2 = new File(f.getParent(), symlinkName);
            String[] cmd = new String[]{"/bin/ln", "-sf", f.getName(), f2.getPath()};
            try {
                int retval = new ProcessExecuter().exec(cmd).getFirst();
                
                
                if (retval != 0) {
                    logger.warning("Command '" + Arrays.toString(cmd) + "' + failed with exitcode=" + retval);
                }
            } catch (IOException e) {
                logger.warning("Got '" + e + "' while doing'" + Arrays.toString(cmd) + "'.");
            }
        }
Code review comment: Done, if the user limits the count. Otherwise I don't have access to the file path or size and can't infer it. I will file an issue in vert.x.

Code under review:
public void writeResponse(AsyncFile file, Type genericType, ServerRequestContext context) throws WebApplicationException {
        ResteasyReactiveRequestContext ctx = ((ResteasyReactiveRequestContext) context);
        ctx.suspend();
        ServerHttpResponse response = context.serverResponse();
        response.setChunked(true);
        file.handler(buffer -> {
            try {
                response.write(buffer.getBytes());
            } catch (Exception x) {
                
                ctx.resume(x);
                return;
            }
            if (response.isWriteQueueFull()) {
                file.pause();
                response.addDrainHandler(new Runnable() {
                    @Override
                    public void run() {
                        file.resume();
                    }
                });
            }
        });

        file.endHandler(new Handler<Void>() {
            @Override
            public void handle(Void event) {
                response.end();
                
                ctx.resume();
            }
        });
    }

Line commented on: response.setChunked(true);

Revised code:
public void writeResponse(AsyncFile file, Type genericType, ServerRequestContext context) throws WebApplicationException {
        ResteasyReactiveRequestContext ctx = ((ResteasyReactiveRequestContext) context);
        ctx.suspend();
        ServerHttpResponse response = context.serverResponse();
        
        if (file.getReadLength() != Long.MAX_VALUE) {
            response.setResponseHeader(HttpHeaders.CONTENT_LENGTH, String.valueOf(file.getReadLength()));
        } else {
            response.setChunked(true);
        }
        file.handler(buffer -> {
            try {
                response.write(buffer.getBytes());
            } catch (Exception x) {
                
                ctx.resume(x);
                return;
            }
            if (response.isWriteQueueFull()) {
                file.pause();
                response.addDrainHandler(new Runnable() {
                    @Override
                    public void run() {
                        file.resume();
                    }
                });
            }
        });

        file.endHandler(new Handler<Void>() {
            @Override
            public void handle(Void event) {
                file.close();
                response.end();
                
                ctx.resume();
            }
        });
    }
Code review comment: this change should be a separate hotfix

Code under review:
public void testSerializationOfUnknownShuffleDescriptor() throws Exception {
		ShuffleDescriptor shuffleDescriptor = new UnknownShuffleDescriptor(resultPartitionID);
		ShuffleDescriptor shuffleDescriptorCopy = CommonTestUtils.createCopySerializable(shuffleDescriptor);
		assertThat(shuffleDescriptorCopy, instanceOf(UnknownShuffleDescriptor.class));
		assertThat(shuffleDescriptorCopy.getResultPartitionID(), is(resultPartitionID));
		assertThat(shuffleDescriptorCopy.isUnknown(), is(true));
	}

Line commented on: assertThat(shuffleDescriptorCopy.getResultPartitionID(), is(resultPartitionID));

Revised code:
public void testSerializationOfUnknownShuffleDescriptor() throws IOException {
		ShuffleDescriptor shuffleDescriptor = new UnknownShuffleDescriptor(resultPartitionID);
		ShuffleDescriptor shuffleDescriptorCopy = CommonTestUtils.createCopySerializable(shuffleDescriptor);
		assertThat(shuffleDescriptorCopy, instanceOf(UnknownShuffleDescriptor.class));
		assertThat(shuffleDescriptorCopy.getResultPartitionID(), is(resultPartitionID));
		assertThat(shuffleDescriptorCopy.isUnknown(), is(true));
	}
Code review comment: I think the comments I left in the other location setting this could apply everywhere

Code under review:
public Response<PathProperties> readToFileWithResponse(ReadToFileOptions options, Duration timeout, Context context) {
        Context newContext;
        options = options == null ? new ReadToFileOptions() : options;
        if (options.isUpn() != null) {
            HttpHeaders headers = new HttpHeaders();
            headers.set("x-ms-upn", options.isUpn() ? "true" : "false");
            if (context == null) {
                newContext = new Context(AddHeadersFromContextPolicy.AZURE_REQUEST_HTTP_HEADERS_KEY, headers);
            } else {
                newContext = context.addData(AddHeadersFromContextPolicy.AZURE_REQUEST_HTTP_HEADERS_KEY, headers);
            }
        } else {
            newContext = null;
        }

        ReadToFileOptions finalOptions = options;
        return DataLakeImplUtils.returnOrConvertException(() -> {
            Response<BlobProperties> response = blockBlobClient.downloadToFileWithResponse(
                new BlobDownloadToFileOptions(finalOptions.getFilePath())
                    .setRange(Transforms.toBlobRange(finalOptions.getRange()))
                    .setParallelTransferOptions(finalOptions.getParallelTransferOptions())
                    .setDownloadRetryOptions(Transforms.toBlobDownloadRetryOptions(finalOptions.getDownloadRetryOptions()))
                    .setRequestConditions(Transforms.toBlobRequestConditions(finalOptions.getDataLakeRequestConditions()))
                    .setRetrieveContentRangeMd5(finalOptions.isRangeGetContentMd5())
                    .setOpenOptions(finalOptions.getOpenOptions()), timeout, newContext);
            return new SimpleResponse<>(response, Transforms.toPathProperties(response.getValue(), response));
        }, LOGGER);
    }

Line commented on: options = options == null ? new ReadToFileOptions() : options;

Revised code:
public Response<PathProperties> readToFileWithResponse(ReadToFileOptions options, Duration timeout, Context context) {
        context = BuilderHelper.addUpnHeader(() -> (options == null) ? null : options.isUpn(), context);
        Context finalContext = context;

        return DataLakeImplUtils.returnOrConvertException(() -> {
            Response<BlobProperties> response = blockBlobClient.downloadToFileWithResponse(
                new BlobDownloadToFileOptions(options.getFilePath())
                    .setRange(Transforms.toBlobRange(options.getRange()))
                    .setParallelTransferOptions(options.getParallelTransferOptions())
                    .setDownloadRetryOptions(Transforms.toBlobDownloadRetryOptions(options.getDownloadRetryOptions()))
                    .setRequestConditions(Transforms.toBlobRequestConditions(options.getDataLakeRequestConditions()))
                    .setRetrieveContentRangeMd5(options.isRangeGetContentMd5())
                    .setOpenOptions(options.getOpenOptions()), timeout, finalContext);
            return new SimpleResponse<>(response, Transforms.toPathProperties(response.getValue(), response));
        }, LOGGER);
    }
Code review comment: why are we using containerLink and not containerName?

Code under review:
public String toString() {

        if (this.cosmosContainerIdentities == null || cosmosContainerIdentities.isEmpty()) {
            return "";
        }

        
        
        
        return String.format(
                "(containers:%s)(pcrc:%d)(awd:%s)",
                cosmosContainerIdentities
                    .stream()
                    .map(ci -> String.join(
                        ".",
                        containerIdAccessor.getContainerLink(ci)))
                    .collect(Collectors.joining(";")),
                numProactiveConnectionRegions,
                aggressiveWarmupDuration);
    }

Line commented on: containerIdAccessor.getContainerLink(ci)))

Revised code:
public String toString() {

        if (this.cosmosContainerIdentities == null || cosmosContainerIdentities.isEmpty()) {
            return "";
        }

        
        
        
        return String.format(
                "(containers:%s)(pcrc:%d)(awd:%s)",
                cosmosContainerIdentities
                    .stream()
                    .map(ci -> String.join(
                        ".",
                        containerIdAccessor.getContainerLink(ci)))
                    .collect(Collectors.joining(";")),
                numProactiveConnectionRegions,
                aggressiveWarmupDuration);
    }
Code review comment: For the code itself, yes, this PR changed the logic. However, for the internal of Flink, we do not change the behavior, and I think current change is the correct thing.  After each job finished, Flink will clean up job related data (such as checkpoint id counter) on [Dispatcher#removeJob](https://github.com/apache/flink/blob/278504a2787a154faf6f6401028d4bbadafbba0a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L1258), and once the cluster shutdown, it will then clean the cluster related configmap (such as HA related leader) on [ClusterEntrypoint#stopClusterServices](https://github.com/apache/flink/blob/278504a2787a154faf6f6401028d4bbadafbba0a/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java#L502).  For `KubernetesLeaderElectionHaServices`, it should only clean cluster related configs on `#internalCleanup` instead of cleanning the configmaps via labels.

Code under review:
public void internalCleanupJobData(JobID jobID) throws Exception {
        kubeClient.deleteConfigMap(getJobSpecificConfigMap(jobID)).get();
        
    }

Line commented on: kubeClient.deleteConfigMap(getJobSpecificConfigMap(jobID)).get();

Revised code:
public void internalCleanupJobData(JobID jobID) throws Exception {
        kubeClient.deleteConfigMap(getJobSpecificConfigMap(jobID)).get();
        
    }
Code review comment: Shall we order these alphabetically and have the same order as the spec (https://github.com/ballerina-platform/ballerina-spec/commit/158691e90bcc05c2f32137688d7669072fa911c0)? O:)

Code under review:
public void loadPredeclaredModules() {
        Map<Name, BPackageSymbol> modules = new HashMap<>();
        modules.put(Names.ERROR, this.langErrorModuleSymbol);
        modules.put(Names.OBJECT, this.langObjectModuleSymbol);
        modules.put(Names.XML, this.langXmlModuleSymbol);
        modules.put(Names.INT, this.langIntModuleSymbol);
        modules.put(Names.FLOAT, this.langFloatModuleSymbol);
        modules.put(Names.DECIMAL, this.langDecimalModuleSymbol);
        modules.put(Names.BOOLEAN, this.langBooleanModuleSymbol);
        modules.put(Names.FUTURE, this.langFutureModuleSymbol);
        modules.put(Names.MAP, this.langMapModuleSymbol);
        modules.put(Names.STREAM, this.langStreamModuleSymbol);
        modules.put(Names.STRING, this.langStringModuleSymbol);
        modules.put(Names.TABLE, this.langTableModuleSymbol);
        modules.put(Names.TYPEDESC, this.langTypedescModuleSymbol);

        this.predeclaredModules = Collections.unmodifiableMap(modules);
    }

Line commented on: modules.put(Names.TYPEDESC, this.langTypedescModuleSymbol);

Revised code:
public void loadPredeclaredModules() {
        Map<Name, BPackageSymbol> modules = new HashMap<>();
        modules.put(Names.BOOLEAN, this.langBooleanModuleSymbol);
        modules.put(Names.DECIMAL, this.langDecimalModuleSymbol);
        modules.put(Names.ERROR, this.langErrorModuleSymbol);
        modules.put(Names.FLOAT, this.langFloatModuleSymbol);
        modules.put(Names.FUTURE, this.langFutureModuleSymbol);
        modules.put(Names.INT, this.langIntModuleSymbol);
        modules.put(Names.MAP, this.langMapModuleSymbol);
        modules.put(Names.OBJECT, this.langObjectModuleSymbol);
        modules.put(Names.STREAM, this.langStreamModuleSymbol);
        modules.put(Names.STRING, this.langStringModuleSymbol);
        modules.put(Names.TABLE, this.langTableModuleSymbol);
        modules.put(Names.TYPEDESC, this.langTypedescModuleSymbol);
        modules.put(Names.XML, this.langXmlModuleSymbol);

        this.predeclaredModules = Collections.unmodifiableMap(modules);
    }
Code review comment: It turned out map initializer literal could contain string keys. Since both map and record literals are the same, we have to check and act accordingly. Specially when `map m = { foo: "value", "foo": "value" }`. Will update the code to handle to handle previous edge case.

Code under review:
public void visit(BLangRecordLiteral recordLiteral) {
        List<BLangRecordLiteral.BLangRecordKeyValue> keyValuePairs = recordLiteral.keyValuePairs;
        keyValuePairs.forEach(kv -> {
            analyzeExpr(kv.valueExpr);
        });

        Set<String> names = new HashSet<>();
        for (BLangRecordLiteral.BLangRecordKeyValue recFieldDecl : keyValuePairs) {
            BLangExpression keyExpr = recFieldDecl.getKey();
            if (keyExpr.getKind() == NodeKind.SIMPLE_VARIABLE_REF) {
                BLangSimpleVarRef keyRef = (BLangSimpleVarRef) keyExpr;
                if (names.contains(keyRef.variableName.value)) {
                    TypeKind assigneeType = recordLiteral.parent.type.getKind();
                    this.dlog.error(keyExpr.pos, DiagnosticCode.DUPLICATE_RECORD_LITERAL,
                            assigneeType.typeName(), keyRef);
                }
                names.add(keyRef.variableName.value);
            }
        }
    }

Line commented on: if (keyExpr.getKind() == NodeKind.SIMPLE_VARIABLE_REF) {

Revised code:
public void visit(BLangRecordLiteral recordLiteral) {
        List<BLangRecordLiteral.BLangRecordKeyValue> keyValuePairs = recordLiteral.keyValuePairs;
        keyValuePairs.forEach(kv -> {
            analyzeExpr(kv.valueExpr);
        });

        Set<Object> names = new TreeSet<>((l, r) -> l.equals(r) ? 0 : 1);
        for (BLangRecordLiteral.BLangRecordKeyValue recFieldDecl : keyValuePairs) {
            BLangExpression key = recFieldDecl.getKey();
            if (key.getKind() == NodeKind.SIMPLE_VARIABLE_REF) {
                BLangSimpleVarRef keyRef = (BLangSimpleVarRef) key;
                if (names.contains(keyRef.variableName.value)) {
                    String assigneeType = recordLiteral.parent.type.getKind().typeName();
                    this.dlog.error(key.pos, DiagnosticCode.DUPLICATE_KEY_IN_RECORD_LITERAL, assigneeType, keyRef);
                }
                names.add(keyRef.variableName.value);
            } else if (key.getKind() == NodeKind.LITERAL) {
                BLangLiteral keyLiteral = (BLangLiteral) key;
                if (names.contains(keyLiteral.value)) {
                    String assigneeType = recordLiteral.parent.type.getKind().typeName();
                    this.dlog.error(key.pos, DiagnosticCode.DUPLICATE_KEY_IN_RECORD_LITERAL, assigneeType, keyLiteral);
                }
                names.add(keyLiteral.value);
            }
        }
    }
Code review comment: Wondering if this should be somehow connected to `TaskManagerServices#LOCAL_STATE_SUB_DIRECTORY_ROOT`.

Code under review:
private WorkingDirectory(File root) throws IOException {
        this.root = root;
        createDirectory(root);

        this.tmp = new File(root, "tmp");
        createDirectory(tmp);
        FileUtils.cleanDirectory(tmp);

        localState = new File(root, "localState");
        createDirectory(localState);
    }

Line commented on: localState = new File(root, "localState");

Revised code:
private WorkingDirectory(File root) throws IOException {
        this.root = root;
        createDirectory(root);

        this.tmp = new File(root, "tmp");
        createDirectory(tmp);
        FileUtils.cleanDirectory(tmp);

        localState = new File(root, "localState");
        createDirectory(localState);
    }
Code review comment: At some point we should make this a little nice so we can easily add more values.  I am thinking of the following HTTP clients that I personally use:  `httpie`, `IntelliJ HTTP Client` and `Postman` (although I am not sure what values they use for the header - if any)

Code under review:
static String pickFirstSupportedAndAcceptedContentType(RoutingContext context) {
            List<MIMEHeader> acceptableTypes = context.parsedHeaders().accept();

            String userAgent = context.request().getHeader("User-Agent");
            if (userAgent != null && (userAgent.toLowerCase(Locale.ROOT).startsWith("wget/")
                    || userAgent.toLowerCase(Locale.ROOT).startsWith("curl/"))) {
                MIMEHeader result = context.parsedHeaders().findBestUserAcceptedIn(acceptableTypes, SUPPORTED_CURL);
                return result == null ? null : result.value();
            } else {
                MIMEHeader result = context.parsedHeaders().findBestUserAcceptedIn(acceptableTypes, SUPPORTED);
                return result == null ? null : result.value();
            }
        }

Line commented on: || userAgent.toLowerCase(Locale.ROOT).startsWith("curl/"))) {

Revised code:
static String pickFirstSupportedAndAcceptedContentType(RoutingContext context) {
            List<MIMEHeader> acceptableTypes = context.parsedHeaders().accept();

            String userAgent = context.request().getHeader("User-Agent");
            if (userAgent != null && (userAgent.toLowerCase(Locale.ROOT).startsWith("wget/")
                    || userAgent.toLowerCase(Locale.ROOT).startsWith("curl/"))) {
                MIMEHeader result = context.parsedHeaders().findBestUserAcceptedIn(acceptableTypes, SUPPORTED_CURL);
                return result == null ? null : result.value();
            } else {
                MIMEHeader result = context.parsedHeaders().findBestUserAcceptedIn(acceptableTypes, SUPPORTED);
                return result == null ? null : result.value();
            }
        }