muttakin2301/ccr2000 huggingface dataset:
Code review comment: Shouldn't this directory be created even in case the job does not stage any files? That would allow us to delete it. If so, missing files would indicate an error which we might not want to swallow.

Code under review:
public void removeArtifacts(String stagingSessionToken) throws Exception {
    StagingSessionToken parsedToken = StagingSessionToken.decode(stagingSessionToken);
    ResourceId dir = getJobDirResourceId(parsedToken);
    ResourceId manifestResourceId = dir.resolve(MANIFEST, StandardResolveOptions.RESOLVE_FILE);

    LOG.debug("Removing dir {}", dir);

    ProxyManifest proxyManifest =
        BeamFileSystemArtifactRetrievalService.loadManifest(manifestResourceId);
    for (Location location : proxyManifest.getLocationList()) {
      String uri = location.getUri();
      LOG.debug("Removing artifact: {}", uri);
      FileSystems.delete(
          Collections.singletonList(FileSystems.matchNewResource(uri, false /* is directory */)));
    }

    ResourceId artifactsResourceId =
        dir.resolve(ARTIFACTS, StandardResolveOptions.RESOLVE_DIRECTORY);
    LOG.debug("Removing artifacts: {}", artifactsResourceId);
    FileSystems.delete(
        Collections.singletonList(artifactsResourceId), StandardMoveOptions.IGNORE_MISSING_FILES);
    LOG.debug("Removing manifest: {}", manifestResourceId);
    FileSystems.delete(Collections.singletonList(manifestResourceId));
    LOG.debug("Removing empty dir: {}", dir);
    FileSystems.delete(Collections.singletonList(dir));
    LOG.info("Removed dir {}", dir);
  }

Line commented on: Collections.singletonList(artifactsResourceId), StandardMoveOptions.IGNORE_MISSING_FILES);

Revised code:
public void removeArtifacts(String stagingSessionToken) throws Exception {
    StagingSessionToken parsedToken = StagingSessionToken.decode(stagingSessionToken);
    ResourceId dir = getJobDirResourceId(parsedToken);
    ResourceId manifestResourceId = dir.resolve(MANIFEST, StandardResolveOptions.RESOLVE_FILE);

    LOG.debug("Removing dir {}", dir);

    ProxyManifest proxyManifest =
        BeamFileSystemArtifactRetrievalService.loadManifest(manifestResourceId);
    for (Location location : proxyManifest.getLocationList()) {
      String uri = location.getUri();
      LOG.debug("Removing artifact: {}", uri);
      FileSystems.delete(
          Collections.singletonList(FileSystems.matchNewResource(uri, false /* is directory */)));
    }

    ResourceId artifactsResourceId =
        dir.resolve(ARTIFACTS, StandardResolveOptions.RESOLVE_DIRECTORY);
    if (!proxyManifest.getLocationList().isEmpty()) {
      
      LOG.debug("Removing artifacts dir: {}", artifactsResourceId);
      FileSystems.delete(Collections.singletonList(artifactsResourceId));
    }
    LOG.debug("Removing manifest: {}", manifestResourceId);
    FileSystems.delete(Collections.singletonList(manifestResourceId));
    LOG.debug("Removing empty dir: {}", dir);
    FileSystems.delete(Collections.singletonList(dir));
    LOG.info("Removed dir {}", dir);
  }
Code review comment: For status from bitmap to single value,calling ```clear()```  is necessary. ```new a bitmap``` here costs more memory; We just need to keep the status transfer logic correct,NPE can be avoid.

Code under review:
public void clear() {
        this.bitmapType = EMPTY;
        this.bitmap = null;
    }

Line commented on: this.bitmap = null;

Revised code:
public void clear() {
        this.bitmapType = EMPTY;
        this.singleValue = -1;
        this.bitmap = null;
    }
Code review comment: This is a known issue tracked via https://github.com/AzureAD/azure-activedirectory-library-for-java/blob/dev/src/main/java/com/microsoft/aad/adal4j/AuthenticationContext.java.

Code under review:
protected RestClient buildRestClient(AzureTokenCredentials credentials, AzureEnvironment.Endpoint endpoint) {
        RestClient client =  restClientBuilder
                .withBaseUrl(credentials.environment(), endpoint)
                .withCredentials(credentials)
                .withInterceptor(new ProviderRegistrationInterceptor(credentials))
                .build();
        if (client.httpClient().proxy() != null) {
            credentials.withProxy(client.httpClient().proxy());
        }
        return client;
    }

Line commented on: credentials.withProxy(client.httpClient().proxy());

Revised code:
protected RestClient buildRestClient(AzureTokenCredentials credentials, AzureEnvironment.Endpoint endpoint) {
        RestClient client =  restClientBuilder
                .withBaseUrl(credentials.environment(), endpoint)
                .withCredentials(credentials)
                .withInterceptor(new ProviderRegistrationInterceptor(credentials))
                .build();
        if (client.httpClient().proxy() != null) {
            credentials.withProxy(client.httpClient().proxy());
        }
        return client;
    }
Code review comment: ```suggestion         throw new RuntimeException("Do not implement uncheckedCastTo"); ```

Code under review:
protected Expression uncheckedCastTo(DataType targetType) throws AnalysisException {
        throw new  RuntimeException();
    }

Line commented on: throw new  RuntimeException();

Revised code:
protected Expression uncheckedCastTo(DataType targetType) throws AnalysisException {
        throw new RuntimeException("Do not implement uncheckedCastTo");
    }
Code review comment: Makes sense. I'll update the message.

Code under review:
private void checkProperConstructorInProvider(AnnotationInstance i) {
        ClassInfo targetClass = i.target().asClass();
        if (!targetClass.hasNoArgsConstructor()
                || targetClass.methods().stream().filter(m -> m.name().equals("<init>")).count() > 1) {
            LOGGER.warn(
                    "Classes annotated with @Provider should have a single, no-argument constructor. Offending class is "
                            + targetClass);
        }
    }

Line commented on: "Classes annotated with @Provider should have a single, no-argument constructor. Offending class is "

Revised code:
private void checkProperConstructorInProvider(AnnotationInstance i) {
        ClassInfo targetClass = i.target().asClass();
        if (!targetClass.hasNoArgsConstructor()
                || targetClass.methods().stream().filter(m -> m.name().equals("<init>")).count() > 1) {
            LOGGER.warn(
                    "Classes annotated with @Provider should have a single, no-argument constructor, otherwise dependency injection won't work properly. Offending class is "
                            + targetClass);
        }
    }
Code review comment: We potentially could, but I don't know if they are all support by Spring Data anyway. These are the ones that show up often.

Code under review:
protected boolean isIntLongOrBoolean(DotName dotName) {
        return DotNames.BOOLEAN.equals(dotName) || DotNames.PRIMITIVE_BOOLEAN.equals(dotName)
                || DotNames.INTEGER.equals(dotName) || DotNames.PRIMITIVE_INTEGER.equals(dotName)
                || DotNames.LONG.equals(dotName) || DotNames.PRIMITIVE_LONG.equals(dotName);
    }

Line commented on: || DotNames.LONG.equals(dotName) || DotNames.PRIMITIVE_LONG.equals(dotName);

Revised code:
protected boolean isIntLongOrBoolean(DotName dotName) {
        return DotNames.BOOLEAN.equals(dotName) || DotNames.PRIMITIVE_BOOLEAN.equals(dotName)
                || DotNames.INTEGER.equals(dotName) || DotNames.PRIMITIVE_INTEGER.equals(dotName)
                || DotNames.LONG.equals(dotName) || DotNames.PRIMITIVE_LONG.equals(dotName);
    }
Code review comment: I think this test is there only for investigation. I can convert it to try-with-resources but would keep it disabled. @cescoffier agree with this approach ?

Code under review:
void test() throws InterruptedException {
        MongoClient mongoClient = MongoClients.create("mongodb:
        MongoDatabase database = mongoClient.getDatabase("test");
        MongoCollection<Document> collection = database.getCollection("restaurants");
        new Thread(() -> collection.watch().forEach(printBlock)).start();
        Document doc = new Document().append("hello", "world");
        collection.insertOne(doc);
        collection.updateOne(eq("hello", "world"), new Document("$set", new Document().append("hello", "foo")));
        new Thread(() -> collection.watch().forEach(printBlock)).start();
        Thread.sleep(5000);
        mongoClient.close();
    }

Line commented on: mongoClient.close();

Revised code:
void test() throws InterruptedException {
        MongoClient mongoClient = MongoClients.create("mongodb:
        MongoDatabase database = mongoClient.getDatabase("test");
        MongoCollection<Document> collection = database.getCollection("restaurants");
        new Thread(() -> collection.watch().forEach(printBlock)).start();
        Document doc = new Document().append("hello", "world");
        collection.insertOne(doc);
        collection.updateOne(eq("hello", "world"), new Document("$set", new Document().append("hello", "foo")));
        new Thread(() -> collection.watch().forEach(printBlock)).start();
        Thread.sleep(5000);
    }
Code review comment: `convertValue()` will return `null` with no exceptions. So, no additional check needed.

Code under review:
public ChangeFeedMetaData getChangeFeedMetaData() {

        if (this.changeFeedProcessorItemAsJsonNode == null) {
            this.changeFeedProcessorItemAsJsonNode = constructChangeFeedProcessorItemAsJsonNode();
        }

        if (this.changeFeedMetaDataInternal == null) {
            this.changeFeedMetaDataInternal = Utils.getSimpleObjectMapper().convertValue(this.changeFeedMetaData, ChangeFeedMetaData.class);
        }

        return this.changeFeedMetaDataInternal;
    }

Line commented on: this.changeFeedMetaDataInternal = Utils.getSimpleObjectMapper().convertValue(this.changeFeedMetaData, ChangeFeedMetaData.class);

Revised code:
public ChangeFeedMetaData getChangeFeedMetaData() {

        if (this.changeFeedMetaDataInternal == null) {
            this.changeFeedMetaDataInternal = Utils.getSimpleObjectMapper().convertValue(this.changeFeedMetaData, ChangeFeedMetaData.class);
        }

        return this.changeFeedMetaDataInternal;
    }
Code review comment: why are we now assigning to local variable? Why were we using private field before?

Code under review:
public Row deserialize(byte[] message) throws IOException {
		try {
			inputStream.setBuffer(message);
			final IndexedRecord read = datumReader.read(record, decoder);
			return convertRecord(schema, (RowTypeInfo) typeInfo, read);
		} catch (Exception e) {
			throw new IOException("Failed to deserialize Avro record.", e);
		}
	}

Line commented on: final IndexedRecord read = datumReader.read(record, decoder);

Revised code:
public Row deserialize(byte[] message) throws IOException {
		try {
			inputStream.setBuffer(message);
			record = datumReader.read(record, decoder);
			return convertAvroRecordToRow(schema, typeInfo, record);
		} catch (Exception e) {
			throw new IOException("Failed to deserialize Avro record.", e);
		}
	}
Code review comment: wider type is nessesary, if there is some bug, you should fix it, but not delete this widerType

Code under review:
public If(Expression arg0, Expression arg1, Expression arg2) {
        super("if", arg0, arg1, arg2);
    }

Line commented on: super("if", arg0, arg1, arg2);

Revised code:
public If(Expression arg0, Expression arg1, Expression arg2) {
        super("if", arg0, arg1, arg2);
    }
Code review comment: @fsk119  Here I made some modifications. Previously it was: If it is fixArgumentType, then return argumentType. In fact, there is a problem. We should only return argumentType if it is a DEFAULT node. Other nodes should still return the actual operand's type.

Code under review:
public RelDataType getOperandType(int ordinal) {
        if (!isFixedParameters()) {
            return super.getOperandType(ordinal);
        }

        SqlNode operand = rewrittenOperands.get(ordinal);
        if (operand.getKind() == SqlKind.DEFAULT) {
            return fixArgumentTypes.get(ordinal);
        }

        final RelDataType type = SqlTypeUtil.deriveType(this, operand);
        final SqlValidatorNamespace namespace = getValidator().getNamespace(operand);
        return namespace != null ? namespace.getType() : type;
    }

Line commented on: final RelDataType type = SqlTypeUtil.deriveType(this, operand);

Revised code:
public RelDataType getOperandType(int ordinal) {
        if (!isFixedParameters()) {
            return super.getOperandType(ordinal);
        }

        SqlNode operand = rewrittenOperands.get(ordinal);
        if (operand.getKind() == SqlKind.DEFAULT) {
            return fixArgumentTypes.get(ordinal);
        }

        final RelDataType type = SqlTypeUtil.deriveType(this, operand);
        final SqlValidatorNamespace namespace = getValidator().getNamespace(operand);
        return namespace != null ? namespace.getType() : type;
    }
Code review comment: Could we add the `forEach()` onto the util call to save a variable allocation?

Code under review:
public static Resource mapResourceAttributes(List<String> resourceAttributes) {
        AttributesBuilder attributesBuilder = Attributes.builder();

        Map<String, String> resourceAttributesMap = OpenTelemetryUtil.convertKeyValueListToMap(resourceAttributes);
        resourceAttributesMap.forEach(attributesBuilder::put);

        return Resource.create(attributesBuilder.build());
    }

Line commented on: Map<String, String> resourceAttributesMap = OpenTelemetryUtil.convertKeyValueListToMap(resourceAttributes);

Revised code:
public static Resource mapResourceAttributes(List<String> resourceAttributes) {
        AttributesBuilder attributesBuilder = Attributes.builder();

        OpenTelemetryUtil.convertKeyValueListToMap(resourceAttributes).forEach(attributesBuilder::put);

        return Resource.create(attributesBuilder.build());
    }
Code review comment: Yeah, tested it by using the same flux to read multiple times. Same data was returned each time.

Code under review:
public Mono<Void> runAsync() {
        return blobAsyncClient.upload(randomByteBufferFlux, null, true).then();
    }

Line commented on: return blobAsyncClient.upload(randomByteBufferFlux, null, true).then();

Revised code:
public Mono<Void> runAsync() {
        return blobAsyncClient.upload(randomByteBufferFlux, null, true).then();
    }
Code review comment: Shall we check  ```suggestion         if (expType.tag == TypeTags.ARRAY) { ``` instead?

Code under review:
public void visit(BLangQueryExpr queryExpr) {
        List<? extends FromClauseNode> fromClauseList = queryExpr.getFromClauseNodes();
        List<? extends WhereClauseNode> whereClauseList = queryExpr.getWhereClauseNode();
        SymbolEnv parentEnv = env;
        for (FromClauseNode fromClause : fromClauseList) {
            parentEnv = typeCheckFromClause((BLangFromClause) fromClause, parentEnv);
        }
        BLangSelectClause selectClause = (BLangSelectClause) queryExpr.getSelectClauseNode();
        SymbolEnv whereEnv = parentEnv;
        for (WhereClauseNode whereClauseNode : whereClauseList) {
            whereEnv = typeCheckWhereClause((BLangWhereClause) whereClauseNode, selectClause, parentEnv);
        }

        if (queryExpr.expectedType.tag != symTable.noType.tag) {
            BType expectedType = ((BArrayType) queryExpr.expectedType).eType;
            checkExpr(selectClause.expression, whereEnv, expectedType);
            resultType = queryExpr.expectedType;
        } else {
            checkExpr(selectClause.expression, whereEnv);
        }
    }

Line commented on: if (queryExpr.expectedType.tag != symTable.noType.tag) {

Revised code:
public void visit(BLangQueryExpr queryExpr) {
        List<? extends FromClauseNode> fromClauseList = queryExpr.getFromClauseNodes();
        List<? extends WhereClauseNode> whereClauseList = queryExpr.getWhereClauseNode();
        SymbolEnv parentEnv = env;
        for (FromClauseNode fromClause : fromClauseList) {
            parentEnv = typeCheckFromClause((BLangFromClause) fromClause, parentEnv);
        }
        BLangSelectClause selectClause = (BLangSelectClause) queryExpr.getSelectClauseNode();
        SymbolEnv whereEnv = parentEnv;
        for (WhereClauseNode whereClauseNode : whereClauseList) {
            whereEnv = typeCheckWhereClause((BLangWhereClause) whereClauseNode, selectClause, parentEnv);
        }

        if (expType.tag != TypeTags.NONE) {
            BType expectedType = ((BArrayType) queryExpr.expectedType).eType;
            BType type = checkExpr(selectClause.expression, whereEnv, expectedType);
            if (type == symTable.semanticError) {
                resultType = type;
            } else {
                resultType = new BArrayType(expectedType);
            }
        } else {
            checkExpr(selectClause.expression, whereEnv);
        }
    }
Code review comment: ? Mailbox executor should be already constructed from `ContinuousFileReaderOperatorFactory`

Code under review:
public void open() throws Exception {
		super.open();

		checkState(this.serializer != null, "The serializer has not been set. " +
			"Probably the setOutputType() was not called. Please report it.");

		this.state = ReaderState.IDLE;
		this.format.setRuntimeContext(getRuntimeContext());
		this.format.configure(new Configuration());

		this.sourceContext = StreamSourceContexts.getSourceContext(
				getOperatorConfig().getTimeCharacteristic(),
				getProcessingTimeService(),
				new Object(), 
				getContainingTask().getStreamStatusMaintainer(),
				output,
				getRuntimeContext().getExecutionConfig().getAutoWatermarkInterval(),
				-1);

		this.reusedRecord = serializer.createInstance();
		this.completedSplitsCounter = getMetricGroup().counter("numSplitsProcessed");
		this.executor = executor != null ? executor : getContainingTask().getMailboxExecutorFactory().createExecutor(getOperatorConfig().getChainIndex());
		this.splits = this.splits == null ? new PriorityQueue<>() : this.splits;

		if (!splits.isEmpty()) {
			enqueueMail();
		}
	}

Line commented on: this.executor = executor != null ? executor : getContainingTask().getMailboxExecutorFactory().createExecutor(getOperatorConfig().getChainIndex());

Revised code:
public void open() throws Exception {
		super.open();

		checkState(this.serializer != null, "The serializer has not been set. " +
			"Probably the setOutputType() was not called. Please report it.");

		this.state = ReaderState.IDLE;
		this.format.setRuntimeContext(getRuntimeContext());
		this.format.configure(new Configuration());

		this.sourceContext = StreamSourceContexts.getSourceContext(
			getOperatorConfig().getTimeCharacteristic(),
			getProcessingTimeService(),
			new Object(), 
			getContainingTask().getStreamStatusMaintainer(),
			output,
			getRuntimeContext().getExecutionConfig().getAutoWatermarkInterval(),
			-1);

		this.reusedRecord = serializer.createInstance();
		this.completedSplitsCounter = getMetricGroup().counter("numSplitsProcessed");
		this.executor = executor != null ? executor : getContainingTask().getMailboxExecutorFactory().createExecutor(getOperatorConfig().getChainIndex());
		this.splits = this.splits == null ? new PriorityQueue<>() : this.splits;

		if (!splits.isEmpty()) {
			enqueueProcessRecord();
		}
	}
Code review comment: That has been a major problem in throughput control (that we did not add validation)

Code under review:
private IMetadataReader getMetadataReader() {
        switch (this.config.getMetadataConfig().getStorageType()) {
            case KAFKA:
                return this.kafkaOffsetStorageReader;
            case COSMOS:
                CosmosAsyncContainer metadataContainer =
                    this.cosmosClient
                        .getDatabase(this.config.getContainersConfig().getDatabaseName())
                        .getContainer(this.config.getMetadataConfig().getStorageName());
                return new MetadataCosmosStorageManager(metadataContainer);
            default:
                throw new IllegalArgumentException("Metadata storage type " + this.config.getMetadataConfig().getStorageType() + " is not supported");
        }
    }

Line commented on: CosmosAsyncContainer metadataContainer =

Revised code:
private IMetadataReader getMetadataReader() {
        switch (this.config.getMetadataConfig().getStorageType()) {
            case KAFKA:
                return this.kafkaOffsetStorageReader;
            case COSMOS:
                CosmosAsyncContainer metadataContainer =
                    this.cosmosClient
                        .getDatabase(this.config.getContainersConfig().getDatabaseName())
                        .getContainer(this.config.getMetadataConfig().getStorageName());
                
                
                metadataContainer.read()
                    .doOnNext(containerResponse -> {
                        PartitionKeyDefinition partitionKeyDefinition = containerResponse.getProperties().getPartitionKeyDefinition();
                        if (partitionKeyDefinition.getPaths().size() != 1 || !partitionKeyDefinition.getPaths().get(0).equals("/id")) {
                            throw new IllegalStateException("Cosmos Metadata container need to be partitioned by /id");
                        }
                    })
                    .block();
                return new MetadataCosmosStorageManager(metadataContainer);
            default:
                throw new IllegalArgumentException("Metadata storage type " + this.config.getMetadataConfig().getStorageType() + " is not supported");
        }
    }
Code review comment: nitpicking: `currentDevModeNewThreadCreationClassLoader` says: > This must be accessed under the {@link #devModeThreads} lock  Yet, I don't see a lock/syncronized here, probably because it's not needed because there is no concurrency here.

Code under review:
public ThreadFactory createThreadFactory(LaunchMode launchMode) {
        Optional<ClassLoader> nonDevModeTccl;
        if (launchMode == LaunchMode.DEVELOPMENT) {
            currentDevModeNewThreadCreationClassLoader = Thread.currentThread().getContextClassLoader();
            nonDevModeTccl = Optional.empty(); 
        } else {
            nonDevModeTccl = Optional.of(Thread.currentThread().getContextClassLoader());
        }
        AtomicInteger threadCount = new AtomicInteger(0);
        return runnable -> {
            VertxThread thread = VertxThreadFactory.INSTANCE.newVertxThread(runnable,
                    "executor-thread-" + threadCount.getAndIncrement(), true, 0, null);
            thread.setDaemon(true);
            if (launchMode == LaunchMode.DEVELOPMENT) {
                synchronized (devModeThreads) {
                    setNewThreadTccl(thread);
                    devModeThreads.add(thread);
                }
            } else {
                thread.setContextClassLoader(nonDevModeTccl.get());
            }
            return thread;
        };
    }

Line commented on: currentDevModeNewThreadCreationClassLoader = Thread.currentThread().getContextClassLoader();

Revised code:
public ThreadFactory createThreadFactory(LaunchMode launchMode) {
        Optional<ClassLoader> nonDevModeTccl = setupThreadFactoryTccl(launchMode);
        AtomicInteger threadCount = new AtomicInteger(0);
        return runnable -> {
            VertxThread thread = createVertxThread(runnable,
                    "executor-thread-" + threadCount.getAndIncrement(), true, 0, null, launchMode, nonDevModeTccl);
            thread.setDaemon(true);
            return thread;
        };
    }
Code review comment: Up to you if you want to go to this depth though since it could be a lot of work and can always be improved upon later.

Code under review:
public void encode(T value, OutputStream os) throws IOException {
    ZstdCompressCtx ctx = new ZstdCompressCtx();
    try {
      ctx.setLevel(level);
      ctx.setMagicless(true); 
      ctx.setDictID(false); 
      ctx.loadDict(dict);

      byte[] encoded = CoderUtils.encodeToByteArray(innerCoder, value);
      ctx.setPledgedSrcSize(encoded.length);
      byte[] compressed = ctx.compress(encoded);
      ByteArrayCoder.of().encode(compressed, os);
    } finally {
      ctx.close();
    }
  }

Line commented on: byte[] encoded = CoderUtils.encodeToByteArray(innerCoder, value);

Revised code:
public void encode(T value, OutputStream os) throws IOException {
    ZstdCompressCtx ctx = new ZstdCompressCtx();
    try {
      ctx.setLevel(level);
      ctx.setMagicless(true); 
      ctx.setDictID(false); 
      ctx.loadDict(dict);

      byte[] encoded = CoderUtils.encodeToByteArray(innerCoder, value);
      byte[] compressed = ctx.compress(encoded);
      ByteArrayCoder.of().encode(compressed, os);
    } finally {
      ctx.close();
    }
  }
Code review comment: We all agree that this configuration is fixed at build time and cannot be changed later? Otherwise, it won't work that well.

Code under review:
private void registerExceptionHandler(BuildProducer<ReflectiveClassBuildItem> reflectiveClasses) {
        Properties properties = buildKafkaStreamsProperties();
        String exceptionHandlerClassName = properties
                .getProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG);

        if (exceptionHandlerClassName == null)
            registerDefaultExceptionHandler(reflectiveClasses);
        else
            registerClassName(reflectiveClasses, exceptionHandlerClassName);
    }

Line commented on: registerClassName(reflectiveClasses, exceptionHandlerClassName);

Revised code:
private void registerExceptionHandler(BuildProducer<ReflectiveClassBuildItem> reflectiveClasses) {
        Properties properties = buildKafkaStreamsProperties();
        String exceptionHandlerClassName = properties
                .getProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG);

        if (exceptionHandlerClassName == null) {
            registerDefaultExceptionHandler(reflectiveClasses);
        } else {
            registerClassName(reflectiveClasses, exceptionHandlerClassName);
        }
    }
Code review comment: Sounds like imperfect unit tests. Is the delay a little short ?

Code under review:
private void run() {
        try {
            
            Thread.sleep(1000);
            while (true) {
                maintenance();
                waitForTrigger(2000);
            }
        } catch (Exception e) {
            
            System.err.println("Fatal exception in FilesArchived-maintainer thread: "+e);
        }
    }

Line commented on: Thread.sleep(1000);

Revised code:
private void run() {
        try {
            
            Thread.sleep(1000);
            while (true) {
                maintenance();
                waitForTrigger(2000);
            }
        } catch (Exception e) {
            
            System.err.println("Fatal exception in FilesArchived-maintainer thread: "+e);
        }
    }
Code review comment: I believe these conditions are the same? The only JobType we ever trigger, which does not have a zone, is the component job, unless we are doing something very wrong.

Code under review:
private Version deployVersionFor(DeploymentJobs.JobType jobType, Controller controller) {
        return jobType == JobType.component
               ? controller.systemVersion()
               : deployVersionIn(jobType.zone(controller.system()).get(), controller);
    }

Line commented on: : deployVersionIn(jobType.zone(controller.system()).get(), controller);

Revised code:
private Version deployVersionFor(DeploymentJobs.JobType jobType, Controller controller) {
        return jobType == JobType.component
               ? controller.systemVersion()
               : deployVersionIn(jobType.zone(controller.system()).get(), controller);
    }
Code review comment: I could do: ```         this.formatType =                 ConfigurationUtils.convertValue(                         line.getOptionValue(                                 SAVEPOINT_FORMAT_OPTION, SavepointFormatType.DEFAULT.toString()),                         SavepointFormatType.class); ```  Do you think that would be better? I would not like to extend the `convertValue` to handle default values or treat `null` specially. The default value for `ConfigOption(s)` is handled on a higher level.

Code under review:
public CancelOptions(CommandLine line) {
        super(line);
        this.args = line.getArgs();
        this.withSavepoint = line.hasOption(CANCEL_WITH_SAVEPOINT_OPTION.getOpt());
        this.targetDirectory = line.getOptionValue(CANCEL_WITH_SAVEPOINT_OPTION.getOpt());
        if (line.hasOption(SAVEPOINT_FORMAT_OPTION)) {
            formatType =
                    ConfigurationUtils.convertValue(
                            line.getOptionValue(SAVEPOINT_FORMAT_OPTION),
                            SavepointFormatType.class);
        } else {
            formatType = SavepointFormatType.DEFAULT;
        }
    }

Line commented on: formatType = SavepointFormatType.DEFAULT;

Revised code:
public CancelOptions(CommandLine line) {
        super(line);
        this.args = line.getArgs();
        this.withSavepoint = line.hasOption(CANCEL_WITH_SAVEPOINT_OPTION.getOpt());
        this.targetDirectory = line.getOptionValue(CANCEL_WITH_SAVEPOINT_OPTION.getOpt());
        if (line.hasOption(SAVEPOINT_FORMAT_OPTION)) {
            formatType =
                    ConfigurationUtils.convertValue(
                            line.getOptionValue(SAVEPOINT_FORMAT_OPTION),
                            SavepointFormatType.class);
        } else {
            formatType = SavepointFormatType.DEFAULT;
        }
    }
Code review comment: Avoid using Lists.newArrayList(), just using new ArrayList<>() here

Code under review:
public void assertAppendValueWhenParametersIsEmpty() {
        Collection assignments = Lists.newArrayList();
        List parameters = Lists.newArrayList();
        InsertValueContext insertValueContext = new InsertValueContext(assignments,parameters, 0);

        Object value = "test";
        String type = "String";
        insertValueContext.appendValue(value, type);

        List<ExpressionSegment> valueExpressions = insertValueContext.getValueExpressions();
        assertThat(valueExpressions.size(), is(1));
        DerivedLiteralExpressionSegment segmentInInsertValueContext = (DerivedLiteralExpressionSegment) valueExpressions.get(0);
        assertThat(segmentInInsertValueContext.getType(), is(type));
        assertThat(segmentInInsertValueContext.getLiterals(), is(value));
    }

Line commented on: Collection assignments = Lists.newArrayList();

Revised code:
public void assertAppendValueWhenParametersIsEmpty() {
        Collection<ExpressionSegment> assignments = Collections.emptyList();
        List<Object> parameters = Collections.emptyList();
        InsertValueContext insertValueContext = new InsertValueContext(assignments,parameters, 0);
        Object value = "test";
        String type = "String";
        insertValueContext.appendValue(value, type);
        List<ExpressionSegment> valueExpressions = insertValueContext.getValueExpressions();
        assertThat(valueExpressions.size(), is(1));
        DerivedLiteralExpressionSegment segmentInInsertValueContext = (DerivedLiteralExpressionSegment) valueExpressions.get(0);
        assertThat(segmentInInsertValueContext.getType(), is(type));
        assertThat(segmentInInsertValueContext.getLiterals(), is(value));
    }
Code review comment: Won't this always be true if deployTime is present? ```suggestion             if (deployTime.isPresent() && Instant.now().isBefore(deployTime.get().plus(Duration.ofMinutes(1)))) ```

Code under review:
public HttpResponse getLogs(String logServerUri, Optional<Instant> deployTime) {
        HttpGet get = new HttpGet(logServerUri);
        try {
            return new ProxyResponse(httpClient.execute(get));
        } catch (IOException e) {
            
            
            if (deployTime.isPresent() && Instant.now().isAfter(deployTime.get().minus(Duration.ofMinutes(1))))
                return new HttpResponse(200) {
                    @Override
                    public void render(OutputStream outputStream) throws IOException {
                        outputStream.write("".getBytes(StandardCharsets.UTF_8));
                    }
                };
            return HttpErrorResponse.internalServerError("Failed to get logs: " + Exceptions.toMessageString(e));
        }
    }

Line commented on: if (deployTime.isPresent() && Instant.now().isAfter(deployTime.get().minus(Duration.ofMinutes(1))))

Revised code:
public HttpResponse getLogs(String logServerUri, Optional<Instant> deployTime) {
        HttpGet get = new HttpGet(logServerUri);
        try {
            return new ProxyResponse(httpClient.execute(get));
        } catch (IOException e) {
            
            
            if (deployTime.isPresent() && Instant.now().isBefore(deployTime.get().plus(Duration.ofMinutes(1))))
                return new HttpResponse(200) {
                    @Override
                    public void render(OutputStream outputStream) throws IOException {
                        outputStream.write("".getBytes(StandardCharsets.UTF_8));
                    }
                };
            return HttpErrorResponse.internalServerError("Failed to get logs: " + Exceptions.toMessageString(e));
        }
    }
Code review comment: Oh sorry, now I understand your point, yeah I will move this to `init`

Code under review:
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
        String metricsPath = getInitParameter("metrics.path");
        String contextPath = (appContextPath.equals("/") ? "" : appContextPath) + metricsPath;
        String requestPath = request.getRequestURI();
        String method = request.getMethod();
        Stream<String> acceptHeaders = Collections.list(request.getHeaders("Accept")).stream();

        metricsHandler.handleRequest(requestPath, contextPath, method, acceptHeaders, (status, message, headers) -> {
            headers.forEach(response::addHeader);
            response.setStatus(status);
            response.getWriter().write(message);
        });
    }

Line commented on: String metricsPath = getInitParameter("metrics.path");

Revised code:
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
        String requestPath = request.getRequestURI();
        String method = request.getMethod();
        Stream<String> acceptHeaders = Collections.list(request.getHeaders("Accept")).stream();

        metricsHandler.handleRequest(requestPath, contextPath, method, acceptHeaders, (status, message, headers) -> {
            headers.forEach(response::addHeader);
            response.setStatus(status);
            response.getWriter().write(message);
        });
    }
Code review comment: Shall we also update the comment?

Code under review:
private BType determineRawTemplateLiteralType(BLangRawTemplateLiteral rawTemplateLiteral, BType expType) {
        
        
        if (expType == symTable.noType || containsAnyType(expType)) {
            return symTable.rawTemplateType;
        }

        BType type = types.checkType(rawTemplateLiteral, expType, symTable.rawTemplateType,
                                     DiagnosticCode.INVALID_RAW_TEMPLATE_TYPE);

        if (type == symTable.semanticError) {
            return type;
        }

        
        if (!Symbols.isFlagOn(type.tsymbol.flags, Flags.ABSTRACT)) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.INVALID_RAW_TEMPLATE_ASSIGNMENT, type);
            return symTable.semanticError;
        }

        
        BObjectType litObjType = (BObjectType) type;
        BObjectTypeSymbol objTSymbol = (BObjectTypeSymbol) litObjType.tsymbol;

        if (litObjType.fields.size() > 2) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.INVALID_NUM_FIELDS, litObjType);
            type = symTable.semanticError;
        }

        if (!objTSymbol.attachedFuncs.isEmpty()) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.METHODS_NOT_ALLOWED, litObjType);
            type = symTable.semanticError;
        }

        return type;
    }

Line commented on: if (expType == symTable.noType || containsAnyType(expType)) {

Revised code:
private BType determineRawTemplateLiteralType(BLangRawTemplateLiteral rawTemplateLiteral, BType expType) {
        
        
        if (expType == symTable.noType || containsAnyType(expType)) {
            return symTable.rawTemplateType;
        }

        BType compatibleType = getCompatibleRawTemplateType(expType, rawTemplateLiteral.pos);
        BType type = types.checkType(rawTemplateLiteral, compatibleType, symTable.rawTemplateType,
                                     DiagnosticCode.INVALID_RAW_TEMPLATE_TYPE);

        if (type == symTable.semanticError) {
            return type;
        }

        
        if (!Symbols.isFlagOn(type.tsymbol.flags, Flags.ABSTRACT)) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.INVALID_RAW_TEMPLATE_ASSIGNMENT, type);
            return symTable.semanticError;
        }

        
        BObjectType litObjType = (BObjectType) type;
        BObjectTypeSymbol objTSymbol = (BObjectTypeSymbol) litObjType.tsymbol;

        if (litObjType.fields.size() > 2) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.INVALID_NUM_FIELDS, litObjType);
            type = symTable.semanticError;
        }

        if (!objTSymbol.attachedFuncs.isEmpty()) {
            dlog.error(rawTemplateLiteral.pos, DiagnosticCode.METHODS_NOT_ALLOWED, litObjType);
            type = symTable.semanticError;
        }

        return type;
    }
Code review comment: We should have been able to schedule the async API on a single thread, and forced them to run sequentially; however, I didn't have success with that. I am blocking the async API call for now, until I figure that out.

Code under review:
public static void deleteAllModels() throws InterruptedException {
        System.out.println("DELETING MODELS");

        
        List<String> models = asList(RoomModelId, WifiModelId, BuildingModelId, FloorModelId, HvacModelId);

        
        
        models
            .forEach(modelId -> {
                try {
                    client.deleteModel(modelId).block();
                    System.out.println("Deleted model: " + modelId);
                } catch (ErrorResponseException ex) {
                    if (ex.getResponse().getStatusCode() != HttpStatus.SC_NOT_FOUND) {
                        System.err.println("Could not delete model " + modelId + " due to " + ex);
                    }
                }
            });
    }


Revised code:
public static void deleteAllModels() throws InterruptedException {
        System.out.println("DELETING MODELS");

        
        List<String> models = asList(RoomModelId, WifiModelId, BuildingModelId, FloorModelId, HvacModelId);

        
        
        models
            .forEach(modelId -> {
                try {
                    client.deleteModel(modelId).block();
                    System.out.println("Deleted model: " + modelId);
                } catch (ErrorResponseException ex) {
                    if (ex.getResponse().getStatusCode() != HttpStatus.SC_NOT_FOUND) {
                        System.err.println("Could not delete model " + modelId + " due to " + ex);
                    }
                }
            });
    }
Code review comment: Note really, the `qualifiers` set is always null for default qualifiers and `CollectionHelpers.toImmutableSmallSet()` returns `null` for `null` param.

Code under review:
public RemovedBeanImpl(Kind kind, String description, Set<Type> types, Set<Annotation> qualifiers) {
        this.kind = kind;
        this.description = description;
        this.types = CollectionHelpers.toImmutableSmallSet(types);
        this.qualifiers = CollectionHelpers.toImmutableSmallSet(qualifiers);
    }

Line commented on: this.qualifiers = CollectionHelpers.toImmutableSmallSet(qualifiers);

Revised code:
public RemovedBeanImpl(Kind kind, String description, Set<Type> types, Set<Annotation> qualifiers) {
        this.kind = kind;
        this.description = description;
        this.types = CollectionHelpers.toImmutableSmallSet(types);
        this.qualifiers = CollectionHelpers.toImmutableSmallSet(qualifiers);
    }
Code review comment: Probably want to skip all of this if target is empty?

Code under review:
private void autoscale(ApplicationId application, List<Node> applicationNodes) {
        try (MaintenanceDeployment deployment = new MaintenanceDeployment(application, deployer, nodeRepository())) {
            if ( ! deployment.isValid()) return; 
            nodesByCluster(applicationNodes).forEach((clusterId, clusterNodes) -> {
                Optional<AllocatableClusterResources> target = autoscaler.autoscale(clusterNodes);

                Instant lastLogTime = lastLogged.get(new Pair<>(application, clusterId));
                if (lastLogTime == null || lastLogTime.isBefore(nodeRepository().clock().instant().minus(Duration.ofHours(1)))) {
                    int currentGroups = (int) clusterNodes.stream().map(node -> node.allocation().get().membership().cluster().group()).distinct().count();
                    ClusterSpec.Type clusterType = clusterNodes.get(0).allocation().get().membership().cluster().type();
                    target.ifPresent(t -> log.info("Autoscale: " + application + clusterType + " " + clusterId +
                                                   " from " + toString(clusterNodes.size(), currentGroups, clusterNodes.get(0).flavor().resources()) +
                                                   " to " + toString(t.nodes(), t.groups(), t.advertisedResources())));
                    lastLogged.put(new Pair<>(application, clusterId), nodeRepository().clock().instant());
                }
            });
        }
    }

Line commented on: if (lastLogTime == null || lastLogTime.isBefore(nodeRepository().clock().instant().minus(Duration.ofHours(1)))) {

Revised code:
private void autoscale(ApplicationId application, List<Node> applicationNodes) {
        try (MaintenanceDeployment deployment = new MaintenanceDeployment(application, deployer, nodeRepository())) {
            if ( ! deployment.isValid()) return; 
            nodesByCluster(applicationNodes).forEach((clusterId, clusterNodes) -> autoscale(application, clusterId, clusterNodes));
        }
    }
Code review comment: I don't know the reason, but this test was designed to prevent it from being launched in native mode. That's why the presence of both `@SubstrateTest` and `@NativeImageTest` is checked here. It should probably stay that way until `@SubstrateTest` is removed.

Code under review:
public void testInjection() {
        Assumptions.assumeFalse(getClass().isAnnotationPresent(NativeImageTest.class));
        Assert.assertTrue(checkks.call().getState() == HealthCheckResponse.State.UP);
    }

Line commented on: Assert.assertTrue(checkks.call().getState() == HealthCheckResponse.State.UP);

Revised code:
public void testInjection() {
        Assert.assertTrue(simpleHealthCheck.call().getState() == HealthCheckResponse.State.UP);
    }
Code review comment: got your point，use: ``` for (int i = 0; i < jsonFieldMapping.length; i++) { 	returnRecord.setField(jsonFieldMapping[i], jsonRow.getField(i)); } ```

Code under review:
public RowData readRecord(RowData reuse, byte[] bytes, int offset, int numBytes) throws IOException {
			GenericRowData returnRecord = rowData;

			if (this.getDelimiter() != null && this.getDelimiter().length == 1
				&& this.getDelimiter()[0] == NEW_LINE && offset + numBytes >= 1
				&& bytes[offset + numBytes - 1] == CARRIAGE_RETURN){
				numBytes -= 1;
			}
			byte[] trimBytes = Arrays.copyOfRange(bytes, offset, offset + numBytes);
			GenericRowData jsonRow = (GenericRowData) deserializationSchema.deserialize(trimBytes);
			if (jsonRow != null) {
				for (int i = 0; i < jsonSelectFieldNames.size(); i++) {
					returnRecord.setField(jsonFieldMapping[i], jsonRow.getField(i));
				}
			}
			emitted++;
			return returnRecord;
		}

Line commented on: returnRecord.setField(jsonFieldMapping[i], jsonRow.getField(i));

Revised code:
public RowData readRecord(RowData reuse, byte[] bytes, int offset, int numBytes) throws IOException {
			
			if (this.getDelimiter() != null && this.getDelimiter().length == 1
				&& this.getDelimiter()[0] == NEW_LINE && offset + numBytes >= 1
				&& bytes[offset + numBytes - 1] == CARRIAGE_RETURN) {
				numBytes -= 1;
			}
			byte[] trimBytes = Arrays.copyOfRange(bytes, offset, offset + numBytes);
			GenericRowData jsonRow = (GenericRowData) deserializationSchema.deserialize(trimBytes);

			if (jsonRow == null) {
				return null;
			}

			GenericRowData returnRecord = rowData;
			for (int i = 0; i < jsonSelectFieldToJsonFieldMapping.length; i++) {
				returnRecord.setField(jsonSelectFieldToProjectFieldMapping[i],
					jsonRow.getField(jsonSelectFieldToJsonFieldMapping[i]));
			}

			emitted++;
			return returnRecord;
		}
Code review comment: Instead of peeking prior env, type narrow to do clause

Code under review:
private SymbolEnv handleFilterClauses (BLangExpression filterExpression) {
        checkExpr(filterExpression, queryEnvs.peek(), symTable.booleanType);
        BType actualType = filterExpression.getBType();
        if (TypeTags.TUPLE == actualType.tag) {
            dlog.error(filterExpression.pos, DiagnosticErrorCode.INCOMPATIBLE_TYPES,
                    symTable.booleanType, actualType);
        }
        SymbolEnv filterEnv;
        if (selectClauses.peek() != null) {
           filterEnv = typeNarrower.evaluateTruth(filterExpression, selectClauses.peek(), queryEnvs.pop());
           queryEnvs.push(filterEnv);
        } else {
            filterEnv = queryEnvs.peek();
        }
        return filterEnv;
    }

Line commented on: filterEnv = queryEnvs.peek();

Revised code:
private SymbolEnv handleFilterClauses (BLangExpression filterExpression) {
        checkExpr(filterExpression, queryEnvs.peek(), symTable.booleanType);
        BType actualType = filterExpression.getBType();
        if (TypeTags.TUPLE == actualType.tag) {
            dlog.error(filterExpression.pos, DiagnosticErrorCode.INCOMPATIBLE_TYPES,
                    symTable.booleanType, actualType);
        }
        SymbolEnv filterEnv = typeNarrower.evaluateTruth(filterExpression, queryFinalClauses.peek(), queryEnvs.pop());
        queryEnvs.push(filterEnv);
        return filterEnv;
    }
Code review comment: That is necessary when the JsonPlatform is added to a Collection (Set).

Code under review:
public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        if (o == null || getClass() != o.getClass()) {
            return false;
        }
        JsonPlatform that = (JsonPlatform) o;
        return Objects.equals(platformKey, that.platformKey);
    }

Line commented on: return Objects.equals(platformKey, that.platformKey);

Revised code:
public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        if (o == null || getClass() != o.getClass()) {
            return false;
        }
        JsonPlatform that = (JsonPlatform) o;
        return Objects.equals(platformKey, that.platformKey);
    }
Code review comment: It looks like there is a way to provide flow control without buffering. Would one of the BlockingSubscriber implementations help simplify the contract here?

Code under review:
private void start() throws ApiException {
    this.subscriber.addListener(this, SystemExecutors.getFuturesExecutor());
    this.subscriber.startAsync();
    this.subscriber.awaitRunning();
    try {
      this.subscriber.allowFlow(
          FlowControlRequest.newBuilder()
              .setAllowedBytes(flowControlSettings.bytesOutstanding())
              .setAllowedMessages(flowControlSettings.messagesOutstanding())
              .build());
    } catch (Throwable t) {
      throw ExtractStatus.toCanonical(t).underlying;
    }
  }

Line commented on: this.subscriber.allowFlow(

Revised code:
private void start() {
    this.subscriber.addListener(this, SystemExecutors.getFuturesExecutor());
    this.subscriber.startAsync();
    this.subscriber.awaitRunning();
    try {
      this.subscriber.allowFlow(
          FlowControlRequest.newBuilder()
              .setAllowedBytes(flowControlSettings.bytesOutstanding())
              .setAllowedMessages(flowControlSettings.messagesOutstanding())
              .build());
    } catch (Throwable t) {
      throw ExtractStatus.toCanonical(t).underlying;
    }
  }