muttakin2301/ccr_train_1000 huggingface dataset:
Code review comment: It would be better to have this logic in one place and not distributed across two different classes (head is being marked as ended in the processor, while non heads are marked via task).  Also what you proposed adds an (additional) cyclic dependency between Task and Processor, which is also not good.  Maybe the best approach would be to encapsulate this whole logic in `OperatorChain` class and pass it to the processors? For example by adding `OperatorChain#endInput(int inputId)` method, which would: - if the head is `TwoInput`, it would track which inputs have ended. If all of them, then it would propagate the `endInput` call down the operator chain. - if the head is `OneInput`, it would just end all of the inputs  For the future, we can not be sure how will be `TwoInputStreamOperator` chaining implemented, but if we keep `OperatorChain` class, the support for propagating `endInput` would need to be adjusted only there.

Code before review:
private boolean checkFinished() throws Exception {
boolean isFinished = input.isFinished();
if (isFinished) {
if (streamOperator instanceof BoundedOneInput) {
synchronized (lock) {
((BoundedOneInput) streamOperator).endInput();
}
}
}
return isFinished;
}

Line under review: if (streamOperator instanceof BoundedOneInput) {

Code after addressing the review:
private boolean checkFinished() throws Exception {
boolean isFinished = input.isFinished();
if (isFinished) {
synchronized (lock) {
operatorChain.endInput(1);
}
}
return isFinished;
}
Code review comment: Openjdk has  com.sun.management available , and is giving numbers. I will try with another jdk maybe IBM 

Code before review:
private void printSystemInformation(StringBuilder stringBuilder) {
try {
long totalMemory = Runtime.getRuntime().totalMemory() / 1024;
long freeMemory = Runtime.getRuntime().freeMemory() / 1024;
long maxMemory = Runtime.getRuntime().maxMemory() / 1024;
String used_Memory = totalMemory - freeMemory + " KB";
String available_Memory = (maxMemory - (totalMemory - freeMemory)) + " KB";
OperatingSystemMXBean mbean = (com.sun.management.OperatingSystemMXBean)
ManagementFactory.getOperatingSystemMXBean();
String processCpuLoad = Double.toString(mbean.getProcessCpuLoad());
String systemCpuLoad = Double.toString(mbean.getSystemCpuLoad());
stringBuilder.append("System State Information -------").append(System.lineSeparator());
stringBuilder.append("Used Memory : " + used_Memory).append(System.lineSeparator());
stringBuilder.append("Available Memory : " + available_Memory).append(System.lineSeparator());
stringBuilder.append("CPU Process Load : " + processCpuLoad).append(System.lineSeparator());
stringBuilder.append("CPU System Load : " + systemCpuLoad).append(System.lineSeparator());
} catch (Exception e) {
}
}

Line under review: OperatingSystemMXBean mbean = (com.sun.management.OperatingSystemMXBean)

Code after addressing the review:
private void printSystemInformation(StringBuilder stringBuilder) {
try {
long totalMemory = Runtime.getRuntime().totalMemory() / 1024;
long freeMemory = Runtime.getRuntime().freeMemory() / 1024;
long maxMemory = Runtime.getRuntime().maxMemory() / 1024;
String usedMemory = totalMemory - freeMemory + " KB";
String availableMemory = (maxMemory - (totalMemory - freeMemory)) + " KB";
String processCpuLoad = Double.toString(this.mbean.getProcessCpuLoad());
String systemCpuLoad = Double.toString(this.mbean.getSystemCpuLoad());
stringBuilder.append("System State Information -------").append(System.lineSeparator());
stringBuilder.append("Used Memory : " + usedMemory).append(System.lineSeparator());
stringBuilder.append("Available Memory : " + availableMemory).append(System.lineSeparator());
stringBuilder.append("CPU Process Load : " + processCpuLoad).append(System.lineSeparator());
stringBuilder.append("CPU System Load : " + systemCpuLoad).append(System.lineSeparator());
} catch (Exception e) {
}
}
Code review comment: Actually, the JsonParser would detect this ... 

Code before review:
static void fillTensor(TokenBuffer buffer, TensorFieldValue tensorFieldValue) {
Tensor.Builder builder = Tensor.Builder.of(tensorFieldValue.getDataType().getTensorType());
expectOneOf(buffer.current(), JsonToken.START_OBJECT, JsonToken.START_ARRAY);
int initNesting = buffer.nesting();
while ( ! buffer.isEmpty()) {
Supplier<Token> lookahead = buffer.lookahead();
Token next = lookahead.get();
if (TENSOR_CELLS.equals(next.name) && ! primitiveContent(next.token, lookahead.get().token)) {
buffer.next();
readTensorCells(buffer, builder);
}
else if (TENSOR_VALUES.equals(next.name) && builder.type().dimensions().stream().allMatch(Dimension::isIndexed)) {
buffer.next();
readTensorValues(buffer, builder);
}
else if (TENSOR_BLOCKS.equals(next.name)) {
buffer.next();
readTensorBlocks(buffer, builder);
}
else if (TENSOR_TYPE.equals(next.name) && next.token == JsonToken.VALUE_STRING) {
buffer.next();
}
else if (buffer.nesting() == initNesting && JsonToken.END_OBJECT == next.token) {
buffer.next();
break;
}
else {
readDirectTensorValue(buffer, builder);
break;
}
}
if (buffer.nesting() + 1 != initNesting)
throw new IllegalArgumentException("incomplete JSON structure for " + tensorFieldValue);
expectOneOf(buffer.current(), JsonToken.END_OBJECT, JsonToken.END_ARRAY);
tensorFieldValue.assign(builder.build());
}

Line under review: throw new IllegalArgumentException("incomplete JSON structure for " + tensorFieldValue);

Code after addressing the review:
static void fillTensor(TokenBuffer buffer, TensorFieldValue tensorFieldValue) {
Tensor.Builder builder = Tensor.Builder.of(tensorFieldValue.getDataType().getTensorType());
expectOneOf(buffer.current(), JsonToken.START_OBJECT, JsonToken.START_ARRAY);
int initNesting = buffer.nesting();
while (true) {
Supplier<Token> lookahead = buffer.lookahead();
Token next = lookahead.get();
if (TENSOR_CELLS.equals(next.name) && ! primitiveContent(next.token, lookahead.get().token)) {
buffer.next();
readTensorCells(buffer, builder);
}
else if (TENSOR_VALUES.equals(next.name) && builder.type().dimensions().stream().allMatch(Dimension::isIndexed)) {
buffer.next();
readTensorValues(buffer, builder);
}
else if (TENSOR_BLOCKS.equals(next.name)) {
buffer.next();
readTensorBlocks(buffer, builder);
}
else if (TENSOR_TYPE.equals(next.name) && next.token == JsonToken.VALUE_STRING) {
buffer.next();
}
else if (buffer.nesting() == initNesting && JsonToken.END_OBJECT == next.token) {
buffer.next();
break;
}
else {
readDirectTensorValue(buffer, builder);
break;
}
}
expectOneOf(buffer.current(), JsonToken.END_OBJECT, JsonToken.END_ARRAY);
tensorFieldValue.assign(builder.build());
}
Code review comment: Fixed (Introduced separate `roCellContainng` to handle the readonly case)

Code before review:
public static CellSemType cellContaining(Env env, SemType ty, CellAtomicType.CellMutability mut) {
assert !(ty instanceof CellSemType);
CellAtomicType atomicCell = CellAtomicType.from(ty, mut);
TypeAtom atom = env.cellAtom(atomicCell);
BddNode bdd = bddAtom(atom);
ComplexSemType complexSemType = PredefinedType.basicSubtype(BasicTypeCode.BT_CELL, bdd);
return new CellSemType(complexSemType.subtypeDataList);
}

Line under review: TypeAtom atom = env.cellAtom(atomicCell);

Code after addressing the review:
public static CellSemType cellContaining(Env env, SemType ty, CellAtomicType.CellMutability mut) {
assert Core.isNever(ty) || !Core.isSubtypeSimple(ty, PredefinedType.CELL);
CellAtomicType atomicCell = CellAtomicType.from(ty, mut);
TypeAtom atom = env.cellAtom(atomicCell);
BddNode bdd = bddAtom(atom);
ComplexSemType complexSemType = PredefinedType.basicSubtype(BasicTypeCode.BT_CELL, bdd);
return CellSemType.from(complexSemType.subtypeDataList);
}
Code review comment: According to this, order doesn't matter.  https://stackoverflow.com/a/37986481/4220757

Code before review:
private static Mono<Void> generateEvents(AtomicBoolean isRunning) {
final Logger logger = LoggerFactory.getLogger("Producer");
final Scheduler scheduler = Schedulers.elastic();
final Duration operationTimeout = Duration.ofSeconds(5);
final String[] machineIds = new String[]{"2A", "9B", "6C"};
final Random random = new Random();
final EventHubProducerAsyncClient client = new EventHubClientBuilder()
.connectionString(EH_CONNECTION_STRING)
.buildAsyncProducerClient();
return Mono.<Void>fromRunnable(() -> {
while (isRunning.get()) {
int milliseconds = random.nextInt(1000);
try {
TimeUnit.MILLISECONDS.sleep(milliseconds);
} catch (InterruptedException ignored) {
}
final String machineId = machineIds[random.nextInt(machineIds.length)];
final int temperature = Math.abs(random.nextInt() % 101);
logger.info("[{}] Temperature: {}C", machineId, temperature);
final EventData event = new EventData(String.valueOf(temperature));
final CreateBatchOptions batchOptions = new CreateBatchOptions().setPartitionKey(machineId);
client.createBatch(batchOptions).flatMap(batch -> {
batch.tryAdd(event);
return client.send(batch);
}).block(operationTimeout);
}
}).subscribeOn(scheduler)
.doFinally(signal -> {
logger.info("Disposing of producer.");
client.close();
});
}

Line under review: }).subscribeOn(scheduler)

Code after addressing the review:
private static Mono<Void> generateEvents(AtomicBoolean isRunning) {
final Logger logger = LoggerFactory.getLogger("Producer");
final Scheduler scheduler = Schedulers.elastic();
final Duration operationTimeout = Duration.ofSeconds(5);
final String[] machineIds = new String[]{"2A", "9B", "6C"};
final Random random = new Random();
final EventHubProducerAsyncClient client = new EventHubClientBuilder()
.connectionString(EH_CONNECTION_STRING)
.buildAsyncProducerClient();
return Mono.<Void>fromRunnable(() -> {
while (isRunning.get()) {
int milliseconds = random.nextInt(1000);
try {
TimeUnit.MILLISECONDS.sleep(milliseconds);
} catch (InterruptedException ignored) {
}
final String machineId = machineIds[random.nextInt(machineIds.length)];
final int temperature = Math.abs(random.nextInt() % 101);
logger.info("[{}] Temperature: {}C", machineId, temperature);
final EventData event = new EventData(String.valueOf(temperature));
final CreateBatchOptions batchOptions = new CreateBatchOptions().setPartitionKey(machineId);
client.createBatch(batchOptions).flatMap(batch -> {
batch.tryAdd(event);
return client.send(batch);
}).block(operationTimeout);
}
}).subscribeOn(scheduler)
.doFinally(signal -> {
logger.info("Disposing of producer.");
client.close();
});
}
Code review comment: That's a sanity check I copied over from the old `ClassPathPackagedProgramRetrieverTest`. I thought it's reasonable to check whether the jar loading without modifying the `java.class.path` System property. Isn't that essentially the same as what you suggest? ...just with less code?

Code before review:
public void testJarFromSystemClasspath() throws MalformedURLException {
classpathProvider.setSystemClasspathWithTwoEntryClasses();
final Collection<String> systemClasspath =
StreamSupport.stream(
FromClasspathEntryClassInformationProvider.extractSystemClasspath()
.spliterator(),
false)
.map(File::getName)
.collect(Collectors.toList());
final Collection<String> expectedContent =
StreamSupport.stream(
classpathProvider
.getURLUserClasspathWithTwoEntryClasses()
.spliterator(),
false)
.map(URL::getPath)
.map(FilenameUtils::getName)
.filter(name -> name.endsWith("jar"))
.collect(Collectors.toList());
assertThat(
systemClasspath,
IsIterableContainingInAnyOrder.containsInAnyOrder(expectedContent.toArray()));
}

Line under review: false)

Code after addressing the review:
public void testJarFromSystemClasspath() throws MalformedURLException {
multipleEntryClassesClasspathProvider.setSystemClasspath();
final Collection<String> systemClasspath =
StreamSupport.stream(
FromClasspathEntryClassInformationProvider.extractSystemClasspath()
.spliterator(),
false)
.map(File::getName)
.collect(Collectors.toList());
final Collection<String> expectedContent =
StreamSupport.stream(
multipleEntryClassesClasspathProvider
.getURLUserClasspath()
.spliterator(),
false)
.map(URL::getPath)
.map(FilenameUtils::getName)
.filter(name -> name.endsWith("jar"))
.collect(Collectors.toList());
assertThat(
systemClasspath,
IsIterableContainingInAnyOrder.containsInAnyOrder(expectedContent.toArray()));
}
Code review comment: space after reason is not applicable if the details record is empty. 

Code before review:
public String stringValue() {
return "error " + reason + " " + Optional.ofNullable(details).map(Object::toString).orElse("");
}

Line under review: return "error " + reason + " " + Optional.ofNullable(details).map(Object::toString).orElse("");

Code after addressing the review:
public String stringValue() {
return "error " + reason + Optional.ofNullable(details).map(details -> " " + details).orElse("");
}
Code review comment: The same comment about moving it inside `SavepointFormatType`

Code before review:
public SavepointOptions(CommandLine line) {
super(line);
args = line.getArgs();
dispose = line.hasOption(SAVEPOINT_DISPOSE_OPTION.getOpt());
disposeSavepointPath = line.getOptionValue(SAVEPOINT_DISPOSE_OPTION.getOpt());
jarFile = line.getOptionValue(JAR_OPTION.getOpt());
if (line.hasOption(SAVEPOINT_FORMAT_OPTION)) {
formatType =
ConfigurationUtils.convertValue(
line.getOptionValue(SAVEPOINT_FORMAT_OPTION),
SavepointFormatType.class);
} else {
formatType = SavepointFormatType.DEFAULT;
}
}

Line under review: formatType = SavepointFormatType.DEFAULT;

Code after addressing the review:
public SavepointOptions(CommandLine line) {
super(line);
args = line.getArgs();
dispose = line.hasOption(SAVEPOINT_DISPOSE_OPTION.getOpt());
disposeSavepointPath = line.getOptionValue(SAVEPOINT_DISPOSE_OPTION.getOpt());
jarFile = line.getOptionValue(JAR_OPTION.getOpt());
if (line.hasOption(SAVEPOINT_FORMAT_OPTION)) {
formatType =
ConfigurationUtils.convertValue(
line.getOptionValue(SAVEPOINT_FORMAT_OPTION),
SavepointFormatType.class);
} else {
formatType = SavepointFormatType.DEFAULT;
}
}
Code review comment: Consider something like ```suggestion         return find(tenant, zoneBuckets).map(ArchiveBucket::bucketArn).orElseGet(() -> assignToBucket(zoneId, tenant)); ``` even better if `find()` does the mapping since both users just want the bucket ARN anyway...

Code before review:
private String findOrAssignBucket(ZoneId zoneId, TenantName tenant) {
var zoneBuckets = curatorDb.readArchiveBuckets(zoneId);
if (find(tenant, zoneBuckets).isPresent()) return find(tenant, zoneBuckets).get().bucketArn();
else return assignToBucket(zoneId, tenant);
}

Line under review: else return assignToBucket(zoneId, tenant);

Code after addressing the review:
private String findOrAssignBucket(ZoneId zoneId, TenantName tenant) {
var zoneBuckets = curatorDb.readArchiveBuckets(zoneId);
return find(tenant, zoneBuckets).orElseGet(() -> assignToBucket(zoneId, tenant));
}
Code review comment: These two asserts only checks for the counts. We need to assert the content as well, to see whether they were updated correctly or not.

Code before review:
public void testSeparatedListNodeAllNodeModification() {
SyntaxTree syntaxTree = parseFile("separated_node_list_modify.bal");
ModulePartNode oldRoot = syntaxTree.rootNode();
ModuleVariableDeclarationNode oldModuleVariableDeclarationNode =
(ModuleVariableDeclarationNode) oldRoot.members().get(0);
ListConstructorExpressionNode oldSeperatedlistNode =
(ListConstructorExpressionNode) oldModuleVariableDeclarationNode.initializer();
TokenModifier tokenModifier = new TokenModifier();
ModulePartNode newRoot = (ModulePartNode) oldRoot.apply(tokenModifier);
ModuleVariableDeclarationNode newModuleVariableDeclarationNode =
(ModuleVariableDeclarationNode) newRoot.members().get(0);
ListConstructorExpressionNode newSeperatedlistNode =
(ListConstructorExpressionNode) newModuleVariableDeclarationNode.initializer();
Assert.assertEquals(oldSeperatedlistNode.expressions().separatorSize(),
newSeperatedlistNode.expressions().separatorSize());
Assert.assertEquals(oldSeperatedlistNode.expressions().size(),
newSeperatedlistNode.expressions().size());
}

Line under review: newSeperatedlistNode.expressions().size());

Code after addressing the review:
public void testSeparatedListNodeAllNodeModification() {
SyntaxTree syntaxTree = parseFile("separated_node_list_modify_all_nodes.bal");
ModulePartNode oldRoot = syntaxTree.rootNode();
WhiteSpaceMinutiaeRemover whiteSpaceMinutiaeRemover = new WhiteSpaceMinutiaeRemover();
ModulePartNode newRoot = (ModulePartNode) oldRoot.apply(whiteSpaceMinutiaeRemover);
String expectedStr = getFileContentAsString("separated_node_list_modify_all_nodes_assert.bal");
String actualStr = newRoot.toString();
Assert.assertEquals(actualStr, expectedStr);
}
Code review comment: No, NA would pick up the context set during the last HA tick.

Code before review:
public NodeAgentContext nextContext() throws InterruptedException {
synchronized (monitor) {
nextContext = null;
Duration untilNextContext = Duration.ZERO;
while (setAndGetIsFrozen(wantFrozen) ||
nextContext == null ||
(untilNextContext = Duration.between(Instant.now(), nextContextAt)).toMillis() > 0) {
if (pendingInterrupt) {
pendingInterrupt = false;
throw new InterruptedException("interrupt() was called before next context was scheduled");
}
try {
monitor.wait(Math.max(untilNextContext.toMillis(), 0L));
} catch (InterruptedException ignored) { }
}
currentContext = nextContext;
return currentContext;
}
}

Line under review: currentContext = nextContext;

Code after addressing the review:
public NodeAgentContext nextContext() throws InterruptedException {
synchronized (monitor) {
nextContext = null;
Duration untilNextContext = Duration.ZERO;
while (setAndGetIsFrozen(wantFrozen) ||
nextContext == null ||
(untilNextContext = Duration.between(Instant.now(), nextContextAt)).toMillis() > 0) {
if (pendingInterrupt) {
pendingInterrupt = false;
throw new InterruptedException("interrupt() was called before next context was scheduled");
}
try {
monitor.wait(Math.max(untilNextContext.toMillis(), 0L));
} catch (InterruptedException ignored) { }
}
currentContext = nextContext;
return currentContext;
}
}
Code review comment: Each Iceberg split contains data files, delete files (for upsert), schema string. Each data file also contains stats for every column. if the table is wide (many columns), each split may go over 10 KB

Code before review:
public byte[] serialize(HybridSourceEnumeratorState enumState) throws IOException {
try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(baos)) {
out.writeInt(enumState.getCurrentSourceIndex());
out.writeInt(enumState.wrappedStateSerializerVersion());
out.writeInt(enumState.getWrappedState().length);
out.write(enumState.getWrappedState());
out.flush();
return baos.toByteArray();
}
}

Line under review: out.writeInt(enumState.getWrappedState().length);

Code after addressing the review:
public byte[] serialize(HybridSourceEnumeratorState enumState) throws IOException {
try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(baos)) {
out.writeInt(enumState.getCurrentSourceIndex());
out.writeInt(enumState.getWrappedStateSerializerVersion());
out.writeInt(enumState.getWrappedState().length);
out.write(enumState.getWrappedState());
out.flush();
return baos.toByteArray();
}
}
Code review comment: There are still has some assertEquals

Code before review:
public void histogram() {
assertEquals(metricsTrackerFacade.getMetricsTrackerManager().getClass(), MetricsTrackerManagerFixture2.class);
((MetricsTrackerManagerFixture2) metricsTrackerFacade.getMetricsTrackerManager()).setMetricsTrackerFactory(new MetricsTrackerFactoryFixture2());
HistogramMetricsTrackerDelegate delegate = metricsTrackerFacade.histogramStartTimer("request");
metricsTrackerFacade.histogramObserveDuration(delegate);
assertThat(delegate.getClass().getName(), is(NoneHistogramMetricsTrackerDelegate.class.getName()));
}

Line under review: assertEquals(metricsTrackerFacade.getMetricsTrackerManager().getClass(), MetricsTrackerManagerFixture2.class);

Code after addressing the review:
public void histogram() {
assertThat(metricsTrackerFacade.getMetricsTrackerManager().getClass().getName(), is(MetricsTrackerManagerFixture2.class.getName()));
((MetricsTrackerManagerFixture2) metricsTrackerFacade.getMetricsTrackerManager()).setMetricsTrackerFactory(new MetricsTrackerFactoryFixture2());
HistogramMetricsTrackerDelegate delegate = metricsTrackerFacade.histogramStartTimer("request");
metricsTrackerFacade.histogramObserveDuration(delegate);
assertThat(delegate.getClass().getName(), is(NoneHistogramMetricsTrackerDelegate.class.getName()));
}
Code review comment: ```suggestion                 if (!isAllowNull && !EngineType.supportNotNullColumn(engineName)) { ```

Code before review:
public void analyze(boolean isOlap, boolean isInternalCatalog, String engineName) throws AnalysisException {
if (isInternalCatalog) {
if (!isAllowNull) {
if (!EngineType.supportNotNullColumn(engineName)) {
throw new AnalysisException(String.format("All columns must be nullable for external table. " +
"Column %s is not nullable, You can rebuild the external table and " +
"We strongly recommend that you use catalog to access external data", name));
}
}
}
analyze(isOlap);
}

Line under review: if (!EngineType.supportNotNullColumn(engineName)) {

Code after addressing the review:
public void analyze(boolean isOlap, boolean isInternalCatalog, String engineName) throws AnalysisException {
if (isInternalCatalog) {
if (!isAllowNull && !EngineType.supportNotNullColumn(engineName)) {
throw new AnalysisException(String.format("All columns must be nullable for external table. " +
"Column %s is not nullable, You can rebuild the external table and " +
"We strongly recommend that you use catalog to access external data", name));
}
}
analyze(isOlap);
}
Code review comment: Please add a method on environment instead of matching the free form and human-readable region.

Code before review:
private void createContainerData(ContainerNodeSpec nodeSpec) {
ContainerData containerData = ContainerData.createClean(environment,
ContainerName.fromHostname(nodeSpec.hostname));
if (nodeSpec.nodeType.equals(NodeType.config.name())) {
logger.info("Creating files needed by config server");
new ConfigServerContainerData(environment, nodeSpec.hostname).writeTo(containerData);
}
if (environment.getRegion().startsWith("aws-")) {
logger.info("Creating files for message of the day and the bash prompt");
new MotdContainerData(nodeSpec, environment).writeTo(containerData);
new PromptContainerData(environment).writeTo(containerData);
}
}

Line under review: if (environment.getRegion().startsWith("aws-")) {

Code after addressing the review:
private void createContainerData(ContainerNodeSpec nodeSpec) {
ContainerData containerData = ContainerData.createClean(environment,
ContainerName.fromHostname(nodeSpec.hostname));
if (nodeSpec.nodeType.equals(NodeType.config.name())) {
logger.info("Creating files needed by config server");
new ConfigServerContainerData(environment, nodeSpec.hostname).writeTo(containerData);
}
if (environment.getRegion().startsWith("aws-")) {
logger.info("Creating files for message of the day and the bash prompt");
new MotdContainerData(nodeSpec, environment).writeTo(containerData);
new PromptContainerData(environment).writeTo(containerData);
}
}
Code review comment: To me it would make sense to auto-add `/` like for the `*Page` variables.

Code before review:
public FormAuthenticationMechanism get() {
String key;
if (!httpConfiguration.encryptionKey.isPresent()) {
if (encryptionKey != null) {
key = encryptionKey;
} else {
byte[] data = new byte[32];
new SecureRandom().nextBytes(data);
key = encryptionKey = Base64.getEncoder().encodeToString(data);
log.warn("Encryption key was not specified for persistent FORM auth, using temporary key " + key);
}
} else {
key = httpConfiguration.encryptionKey.get();
}
FormAuthConfig form = buildTimeConfig.auth.form;
PersistentLoginManager loginManager = new PersistentLoginManager(key, form.cookieName, form.timeout.toMillis(),
form.newCookieInterval.toMillis());
String loginPage = form.loginPage.startsWith("/") ? form.loginPage : "/" + form.loginPage;
String errorPage = form.errorPage.startsWith("/") ? form.errorPage : "/" + form.errorPage;
String landingPage = form.landingPage.startsWith("/") ? form.landingPage : "/" + form.landingPage;
String postLocation = form.postLocation;
String usernameFieldName = form.usernameFieldName;
String passwordFieldName = form.passwordFieldName;
String locationCookie = form.locationCookie;
boolean redirectAfterLogin = form.redirectAfterLogin;
return new FormAuthenticationMechanism(loginPage, postLocation, usernameFieldName, passwordFieldName,
errorPage, landingPage, redirectAfterLogin, locationCookie, loginManager);
}

Line under review: String postLocation = form.postLocation;

Code after addressing the review:
public FormAuthenticationMechanism get() {
String key;
if (!httpConfiguration.encryptionKey.isPresent()) {
if (encryptionKey != null) {
key = encryptionKey;
} else {
byte[] data = new byte[32];
new SecureRandom().nextBytes(data);
key = encryptionKey = Base64.getEncoder().encodeToString(data);
log.warn("Encryption key was not specified for persistent FORM auth, using temporary key " + key);
}
} else {
key = httpConfiguration.encryptionKey.get();
}
FormAuthConfig form = buildTimeConfig.auth.form;
PersistentLoginManager loginManager = new PersistentLoginManager(key, form.cookieName, form.timeout.toMillis(),
form.newCookieInterval.toMillis());
String loginPage = form.loginPage.startsWith("/") ? form.loginPage : "/" + form.loginPage;
String errorPage = form.errorPage.startsWith("/") ? form.errorPage : "/" + form.errorPage;
String landingPage = form.landingPage.startsWith("/") ? form.landingPage : "/" + form.landingPage;
String postLocation = form.postLocation.startsWith("/") ? form.postLocation : "/" + form.postLocation;
String usernameParameter = form.usernameParameter;
String passwordParameter = form.passwordParameter;
String locationCookie = form.locationCookie;
boolean redirectAfterLogin = form.redirectAfterLogin;
return new FormAuthenticationMechanism(loginPage, postLocation, usernameParameter, passwordParameter,
errorPage, landingPage, redirectAfterLogin, locationCookie, loginManager);
}
Code review comment: yes, it's catalog, db and table name are case-sensitive.

Code before review:
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
PaimonTable that = (PaimonTable) o;
return catalogName.equals(that.catalogName) &&
databaseName.equals(that.databaseName) &&
tableName.equals(that.tableName) &&
createTime == that.createTime;
}

Line under review: return catalogName.equals(that.catalogName) &&

Code after addressing the review:
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
PaimonTable that = (PaimonTable) o;
return catalogName.equals(that.catalogName) &&
databaseName.equals(that.databaseName) &&
tableName.equals(that.tableName) &&
createTime == that.createTime;
}
Code review comment: Not sure, if I understand your question.   Before applying any specs the table has a schema as described by  ``` contextResolvedTable.getResolvedSchema()                 .toSourceRowDataType()                 .getLogicalType() ```  Only once the `ReadingMetadataSpec` is applied it will change the type to `spec.getProducedType().get()`. In this particular case of reading metadata in a `TableSourceScan` it is equal to `DynamicSourceUtils.createProducedType(contextResolvedTable.getResolvedSchema(), tableSource);`

Code before review:
private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory typeFactory) {
if (tableSource == null) {
DynamicTableSourceFactory factory =
context.getModuleManager()
.getFactory(Module::getTableSourceFactory)
.orElse(null);
if (factory == null) {
Catalog catalog =
context.getCatalogManager()
.getCatalog(contextResolvedTable.getIdentifier().getCatalogName())
.orElse(null);
factory =
FactoryUtil.getDynamicTableFactory(DynamicTableSourceFactory.class, catalog)
.orElse(null);
}
tableSource =
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
if (sourceAbilities != null) {
RowType newProducedType =
(RowType)
contextResolvedTable
.getResolvedSchema()
.toSourceRowDataType()
.getLogicalType();
for (SourceAbilitySpec spec : sourceAbilities) {
SourceAbilityContext sourceAbilityContext =
new SourceAbilityContext(context, typeFactory, newProducedType);
spec.apply(tableSource, sourceAbilityContext);
if (spec.getProducedType().isPresent()) {
newProducedType = spec.getProducedType().get();
}
}
}
}
return tableSource;
}

Line under review: .getLogicalType();

Code after addressing the review:
private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory typeFactory) {
if (tableSource == null) {
DynamicTableSourceFactory factory =
context.getModuleManager()
.getFactory(Module::getTableSourceFactory)
.orElse(null);
if (factory == null) {
Catalog catalog =
context.getCatalogManager()
.getCatalog(contextResolvedTable.getIdentifier().getCatalogName())
.orElse(null);
factory =
FactoryUtil.getDynamicTableFactory(DynamicTableSourceFactory.class, catalog)
.orElse(null);
}
tableSource =
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
if (sourceAbilities != null) {
RowType newProducedType =
(RowType)
contextResolvedTable
.getResolvedSchema()
.toPhysicalRowDataType()
.getLogicalType();
for (SourceAbilitySpec spec : sourceAbilities) {
SourceAbilityContext sourceAbilityContext =
new SourceAbilityContext(context, typeFactory, newProducedType);
spec.apply(tableSource, sourceAbilityContext);
if (spec.getProducedType().isPresent()) {
newProducedType = spec.getProducedType().get();
}
}
}
}
return tableSource;
}
Code review comment: IMU, if we set `azure.activedirectory.tenant-id=common`, then multi-tenant application will function.

Code before review:
private ClientRegistration azureClientRegistration() {
String tenantId = aadAuthenticationProperties.getTenantId().trim();
Assert.hasText(tenantId, "azure.activedirectory.tenant-id should have text.");
Assert.doesNotContain(tenantId, " ", "azure.activedirectory.tenant-id should not contain ' '.");
Assert.doesNotContain(tenantId, "/", "azure.activedirectory.tenant-id should not contain '/'.");
return ClientRegistration.withRegistrationId("azure")
.clientId(aadAuthenticationProperties.getClientId())
.clientSecret(aadAuthenticationProperties.getClientSecret())
.clientAuthenticationMethod(ClientAuthenticationMethod.POST)
.authorizationGrantType(AuthorizationGrantType.AUTHORIZATION_CODE)
.redirectUriTemplate("{baseUrl}/login/oauth2/code/{registrationId}")
.scope("openid", "https:
.authorizationUri(
String.format(
"https:
tenantId
)
)
.tokenUri(
String.format(
"https:
tenantId
)
)
.userInfoUri("https:
.userNameAttributeName(AADAccessTokenClaim.NAME)
.jwkSetUri(
String.format(
"https:
tenantId
)
)
.clientName("Azure")
.build();
}

Line under review: return ClientRegistration.withRegistrationId("azure")

Code after addressing the review:
private ClientRegistration azureClientRegistration() {
String tenantId = aadAuthenticationProperties.getTenantId().trim();
Assert.hasText(tenantId, "azure.activedirectory.tenant-id should have text.");
Assert.doesNotContain(tenantId, " ", "azure.activedirectory.tenant-id should not contain ' '.");
Assert.doesNotContain(tenantId, "/", "azure.activedirectory.tenant-id should not contain '/'.");
return ClientRegistration.withRegistrationId("azure")
.clientId(aadAuthenticationProperties.getClientId())
.clientSecret(aadAuthenticationProperties.getClientSecret())
.clientAuthenticationMethod(ClientAuthenticationMethod.POST)
.authorizationGrantType(AuthorizationGrantType.AUTHORIZATION_CODE)
.redirectUriTemplate("{baseUrl}/login/oauth2/code/{registrationId}")
.scope(aadAuthenticationProperties.getScope())
.authorizationUri(
String.format(
"https:
tenantId
)
)
.tokenUri(
String.format(
"https:
tenantId
)
)
.userInfoUri("https:
.userNameAttributeName(AADAccessTokenClaim.NAME)
.jwkSetUri(
String.format(
"https:
tenantId
)
)
.clientName("Azure")
.build();
}
Code review comment: You are right, I will add a checkArgument.

Code before review:
void requestExclusiveBuffers(int numExclusiveBuffers) throws IOException {
if (numExclusiveBuffers <= 0) {
return;
}
Collection<MemorySegment> segments = globalPool.requestMemorySegments(numExclusiveBuffers);
synchronized (bufferQueue) {
for (MemorySegment segment : segments) {
bufferQueue.addExclusiveBuffer(
new NetworkBuffer(segment, this), numRequiredBuffers);
}
}
}

Line under review: if (numExclusiveBuffers <= 0) {

Code after addressing the review:
void requestExclusiveBuffers(int numExclusiveBuffers) throws IOException {
checkArgument(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
if (numExclusiveBuffers == 0) {
return;
}
Collection<MemorySegment> segments = globalPool.requestMemorySegments(numExclusiveBuffers);
synchronized (bufferQueue) {
checkState(
unsynchronizedGetFloatingBuffersAvailable() == 0,
"Bug in buffer allocation logic: floating buffer is allocated before exclusive buffers are initialized.");
for (MemorySegment segment : segments) {
bufferQueue.addExclusiveBuffer(
new NetworkBuffer(segment, this), numRequiredBuffers);
}
}
}
Code review comment: your means that it should statistic the usage of disk instead of the data?

Code before review:
public void init() throws LoadBalanceException {
Backend be = infoService.getBackend(beId);
if (be == null) {
throw new LoadBalanceException("backend " + beId + " does not exist");
}
isAvailable = be.isScheduleAvailable() && be.isLoadAvailable() && be.isQueryAvailable();
ImmutableMap<String, DiskInfo> disks = be.getDisks();
for (DiskInfo diskInfo : disks.values()) {
TStorageMedium medium = diskInfo.getStorageMedium();
if (diskInfo.getState() == DiskState.ONLINE) {
totalCapacityMap.put(medium, totalCapacityMap.getOrDefault(medium, 0L) + diskInfo.getTotalCapacityB());
totalUsedCapacityMap.put(medium, totalUsedCapacityMap.getOrDefault(medium, 0L) + (diskInfo.getTotalCapacityB() - diskInfo.getAvailableCapacityB()));
}
RootPathLoadStatistic pathStatistic = new RootPathLoadStatistic(beId, diskInfo.getRootPath(),
diskInfo.getPathHash(), diskInfo.getStorageMedium(),
diskInfo.getTotalCapacityB(), diskInfo.getDiskUsedCapacityB(), diskInfo.getState());
pathStatistics.add(pathStatistic);
}
totalReplicaNumMap = invertedIndex.getReplicaNumByBeIdAndStorageMedium(beId);
for (TStorageMedium medium : TStorageMedium.values()) {
if (!hasMedium(medium)) {
totalReplicaNumMap.put(medium, 0L);
}
}
for (TStorageMedium storageMedium : TStorageMedium.values()) {
classifyPathByLoad(storageMedium);
}
Collections.sort(pathStatistics);
}

Line under review: diskInfo.getTotalCapacityB(), diskInfo.getDiskUsedCapacityB(), diskInfo.getState());

Code after addressing the review:
public void init() throws LoadBalanceException {
Backend be = infoService.getBackend(beId);
if (be == null) {
throw new LoadBalanceException("backend " + beId + " does not exist");
}
isAvailable = be.isScheduleAvailable() && be.isLoadAvailable() && be.isQueryAvailable();
ImmutableMap<String, DiskInfo> disks = be.getDisks();
for (DiskInfo diskInfo : disks.values()) {
TStorageMedium medium = diskInfo.getStorageMedium();
if (diskInfo.getState() == DiskState.ONLINE) {
totalCapacityMap.put(medium, totalCapacityMap.getOrDefault(medium, 0L) + diskInfo.getTotalCapacityB());
totalUsedCapacityMap.put(medium, totalUsedCapacityMap.getOrDefault(medium, 0L) + (diskInfo.getTotalCapacityB() - diskInfo.getAvailableCapacityB()));
}
RootPathLoadStatistic pathStatistic = new RootPathLoadStatistic(beId, diskInfo.getRootPath(),
diskInfo.getPathHash(), diskInfo.getStorageMedium(),
diskInfo.getTotalCapacityB(), diskInfo.getDiskUsedCapacityB(), diskInfo.getState());
pathStatistics.add(pathStatistic);
}
totalReplicaNumMap = invertedIndex.getReplicaNumByBeIdAndStorageMedium(beId);
for (TStorageMedium medium : TStorageMedium.values()) {
if (!hasMedium(medium)) {
totalReplicaNumMap.put(medium, 0L);
}
}
for (TStorageMedium storageMedium : TStorageMedium.values()) {
classifyPathByLoad(storageMedium);
}
Collections.sort(pathStatistics);
}
Code review comment: Printing `acquiredResources` can actually produce quite some text. Hence, I am wondering whether we should really include this information in every `NoResourceAvailableException` since it will be logged for every request later on. Maybe it could be better to log it once in this method and then fail the requests w/o this information.   ``` log.info("Could not acquire the minimum required resources. Failing all pending requests. Acquired {}. Current slot pool status: {}.", acquiredResources, getSlotServiceStatus()); ```  What do you think @rkhachatryan?

Code before review:
private void failPendingRequests(Collection<ResourceRequirement> acquiredResources) {
if (!pendingRequests.isEmpty()) {
final NoResourceAvailableException cause =
new NoResourceAvailableException(
"Could not acquire the minimum required resources. Acquired: "
+ acquiredResources
+ ". Current slot pool status: "
+ getSlotServiceStatus());
cancelPendingRequests(
request -> !isBatchSlotRequestTimeoutCheckDisabled || !request.isBatchRequest(),
cause);
}
}

Line under review: + getSlotServiceStatus());

Code after addressing the review:
private void failPendingRequests(Collection<ResourceRequirement> acquiredResources) {
if (!pendingRequests.isEmpty()) {
final NoResourceAvailableException cause =
new NoResourceAvailableException(
"Could not acquire the minimum required resources. Acquired: "
+ acquiredResources
+ ". Current slot pool status: "
+ getSlotServiceStatus());
cancelPendingRequests(
request -> !isBatchSlotRequestTimeoutCheckDisabled || !request.isBatchRequest(),
cause);
}
}
Code review comment: should this path be more generic?

Code before review:
public void connectionAttributes() throws Exception {
Map<String, String> attributes = new HashMap<String, String>();
attributes.put("_connector_name", "Apache Beam SingleStoreDB I/O");
attributes.put("_connector_version", ReleaseInfo.getReleaseInfo().getVersion());
attributes.put("_product_version", ReleaseInfo.getReleaseInfo().getVersion());
SingleStoreIO.DataSourceConfiguration dataSourceConfiguration =
SingleStoreIO.DataSourceConfiguration.create(serverName + ":" + port)
.withPassword(password)
.withUsername(username);
DataSource dataSource = dataSourceConfiguration.getDataSource();
try (Connection conn = dataSource.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs =
stmt.executeQuery("select * from information_schema.mv_connection_attributes"); ) {
while (rs.next()) {
String attribute = rs.getString(3);
String value = rs.getString(4);
if (attributes.containsKey(attribute)) {
assertEquals(attributes.get(attribute), value);
attributes.remove(attribute);
}
}
} catch (Exception e) {
File file = new File("/home/amakarovych-ua/Test/log");
PrintStream ps = new PrintStream(file);
e.printStackTrace(ps);
ps.close();
throw e;
}
assertTrue(attributes.isEmpty());
}

Line under review: File file = new File("/home/amakarovych-ua/Test/log");

Code after addressing the review:
public void connectionAttributes() throws Exception {
Map<String, String> attributes = new HashMap<String, String>();
attributes.put("_connector_name", "Apache Beam SingleStoreDB I/O");
attributes.put("_connector_version", ReleaseInfo.getReleaseInfo().getVersion());
attributes.put("_product_version", ReleaseInfo.getReleaseInfo().getVersion());
SingleStoreIO.DataSourceConfiguration dataSourceConfiguration =
SingleStoreIO.DataSourceConfiguration.create(serverName + ":" + port)
.withPassword(password)
.withUsername(username);
DataSource dataSource = dataSourceConfiguration.getDataSource();
try (Connection conn = dataSource.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs =
stmt.executeQuery("select * from information_schema.mv_connection_attributes"); ) {
while (rs.next()) {
String attribute = rs.getString(3);
String value = rs.getString(4);
if (attributes.containsKey(attribute)) {
assertEquals(attributes.get(attribute), value);
attributes.remove(attribute);
}
}
}
assertTrue(attributes.isEmpty());
}
Code review comment: We are using the targetType in the switch case

Code before review:
private TypeTestResult checkBuiltInIntSubtypeWidenPossible(BType actualType, BType targetType) {
int actualTag = getReferredType(actualType).tag;
switch (targetType.tag) {
case TypeTags.INT:
if (actualTag == TypeTags.BYTE || TypeTags.isIntegerTypeTag(actualTag)) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.SIGNED32_INT:
if (actualTag == TypeTags.SIGNED16_INT || actualTag == TypeTags.SIGNED8_INT ||
actualTag == TypeTags.UNSIGNED16_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.SIGNED16_INT:
if (actualTag == TypeTags.SIGNED8_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED32_INT:
if (actualTag == TypeTags.UNSIGNED16_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED16_INT:
if (actualTag == TypeTags.UNSIGNED8_INT || actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.BYTE:
if (actualTag == TypeTags.UNSIGNED8_INT) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED8_INT:
if (actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.TYPEREFDESC:
return checkBuiltInIntSubtypeWidenPossible(actualType, getReferredType(targetType));
}
return TypeTestResult.NOT_FOUND;
}

Line under review: int actualTag = getReferredType(actualType).tag;

Code after addressing the review:
private TypeTestResult checkBuiltInIntSubtypeWidenPossible(BType actualType, BType targetType) {
int actualTag = getReferredType(actualType).tag;
switch (targetType.tag) {
case TypeTags.INT:
if (actualTag == TypeTags.BYTE || TypeTags.isIntegerTypeTag(actualTag)) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.SIGNED32_INT:
if (actualTag == TypeTags.SIGNED16_INT || actualTag == TypeTags.SIGNED8_INT ||
actualTag == TypeTags.UNSIGNED16_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.SIGNED16_INT:
if (actualTag == TypeTags.SIGNED8_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED32_INT:
if (actualTag == TypeTags.UNSIGNED16_INT || actualTag == TypeTags.UNSIGNED8_INT ||
actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED16_INT:
if (actualTag == TypeTags.UNSIGNED8_INT || actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.BYTE:
if (actualTag == TypeTags.UNSIGNED8_INT) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.UNSIGNED8_INT:
if (actualTag == TypeTags.BYTE) {
return TypeTestResult.TRUE;
}
break;
case TypeTags.TYPEREFDESC:
return checkBuiltInIntSubtypeWidenPossible(actualType, getReferredType(targetType));
}
return TypeTestResult.NOT_FOUND;
}
Code review comment: Could you extract the common code of initializing the environment?

Code before review:
public void testNonMaterialization() throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(10)
.enableChangelogStateBackend(true)
.getCheckpointConfig()
.setCheckpointStorage(TEMPORARY_FOLDER.newFolder().toURI());
env.setStateBackend(new DelegatedStateBackendWrapper(delegatedStateBackend, t -> t))
.setRestartStrategy(RestartStrategies.fixedDelayRestart(1, 0));
env.configure(
new Configuration()
.set(
StateChangelogOptions.PERIODIC_MATERIALIZATION_INTERVAL,
Duration.ofMinutes(3)));
waitAndAssert(
env,
buildStreamGraph(
env,
new ControlledSource() {
@Override
protected void beforeElement() throws Exception {
if (getRuntimeContext().getAttemptNumber() == 0
&& currentIndex == TOTAL_ELEMENTS / 2) {
waitWhile(() -> completedCheckpointNum.get() <= 0);
throwArtificialFailure();
}
}
}));
}

Line under review: env.configure(

Code after addressing the review:
public void testNonMaterialization() throws Exception {
File checkpointFolder = TEMPORARY_FOLDER.newFolder();
SharedReference<AtomicBoolean> hasMaterialization =
sharedObjects.add(new AtomicBoolean(true));
StreamExecutionEnvironment env =
getEnv(delegatedStateBackend, checkpointFolder, 1000, 1, Long.MAX_VALUE, 0);
waitAndAssert(
buildJobGraph(
env,
new ControlledSource() {
@Override
protected void beforeElement(SourceContext<Integer> ctx)
throws Exception {
if (getRuntimeContext().getAttemptNumber() == 0
&& currentIndex == TOTAL_ELEMENTS / 2) {
waitWhile(() -> completedCheckpointNum.get() <= 0);
hasMaterialization
.get()
.compareAndSet(
true,
!getAllStateHandleId(checkpointFolder)
.isEmpty());
throwArtificialFailure();
}
}
},
generateJobID()));
Preconditions.checkState(!hasMaterialization.get().get());
}
Code review comment: Thanks for this simplification ! Indeed, only needed to test at the source level, not need for the higher level stuff (server, pipeline, sending a message)

Code before review:
public void testCheckpointCoderIsSane() {
SqsUnboundedSource source = new SqsUnboundedSource(mock(SqsIO.Read.class));
CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
}

Line under review: SqsUnboundedSource source = new SqsUnboundedSource(mock(SqsIO.Read.class));

Code after addressing the review:
public void testCheckpointCoderIsSane() {
SqsUnboundedSource source = new SqsUnboundedSource(mock(SqsIO.Read.class));
CoderProperties.coderSerializable(source.getCheckpointMarkCoder());
}
Code review comment: LGTM.  I extracted a single method in Write to populate the displayData with parameters to avoid repeating code.

Code before review:
public void populateDisplayData(DisplayData.Builder builder) {
super.populateDisplayData(builder);
spec.getSpannerConfig().populateDisplayData(builder);
builder.add(
DisplayData.item("batchSizeBytes", spec.getBatchSizeBytes())
.withLabel("Max batch size in sytes"));
builder.add(
DisplayData.item("maxNumMutations", spec.getMaxNumMutations())
.withLabel("Max number of mutated cells in each batch"));
builder.add(
DisplayData.item("maxNumRows", spec.getMaxNumRows())
.withLabel("Max number of rows in each batch"));
builder.add(
DisplayData.item("groupingFactor", spec.getGroupingFactor())
.withLabel("Number of batches to sort over"));
}

Line under review: builder.add(

Code after addressing the review:
public void populateDisplayData(DisplayData.Builder builder) {
super.populateDisplayData(builder);
populateDisplayDataWithParamaters(builder);
}