Repository: apache/iceberg
Pull request #12996: Flink: Add DynamicRecord / DynamicRecordInternal / DynamicRecordInternalSerializer
Author: login: mxm
Labels: name: flink
name: build
Created: 2025-05-07T13:51:54Z | Merged: 2025-05-14T20:09:02Z

This adds the user-facing type DynamicRecord, alongside with its internal representation DynamicRecordInternal and its type information and serializer.

Broken out of https://github.com/apache/iceberg/pull/12424.

The original PR is based on Flink 1.20. This version is based on Flink 2.0.


Diff:
diff --git a/flink/v2.0/build.gradle b/flink/v2.0/build.gradle
index 05d9fb44023f..fbe206f1b3ca 100644
--- a/flink/v2.0/build.gradle
+++ b/flink/v2.0/build.gradle
@@ -68,6 +68,9 @@ project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}") {
 
     implementation libs.datasketches
 
+    // for caching in DynamicSink
+    implementation libs.caffeine
+
     testImplementation libs.flink20.connector.test.utils
     testImplementation libs.flink20.core
     testImplementation libs.flink20.runtime
diff --git a/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
new file mode 100644
index 000000000000..994f7a0865f6
--- /dev/null
+++ b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec partitionSpec;
+  private DistributionMode distributionMode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
+
+  public DynamicRecord(
+      TableIdentifier tableIdentifier,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec partitionSpec,
+      DistributionMode distributionMode,
+      int writeParallelism) {
+    this.tableIdentifier = tableIdentifier;
+    this.branch = branch;
+    this.schema = schema;
+    this.partitionSpec = partitionSpec;
+    this.rowData = rowData;
+    this.distributionMode = distributionMode;
+    this.writeParallelism = writeParallelism;
+  }
+
+  public TableIdentifier tableIdentifier() {
+    return tableIdentifier;
+  }
+
+  public void setTableIdentifier(TableIdentifier tableIdentifier) {
+    this.tableIdentifier = tableIdentifier;
+  }
+
+  public String branch() {
+    return branch;
+  }
+
+  public void setBranch(String branch) {
+    this.branch = branch;
+  }
+
+  public Schema schema() {
+    return schema;
+  }
+
+  public void setSchema(Schema schema) {
+    this.schema = schema;
+  }
+
+  public PartitionSpec spec() {
+    return partitionSpec;
+  }
+
+  public void setPartitionSpec(PartitionSpec partitionSpec) {
+    this.partitionSpec = partitionSpec;
+  }
+
+  public RowData rowData() {
+    return rowData;
+  }
+
+  public void setRowData(RowData rowData) {
+    this.rowData = rowData;
+  }
+
+  public DistributionMode distributionMode() {
+    return distributionMode;
+  }
+
+  public void setDistributionMode(DistributionMode distributionMode) {
+    this.distributionMode = distributionMode;
+  }
+
+  public int writeParallelism() {
+    return writeParallelism;
+  }
+
+  public void writeParallelism(int parallelism) {
+    this.writeParallelism = parallelism;
+  }
+
+  public boolean upsertMode() {
+    return upsertMode;
+  }
+
+  public void setUpsertMode(boolean upsertMode) {
+    this.upsertMode = upsertMode;
+  }
+
+  public List<String> equalityFields() {
+    return equalityFields;
+  }
+
+  public void setEqualityFields(List<String> equalityFields) {
+    this.equalityFields = equalityFields;
+  }
+}
diff --git a/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
new file mode 100644
index 000000000000..958a1e853983
--- /dev/null
+++ b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  // Required for serialization instantiation
+  DynamicRecordInternal() {}
+
+  DynamicRecordInternal(
+      String tableName,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec spec,
+      int writerKey,
+      boolean upsertMode,
+      List<Integer> equalityFieldsIds) {
+    this.tableName = tableName;
+    this.branch = branch;
+    this.schema = schema;
+    this.spec = spec;
+    this.writerKey = writerKey;
+    this.rowData = rowData;
+    this.upsertMode = upsertMode;
+    this.equalityFieldIds = equalityFieldsIds;
+  }
+
+  public String tableName() {
+    return tableName;
+  }
+
+  public void setTableName(String tableName) {
+    this.tableName = tableName;
+  }
+
+  public String branch() {
+    return branch;
+  }
+
+  public void setBranch(String branch) {
+    this.branch = branch;
+  }
+
+  public Schema schema() {
+    return schema;
+  }
+
+  public void setSchema(Schema schema) {
+    this.schema = schema;
+  }
+
+  public RowData rowData() {
+    return rowData;
+  }
+
+  public void setRowData(RowData rowData) {
+    this.rowData = rowData;
+  }
+
+  public PartitionSpec spec() {
+    return spec;
+  }
+
+  public void setSpec(PartitionSpec spec) {
+    this.spec = spec;
+  }
+
+  public int writerKey() {
+    return writerKey;
+  }
+
+  public void setWriterKey(int writerKey) {
+    this.writerKey = writerKey;
+  }
+
+  public boolean upsertMode() {
+    return upsertMode;
+  }
+
+  public void setUpsertMode(boolean upsertMode) {
+    this.upsertMode = upsertMode;
+  }
+
+  public List<Integer> equalityFields() {
+    return equalityFieldIds;
+  }
+
+  public void setEqualityFieldIds(List<Integer> equalityFieldIds) {
+    this.equalityFieldIds = equalityFieldIds;
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(
+        tableName, branch, schema, spec, writerKey, rowData, upsertMode, equalityFieldIds);
+  }
+
+  @Override
+  public boolean equals(Object other) {
+    if (this == other) {
+      return true;
+    }
+
+    if (other == null || getClass() != other.getClass()) {
+      return false;
+    }
+
+    DynamicRecordInternal that = (DynamicRecordInternal) other;
+    boolean tableFieldsMatch =
+        Objects.equals(tableName, that.tableName)
+            && Objects.equals(branch, that.branch)
+            && schema.schemaId() == that.schema.schemaId()
+            && Objects.equals(spec, that.spec)
+            && writerKey == that.writerKey
+            && upsertMode == that.upsertMode
+            && Objects.equals(equalityFieldIds, that.equalityFieldIds);
+    if (!tableFieldsMatch) {
+      return false;
+    }
+
+    if (rowData.getClass().equals(that.rowData.getClass())) {
+      return Objects.equals(rowData, that.rowData);
+    } else {
+      RowDataSerializer rowDataSerializer = new RowDataSerializer(FlinkSchemaUtil.convert(schema));
+      return rowDataSerializer
+          .toBinaryRow(rowData)
+          .equals(rowDataSerializer.toBinaryRow(that.rowData));
+    }
+  }
+}
diff --git a/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
new file mode 100644
index 000000000000..a6f38f7d6bc9
--- /dev/null
+++ b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionSpecParser;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+@Internal
+class DynamicRecordInternalSerializer extends TypeSerializer<DynamicRecordInternal> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final TableSerializerCache serializerCache;
+  private final boolean writeSchemaAndSpec;
+
+  DynamicRecordInternalSerializer(
+      TableSerializerCache serializerCache, boolean writeSchemaAndSpec) {
+    this.serializerCache = serializerCache;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> duplicate() {
+    return new DynamicRecordInternalSerializer(
+        new TableSerializerCache(serializerCache.catalogLoader(), serializerCache.maximumSize()),
+        writeSchemaAndSpec);
+  }
+
+  @Override
+  public DynamicRecordInternal createInstance() {
+    return new DynamicRecordInternal();
+  }
+
+  @Override
+  public void serialize(DynamicRecordInternal toSerialize, DataOutputView dataOutputView)
+      throws IOException {
+    dataOutputView.writeUTF(toSerialize.tableName());
+    dataOutputView.writeUTF(toSerialize.branch());
+    if (writeSchemaAndSpec) {
+      dataOutputView.writeUTF(SchemaParser.toJson(toSerialize.schema()));
+      dataOutputView.writeUTF(PartitionSpecParser.toJson(toSerialize.spec()));
+    } else {
+      dataOutputView.writeInt(toSerialize.schema().schemaId());
+      dataOutputView.writeInt(toSerialize.spec().specId());
+    }
+
+    dataOutputView.writeInt(toSerialize.writerKey());
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      rowDataSerializer =
+          serializerCache.serializer(
+              toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
+    } else {
+      // Check that the schema id can be resolved. Not strictly necessary for serialization.
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
+          serializerCache.serializerWithSchemaAndSpec(
+              toSerialize.tableName(),
+              toSerialize.schema().schemaId(),
+              toSerialize.spec().specId());
+      rowDataSerializer = serializer.f0;
+    }
+
+    rowDataSerializer.serialize(toSerialize.rowData(), dataOutputView);
+    dataOutputView.writeBoolean(toSerialize.upsertMode());
+    dataOutputView.writeInt(toSerialize.equalityFields().size());
+    for (Integer equalityField : toSerialize.equalityFields()) {
+      dataOutputView.writeInt(equalityField);
+    }
+  }
+
+  @Override
+  public DynamicRecordInternal deserialize(DataInputView dataInputView) throws IOException {
+    String tableName = dataInputView.readUTF();
+    String branch = dataInputView.readUTF();
+
+    final Schema schema;
+    final PartitionSpec spec;
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      schema = SchemaParser.fromJson(dataInputView.readUTF());
+      spec = PartitionSpecParser.fromJson(schema, dataInputView.readUTF());
+      rowDataSerializer = serializerCache.serializer(tableName, schema, spec);
+    } else {
+      Integer schemaId = dataInputView.readInt();
+      Integer specId = dataInputView.readInt();
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec =
+          serializerCache.serializerWithSchemaAndSpec(tableName, schemaId, specId);
+      schema = serializerWithSchemaAndSpec.f1;
+      spec = serializerWithSchemaAndSpec.f2;
+      rowDataSerializer = serializerWithSchemaAndSpec.f0;
+    }
+
+    int writerKey = dataInputView.readInt();
+    RowData rowData = rowDataSerializer.deserialize(dataInputView);
+    boolean upsertMode = dataInputView.readBoolean();
+    int numEqualityFields = dataInputView.readInt();
+    final List<Integer> equalityFieldIds;
+    if (numEqualityFields > 0) {
+      equalityFieldIds = Lists.newArrayList();
+    } else {
+      equalityFieldIds = Collections.emptyList();
+    }
+
+    for (int i = 0; i < numEqualityFields; i++) {
+      equalityFieldIds.add(dataInputView.readInt());
+    }
+
+    return new DynamicRecordInternal(
+        tableName, branch, schema, rowData, spec, writerKey, upsertMode, equalityFieldIds);
+  }
+
+  @Override
+  public DynamicRecordInternal deserialize(DynamicRecordInternal reuse, DataInputView dataInputView)
+      throws IOException {
+    String tableName = dataInputView.readUTF();
+    reuse.setTableName(tableName);
+    String branch = dataInputView.readUTF();
+    reuse.setBranch(branch);
+
+    final Schema schema;
+    final PartitionSpec spec;
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      schema = SchemaParser.fromJson(dataInputView.readUTF());
+      spec = PartitionSpecParser.fromJson(schema, dataInputView.readUTF());
+      reuse.setSchema(schema);
+      reuse.setSpec(spec);
+      rowDataSerializer = serializerCache.serializer(tableName, schema, spec);
+    } else {
+      Integer schemaId = dataInputView.readInt();
+      Integer specId = dataInputView.readInt();
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec =
+          serializerCache.serializerWithSchemaAndSpec(tableName, schemaId, specId);
+      schema = serializerWithSchemaAndSpec.f1;
+      spec = serializerWithSchemaAndSpec.f2;
+      rowDataSerializer = serializerWithSchemaAndSpec.f0;
+    }
+
+    int writerKey = dataInputView.readInt();
+    reuse.setWriterKey(writerKey);
+    RowData rowData = rowDataSerializer.deserialize(dataInputView);
+    boolean upsertMode = dataInputView.readBoolean();
+    int numEqualityFields = dataInputView.readInt();
+    final List<Integer> equalityFieldIds;
+    if (numEqualityFields > 0) {
+      equalityFieldIds = Lists.newArrayList();
+    } else {
+      equalityFieldIds = Collections.emptyList();
+    }
+    for (int i = 0; i < numEqualityFields; i++) {
+      equalityFieldIds.add(dataInputView.readInt());
+    }
+    return new DynamicRecordInternal(
+        tableName, branch, schema, rowData, spec, writerKey, upsertMode, equalityFieldIds);
+  }
+
+  @Override
+  public DynamicRecordInternal copy(DynamicRecordInternal from) {
+    return new DynamicRecordInternal(
+        from.tableName(),
+        from.branch(),
+        from.schema(),
+        from.rowData(),
+        from.spec(),
+        from.writerKey(),
+        from.upsertMode(),
+        from.equalityFields());
+  }
+
+  @Override
+  public DynamicRecordInternal copy(DynamicRecordInternal from, DynamicRecordInternal reuse) {
+    reuse.setTableName(from.tableName());
+    reuse.setBranch(from.branch());
+    reuse.setSchema(from.schema());
+    reuse.setSpec(from.spec());
+    reuse.setWriterKey(from.writerKey());
+    reuse.setRowData(from.rowData());
+    reuse.setUpsertMode(from.upsertMode());
+    reuse.setEqualityFieldIds(from.equalityFields());
+    return reuse;
+  }
+
+  @Override
+  public void copy(DataInputView source, DataOutputView target) throws IOException {
+    serialize(deserialize(source), target);
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+    if (this == obj) {
+      return true;
+    }
+    if (obj instanceof DynamicRecordInternalSerializer) {
+      DynamicRecordInternalSerializer other = (DynamicRecordInternalSerializer) obj;
+      return writeSchemaAndSpec == other.writeSchemaAndSpec;
+    }
+    return false;
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hashCode(writeSchemaAndSpec);
+  }
+
+  @Override
+  public boolean isImmutableType() {
+    return false;
+  }
+
+  @Override
+  public int getLength() {
+    return -1;
+  }
+
+  @Override
+  public TypeSerializerSnapshot<DynamicRecordInternal> snapshotConfiguration() {
+    return new DynamicRecordInternalTypeSerializerSnapshot(writeSchemaAndSpec);
+  }
+
+  public static class DynamicRecordInternalTypeSerializerSnapshot
+      implements TypeSerializerSnapshot<DynamicRecordInternal> {
+
+    private boolean writeSchemaAndSpec;
+
+    // Zero args constructor is required to instantiate this class on restore
+    @SuppressWarnings({"unused", "checkstyle:RedundantModifier"})
+    public DynamicRecordInternalTypeSerializerSnapshot() {}
+
+    DynamicRecordInternalTypeSerializerSnapshot(boolean writeSchemaAndSpec) {
+      this.writeSchemaAndSpec = writeSchemaAndSpec;
+    }
+
+    @Override
+    public int getCurrentVersion() {
+      return 0;
+    }
+
+    @Override
+    public void writeSnapshot(DataOutputView out) throws IOException {
+      out.writeBoolean(writeSchemaAndSpec);
+    }
+
+    @Override
+    public void readSnapshot(int readVersion, DataInputView in, ClassLoader userCodeClassLoader)
+        throws IOException {
+      this.writeSchemaAndSpec = in.readBoolean();
+    }
+
+    @Override
+    public TypeSerializerSchemaCompatibility<DynamicRecordInternal> resolveSchemaCompatibility(
+        TypeSerializerSnapshot<DynamicRecordInternal> oldSerializerSnapshot) {
+      return TypeSerializerSchemaCompatibility.compatibleAsIs();
+    }
+
+    @Override
+    public TypeSerializer<DynamicRecordInternal> restoreSerializer() {
+      // Note: We pass in a null serializer cache which would create issues if we tried to use this
+      // restored serializer, but since we are using {@code
+      // TypeSerializerSchemaCompatibility.compatibleAsIs()} above, this serializer will never be
+      // used. A new one will be created via {@code DynamicRecordInternalType}.
+      return new DynamicRecordInternalSerializer(null, writeSchemaAndSpec);
+    }
+  }
+}
diff --git a/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalType.java b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalType.java
new file mode 100644
index 000000000000..c18c8f670daf
--- /dev/null
+++ b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalType.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializerConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.iceberg.flink.CatalogLoader;
+
+@Internal
+class DynamicRecordInternalType extends TypeInformation<DynamicRecordInternal> {
+
+  private final CatalogLoader catalogLoader;
+  private final boolean writeSchemaAndSpec;
+  private final int cacheSize;
+
+  DynamicRecordInternalType(
+      CatalogLoader catalogLoader, boolean writeSchemaAndSpec, int cacheSize) {
+    this.catalogLoader = catalogLoader;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+    this.cacheSize = cacheSize;
+  }
+
+  @Override
+  public boolean isBasicType() {
+    return false;
+  }
+
+  @Override
+  public boolean isTupleType() {
+    return false;
+  }
+
+  @Override
+  public int getArity() {
+    return 0;
+  }
+
+  @Override
+  public int getTotalFields() {
+    return 1;
+  }
+
+  @Override
+  public Class<DynamicRecordInternal> getTypeClass() {
+    return DynamicRecordInternal.class;
+  }
+
+  @Override
+  public boolean isKeyType() {
+    return false;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> createSerializer(SerializerConfig serializerConfig) {
+    return new DynamicRecordInternalSerializer(
+        new TableSerializerCache(catalogLoader, cacheSize), writeSchemaAndSpec);
+  }
+
+  @Override
+  public String toString() {
+    return getClass().getName();
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    return canEqual(o);
+  }
+
+  @Override
+  public int hashCode() {
+    return getClass().getName().hashCode();
+  }
+
+  @Override
+  public boolean canEqual(Object o) {
+    return o instanceof DynamicRecordInternalType;
+  }
+}
diff --git a/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java
new file mode 100644
index 000000000000..6acf873a94dc
--- /dev/null
+++ b/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableSerializerCache.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.io.Serializable;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.CatalogLoader;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+/**
+ * A Cache which holds Flink's {@link RowDataSerializer} for a given table name and schema. This
+ * avoids re-creating the serializer for a given table schema for every incoming record.
+ *
+ * <p>There is an additional optimization built into this class: Users do not have to supply the
+ * full schema / spec, but can also provide their id. This avoids transferring the schema / spec for
+ * every record. If the id is unknown, the schema / spec will be retrieved from the catalog.
+ *
+ * <p>Note that the caller must ensure that ids are only used for known schemas / specs. The id
+ * optimization must not be used in the update path.
+ */
+@Internal
+class TableSerializerCache implements Serializable {
+
+  private final CatalogLoader catalogLoader;
+  private final int maximumSize;
+  private transient Cache<String, SerializerInfo> serializers;
+
+  TableSerializerCache(CatalogLoader catalogLoader, int maximumSize) {
+    this.catalogLoader = catalogLoader;
+    this.maximumSize = maximumSize;
+  }
+
+  RowDataSerializer serializer(String tableName, Schema schema, PartitionSpec spec) {
+    return serializer(tableName, schema, spec, null, null).f0;
+  }
+
+  Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec(
+      String tableName, Integer schemaId, Integer specId) {
+    return serializer(tableName, null, null, schemaId, specId);
+  }
+
+  private Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer(
+      String tableName,
+      @Nullable Schema unknownSchema,
+      @Nullable PartitionSpec unknownSpec,
+      @Nullable Integer schemaId,
+      @Nullable Integer specId) {
+    Preconditions.checkState(
+        (unknownSchema == null && unknownSpec == null) ^ (schemaId == null && specId == null),
+        "Either the full schema/spec or their ids must be provided.");
+
+    if (serializers == null) {
+      // We need to initialize the cache at the first time
+      this.serializers = Caffeine.newBuilder().maximumSize(maximumSize).build();
+    }
+
+    SerializerInfo info = serializers.get(tableName, SerializerInfo::new);
+    Schema schema = unknownSchema != null ? unknownSchema : info.schemas.get(schemaId);
+    PartitionSpec spec = unknownSpec != null ? unknownSpec : info.specs.get(specId);
+
+    if (schema == null || spec == null) {
+      info.update();
+      schema = info.schemas.get(schemaId);
+      spec = info.specs.get(specId);
+    }
+
+    RowDataSerializer serializer =
+        info.serializers.computeIfAbsent(
+            schema, s -> new RowDataSerializer(FlinkSchemaUtil.convert(s)));
+
+    return Tuple3.of(serializer, schema, spec);
+  }
+
+  CatalogLoader catalogLoader() {
+    return catalogLoader;
+  }
+
+  int maximumSize() {
+    return maximumSize;
+  }
+
+  private class SerializerInfo {
+    private final String tableName;
+    private final Map<Schema, RowDataSerializer> serializers;
+    private Map<Integer, Schema> schemas;
+    private Map<Integer, PartitionSpec> specs;
+
+    SerializerInfo(String tableName) {
+      this.tableName = tableName;
+      this.serializers = Maps.newHashMapWithExpectedSize(2);
+      this.schemas = Maps.newHashMapWithExpectedSize(1);
+      this.specs = Maps.newHashMapWithExpectedSize(0);
+    }
+
+    private void update() {
+      Table table = catalogLoader.loadCatalog().loadTable(TableIdentifier.parse(tableName));
+      schemas = table.schemas();
+      specs = table.specs();
+    }
+  }
+
+  @VisibleForTesting
+  Cache<String, SerializerInfo> getCache() {
+    return serializers;
+  }
+}
diff --git a/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializerTestBase.java b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializerTestBase.java
new file mode 100644
index 000000000000..243b7f90959b
--- /dev/null
+++ b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializerTestBase.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import static org.apache.iceberg.types.Types.NestedField.required;
+
+import java.util.Collections;
+import org.apache.flink.api.common.typeutils.SerializerTestBase;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.HadoopCatalogExtension;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+/**
+ * Test base for DynamicRecordInternalSerializer which allows to instantiate different serializer
+ * version, e.g. with writing the schema itself or just the schema id.
+ */
+abstract class DynamicRecordInternalSerializerTestBase
+    extends SerializerTestBase<DynamicRecordInternal> {
+
+  static final String TABLE = "myTable";
+  static final String BRANCH = "myBranch";
+
+  @RegisterExtension
+  static final HadoopCatalogExtension CATALOG_EXTENSION = new HadoopCatalogExtension("db", TABLE);
+
+  static final Schema SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()),
+          required(2, "data", Types.StringType.get()),
+          required(3, "number", Types.FloatType.get()));
+
+  static final PartitionSpec SPEC = PartitionSpec.builderFor(SCHEMA).bucket("id", 10).build();
+
+  private boolean writeFullSchemaAndSpec;
+
+  DynamicRecordInternalSerializerTestBase(boolean writeFullSchemaAndSpec) {
+    this.writeFullSchemaAndSpec = writeFullSchemaAndSpec;
+  }
+
+  @Override
+  protected TypeSerializer<DynamicRecordInternal> createSerializer() {
+    return new DynamicRecordInternalSerializer(
+        new TableSerializerCache(CATALOG_EXTENSION.catalogLoader(), 1), writeFullSchemaAndSpec);
+  }
+
+  @BeforeEach
+  void before() {
+    CATALOG_EXTENSION.catalog().createTable(TableIdentifier.parse(TABLE), SCHEMA, SPEC);
+  }
+
+  @Override
+  protected DynamicRecordInternal[] getTestData() {
+    GenericRowData rowData = new GenericRowData(3);
+    rowData.setField(0, 123L);
+    rowData.setField(1, StringData.fromString("test"));
+    rowData.setField(2, 1.23f);
+
+    return new DynamicRecordInternal[] {
+      new DynamicRecordInternal(
+          TABLE, BRANCH, SCHEMA, rowData, SPEC, 42, false, Collections.emptyList())
+    };
+  }
+
+  @Override
+  protected Class<DynamicRecordInternal> getTypeClass() {
+    return DynamicRecordInternal.class;
+  }
+
+  @Override
+  protected int getLength() {
+    return -1;
+  }
+}
diff --git a/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchema.java b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchema.java
new file mode 100644
index 000000000000..ab8ce98c3594
--- /dev/null
+++ b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchema.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+/** Test writing DynamicRecord with the full schema */
+class TestDynamicRecordInternalSerializerWriteSchema
+    extends DynamicRecordInternalSerializerTestBase {
+
+  TestDynamicRecordInternalSerializerWriteSchema() {
+    super(true);
+  }
+}
diff --git a/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchemaId.java b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchemaId.java
new file mode 100644
index 000000000000..1d8890546214
--- /dev/null
+++ b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicRecordInternalSerializerWriteSchemaId.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+/** Test writing DynamicRecord with only the schema id. */
+class TestDynamicRecordInternalSerializerWriteSchemaId
+    extends DynamicRecordInternalSerializerTestBase {
+
+  TestDynamicRecordInternalSerializerWriteSchemaId() {
+    super(false);
+  }
+}
diff --git a/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestTableSerializerCache.java b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestTableSerializerCache.java
new file mode 100644
index 000000000000..720a0a582e94
--- /dev/null
+++ b/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestTableSerializerCache.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import static org.apache.iceberg.types.Types.DoubleType;
+import static org.apache.iceberg.types.Types.LongType;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.apache.iceberg.types.Types.StringType;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.function.Supplier;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.CatalogLoader;
+import org.apache.iceberg.flink.HadoopCatalogExtension;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+public class TestTableSerializerCache {
+
+  @RegisterExtension
+  static final HadoopCatalogExtension CATALOG_EXTENSION = new HadoopCatalogExtension("db", "table");
+
+  Schema schema1 = new Schema(23, required(1, "id", LongType.get()));
+
+  Schema schema2 =
+      new Schema(
+          42,
+          required(1, "id", LongType.get()),
+          optional(2, "data", StringType.get()),
+          optional(3, "double", DoubleType.get()));
+
+  TableSerializerCache cache = new TableSerializerCache(CATALOG_EXTENSION.catalogLoader(), 10);
+
+  @Test
+  void testFullSchemaCaching() {
+    Supplier<RowDataSerializer> creator1a =
+        () -> cache.serializer("table", schema1, PartitionSpec.unpartitioned());
+    Supplier<RowDataSerializer> creator1b =
+        () -> cache.serializer("table", schema2, PartitionSpec.unpartitioned());
+    Supplier<RowDataSerializer> creator2 =
+        () -> cache.serializer("table2", schema2, PartitionSpec.unpartitioned());
+
+    RowDataSerializer serializer1a = creator1a.get();
+    RowDataSerializer serializer1b = creator1b.get();
+    RowDataSerializer serializer2 = creator2.get();
+    assertThat(serializer1a).isNotSameAs(serializer1b).isNotSameAs(serializer2);
+
+    assertThat(serializer1a).isSameAs(creator1a.get());
+    assertThat(serializer1b).isSameAs(creator1b.get());
+    assertThat(serializer2).isSameAs(creator2.get());
+  }
+
+  @Test
+  void testCachingWithSchemaLookup() {
+    CatalogLoader catalogLoader = CATALOG_EXTENSION.catalogLoader();
+    cache = new TableSerializerCache(catalogLoader, 10);
+
+    Catalog catalog = catalogLoader.loadCatalog();
+    Table table = catalog.createTable(TableIdentifier.of("table"), schema1);
+
+    Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec =
+        cache.serializerWithSchemaAndSpec(
+            "table", table.schema().schemaId(), PartitionSpec.unpartitioned().specId());
+    assertThat(serializerWithSchemaAndSpec).isNotNull();
+    assertThat(serializerWithSchemaAndSpec.f0).isNotNull();
+    assertThat(serializerWithSchemaAndSpec.f1.sameSchema(table.schema())).isTrue();
+    assertThat(serializerWithSchemaAndSpec.f2).isEqualTo(table.spec());
+
+    Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec2 =
+        cache.serializerWithSchemaAndSpec(
+            "table", table.schema().schemaId(), PartitionSpec.unpartitioned().specId());
+
+    assertThat(serializerWithSchemaAndSpec.f0).isSameAs(serializerWithSchemaAndSpec2.f0);
+    assertThat(serializerWithSchemaAndSpec.f1).isSameAs(serializerWithSchemaAndSpec2.f1);
+    assertThat(serializerWithSchemaAndSpec.f2).isSameAs(serializerWithSchemaAndSpec2.f2);
+  }
+
+  @Test
+  void testCacheEviction() {
+    cache = new TableSerializerCache(CATALOG_EXTENSION.catalogLoader(), 0);
+    assertThat(cache.maximumSize()).isEqualTo(0);
+
+    Supplier<RowDataSerializer> creator1 =
+        () -> cache.serializer("table", schema1, PartitionSpec.unpartitioned());
+    Supplier<RowDataSerializer> creator2 =
+        () -> cache.serializer("table2", schema2, PartitionSpec.unpartitioned());
+
+    RowDataSerializer serializer1 = creator1.get();
+    RowDataSerializer serializer2 = creator2.get();
+
+    cache.getCache().cleanUp();
+    assertThat(serializer1).isNotSameAs(creator1.get());
+    assertThat(serializer2).isNotSameAs(creator2.get());
+  }
+
+  @Test
+  void testCacheSize() {
+    cache = new TableSerializerCache(CATALOG_EXTENSION.catalogLoader(), 1000);
+    assertThat(cache.maximumSize()).isEqualTo(1000);
+  }
+}


Reviews:
user: login: gyfora
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:03:37Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:06:44Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:07:57Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:08:34Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:09:05Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:09:36Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:10:32Z
user: login: gyfora
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T06:29:02Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T10:07:29Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T10:08:39Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T10:12:25Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T10:17:28Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:35:38Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:36:05Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:47:14Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:09:23Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T12:16:48Z
user: login: pvary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T12:17:15Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T14:26:21Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T14:26:30Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T14:26:37Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T14:34:47Z
user: login: mxm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T14:35:44Z
user: login: pvary
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T14:37:04Z
user: login: gyfora
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T18:06:49Z

Review comments:
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: gyfora
body: Should we rename this to `isUpsert` or if it denotes an actual mode use an enum instead?
created_at: 2025-05-07T14:58:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: gyfora
body: Should we rename this to partitionSpec in case some other kind of spec appears in the future?
created_at: 2025-05-07T14:59:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionSpecParser;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+@Internal
+class DynamicRecordInternalSerializer extends TypeSerializer<DynamicRecordInternal> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowDataSerializerCache serializerCache;
+  private final boolean writeSchemaAndSpec;
+
+  DynamicRecordInternalSerializer(
+      RowDataSerializerCache serializerCache, boolean writeSchemaAndSpec) {
+    this.serializerCache = serializerCache;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> duplicate() {
+    return new DynamicRecordInternalSerializer(
+        new RowDataSerializerCache(serializerCache.catalogLoader(), serializerCache.maximumSize()),
+        writeSchemaAndSpec);
+  }
+
+  @Override
+  public DynamicRecordInternal createInstance() {
+    return new DynamicRecordInternal();
+  }
+
+  @Override
+  public void serialize(DynamicRecordInternal toSerialize, DataOutputView dataOutputView)
+      throws IOException {
+    dataOutputView.writeUTF(toSerialize.tableName());
+    dataOutputView.writeUTF(toSerialize.branch());
+    if (writeSchemaAndSpec) {
+      dataOutputView.writeUTF(SchemaParser.toJson(toSerialize.schema()));
+      dataOutputView.writeUTF(PartitionSpecParser.toJson(toSerialize.spec()));
+    } else {
+      dataOutputView.writeInt(toSerialize.schema().schemaId());
+      dataOutputView.writeInt(toSerialize.spec().specId());
+    }
+    dataOutputView.writeInt(toSerialize.writerKey());
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      rowDataSerializer =
+          serializerCache.serializer(
+              toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
+    } else {
+      // Check that the schema id can be resolved. Not strictly necessary for serialization.
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
+          serializerCache.serializerWithSchemaAndSpec(
+              toSerialize.tableName(),
+              toSerialize.schema().schemaId(),
+              toSerialize.spec().specId());
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
user: login: gyfora
body: if not strictly necessary why do we do it? What happens if this fails / why would it fail?
created_at: 2025-05-07T15:01:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: gyfora
body: should this be called partitionSpec in case other specs are added in the future?
created_at: 2025-05-07T15:02:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: gyfora
body: a boolean doesn't really describe a mode , should this be an enum or `isUpsert` maybe?
created_at: 2025-05-07T15:02:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: gyfora
body: should this be distribution or distributionMode? (it is already clashing with upsertMode a little)
created_at: 2025-05-07T15:03:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: mxm
body: I was also leaning towards this name in the beginning, but it's Iceberg convention to use this name across the code base. We can rename though if this is a concern. 
created_at: 2025-05-08T06:06:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: mxm
body: Can do but it's consistent with the coding style. We often omit these verbs from the getters in Iceberg. 
created_at: 2025-05-08T06:07:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: Same as https://github.com/apache/iceberg/pull/12996#discussion_r2078952111
created_at: 2025-05-08T06:08:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: I think it does. If enabled, upsert mode will be used. 
created_at: 2025-05-08T06:09:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: See also https://github.com/apache/iceberg/pull/12996#discussion_r2078953365
created_at: 2025-05-08T06:09:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: Yes, it makes sense to rename to distributionMode. 
created_at: 2025-05-08T06:10:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: gyfora
body: in that case `upsert` or `useUpsertMode` would probably a better name
created_at: 2025-05-08T06:29:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: pvary
body: Only this field is nullable?
Shall we use the annotation consistently?
created_at: 2025-05-08T10:07:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  DynamicRecordInternal() {}
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: pvary
body: Do I remember correctly, that we need this because Flink serialization requires it? Shall we add a comment?
created_at: 2025-05-08T10:08:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionSpecParser;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+@Internal
+class DynamicRecordInternalSerializer extends TypeSerializer<DynamicRecordInternal> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowDataSerializerCache serializerCache;
+  private final boolean writeSchemaAndSpec;
+
+  DynamicRecordInternalSerializer(
+      RowDataSerializerCache serializerCache, boolean writeSchemaAndSpec) {
+    this.serializerCache = serializerCache;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> duplicate() {
+    return new DynamicRecordInternalSerializer(
+        new RowDataSerializerCache(serializerCache.catalogLoader(), serializerCache.maximumSize()),
+        writeSchemaAndSpec);
+  }
+
+  @Override
+  public DynamicRecordInternal createInstance() {
+    return new DynamicRecordInternal();
+  }
+
+  @Override
+  public void serialize(DynamicRecordInternal toSerialize, DataOutputView dataOutputView)
+      throws IOException {
+    dataOutputView.writeUTF(toSerialize.tableName());
+    dataOutputView.writeUTF(toSerialize.branch());
+    if (writeSchemaAndSpec) {
+      dataOutputView.writeUTF(SchemaParser.toJson(toSerialize.schema()));
+      dataOutputView.writeUTF(PartitionSpecParser.toJson(toSerialize.spec()));
+    } else {
+      dataOutputView.writeInt(toSerialize.schema().schemaId());
+      dataOutputView.writeInt(toSerialize.spec().specId());
+    }
+    dataOutputView.writeInt(toSerialize.writerKey());
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      rowDataSerializer =
+          serializerCache.serializer(
+              toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
+    } else {
+      // Check that the schema id can be resolved. Not strictly necessary for serialization.
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
+          serializerCache.serializerWithSchemaAndSpec(
+              toSerialize.tableName(),
+              toSerialize.schema().schemaId(),
+              toSerialize.spec().specId());
+      rowDataSerializer = serializer.f0;
+    }
+    rowDataSerializer.serialize(toSerialize.rowData(), dataOutputView);
+    dataOutputView.writeBoolean(toSerialize.upsertMode());
+    dataOutputView.writeInt(toSerialize.equalityFields().size());
+    for (Integer equalityField : toSerialize.equalityFields()) {
+      dataOutputView.writeInt(equalityField);
+    }
+  }
+
+  @Override
+  public DynamicRecordInternal deserialize(DataInputView dataInputView) throws IOException {
+    String tableName = dataInputView.readUTF();
+    String branch = dataInputView.readUTF();
+
+    final Schema schema;
+    final PartitionSpec spec;
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      schema = SchemaParser.fromJson(dataInputView.readUTF());
+      spec = PartitionSpecParser.fromJson(schema, dataInputView.readUTF());
+      rowDataSerializer = serializerCache.serializer(tableName, schema, spec);
+    } else {
+      Integer schemaId = dataInputView.readInt();
+      Integer specId = dataInputView.readInt();
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec =
+          serializerCache.serializerWithSchemaAndSpec(tableName, schemaId, specId);
+      schema = serializerWithSchemaAndSpec.f1;
+      spec = serializerWithSchemaAndSpec.f2;
+      rowDataSerializer = serializerWithSchemaAndSpec.f0;
+    }
+
+    int writerKey = dataInputView.readInt();
+    RowData rowData = rowDataSerializer.deserialize(dataInputView);
+    boolean upsertMode = dataInputView.readBoolean();
+    int numEqualityFields = dataInputView.readInt();
+    final List<Integer> equalityFieldIds;
+    if (numEqualityFields > 0) {
+      equalityFieldIds = Lists.newArrayList();
+    } else {
+      equalityFieldIds = Collections.emptyList();
+    }
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
user: login: pvary
body: Please follow https://iceberg.apache.org/contribute/?h=cont#block-spacing
created_at: 2025-05-08T10:12:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.io.Serializable;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.CatalogLoader;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+@Internal
+class RowDataSerializerCache implements Serializable {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/RowDataSerializerCache.java
user: login: pvary
body: Can we have a javadoc, description about the expected bahavior?
created_at: 2025-05-08T10:17:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: Correct, only this field is currently nullable / optional. We could add some defaults. I was thinking to add a builder, what do you think?
created_at: 2025-05-09T14:35:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  DynamicRecordInternal() {}
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: mxm
body: I'll add a comment.
created_at: 2025-05-09T14:36:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec spec;
+  private DistributionMode mode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: pvary
body: A builder makes sense to me, as we have many parameters
created_at: 2025-05-09T14:47:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionSpecParser;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+@Internal
+class DynamicRecordInternalSerializer extends TypeSerializer<DynamicRecordInternal> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowDataSerializerCache serializerCache;
+  private final boolean writeSchemaAndSpec;
+
+  DynamicRecordInternalSerializer(
+      RowDataSerializerCache serializerCache, boolean writeSchemaAndSpec) {
+    this.serializerCache = serializerCache;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> duplicate() {
+    return new DynamicRecordInternalSerializer(
+        new RowDataSerializerCache(serializerCache.catalogLoader(), serializerCache.maximumSize()),
+        writeSchemaAndSpec);
+  }
+
+  @Override
+  public DynamicRecordInternal createInstance() {
+    return new DynamicRecordInternal();
+  }
+
+  @Override
+  public void serialize(DynamicRecordInternal toSerialize, DataOutputView dataOutputView)
+      throws IOException {
+    dataOutputView.writeUTF(toSerialize.tableName());
+    dataOutputView.writeUTF(toSerialize.branch());
+    if (writeSchemaAndSpec) {
+      dataOutputView.writeUTF(SchemaParser.toJson(toSerialize.schema()));
+      dataOutputView.writeUTF(PartitionSpecParser.toJson(toSerialize.spec()));
+    } else {
+      dataOutputView.writeInt(toSerialize.schema().schemaId());
+      dataOutputView.writeInt(toSerialize.spec().specId());
+    }
+    dataOutputView.writeInt(toSerialize.writerKey());
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      rowDataSerializer =
+          serializerCache.serializer(
+              toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
+    } else {
+      // Check that the schema id can be resolved. Not strictly necessary for serialization.
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
+          serializerCache.serializerWithSchemaAndSpec(
+              toSerialize.tableName(),
+              toSerialize.schema().schemaId(),
+              toSerialize.spec().specId());
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
user: login: mxm
body: This is basically a sanity-test, to test that looking up the serializer by id on the remote side will work. The remote side won't have the schema available, because it is not written in this branch. If there are any issues, we will know about them on the sender side, as opposed on the receiving side.

I've added a JavaDoc which should clarify things.
created_at: 2025-05-09T15:09:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec partitionSpec;
+  private DistributionMode distributionMode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
+
+  public DynamicRecord(
+      TableIdentifier tableIdentifier,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec partitionSpec,
+      DistributionMode distributionMode,
+      int writeParallelism) {
+    this.tableIdentifier = tableIdentifier;
+    this.branch = branch;
+    this.schema = schema;
+    this.partitionSpec = partitionSpec;
+    this.rowData = rowData;
+    this.distributionMode = distributionMode;
+    this.writeParallelism = writeParallelism;
+  }
+
+  public TableIdentifier tableIdentifier() {
+    return tableIdentifier;
+  }
+
+  public void setTableIdentifier(TableIdentifier tableIdentifier) {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: pvary
body: Do we need these setters, if we have a builder?
created_at: 2025-05-12T12:16:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  // Required for serialization instantiation
+  DynamicRecordInternal() {}
+
+  DynamicRecordInternal(
+      String tableName,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec spec,
+      int writerKey,
+      boolean upsertMode,
+      List<Integer> equalityFieldsIds) {
+    this.tableName = tableName;
+    this.branch = branch;
+    this.schema = schema;
+    this.spec = spec;
+    this.writerKey = writerKey;
+    this.rowData = rowData;
+    this.upsertMode = upsertMode;
+    this.equalityFieldIds = equalityFieldsIds;
+  }
+
+  public String tableName() {
+    return tableName;
+  }
+
+  public void setTableName(String tableName) {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: pvary
body: Do we need this setters?
created_at: 2025-05-12T12:17:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import java.io.Serializable;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.CatalogLoader;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+@Internal
+class RowDataSerializerCache implements Serializable {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/RowDataSerializerCache.java
user: login: mxm
body: Added.
created_at: 2025-05-12T14:26:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PartitionSpecParser;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+@Internal
+class DynamicRecordInternalSerializer extends TypeSerializer<DynamicRecordInternal> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowDataSerializerCache serializerCache;
+  private final boolean writeSchemaAndSpec;
+
+  DynamicRecordInternalSerializer(
+      RowDataSerializerCache serializerCache, boolean writeSchemaAndSpec) {
+    this.serializerCache = serializerCache;
+    this.writeSchemaAndSpec = writeSchemaAndSpec;
+  }
+
+  @Override
+  public TypeSerializer<DynamicRecordInternal> duplicate() {
+    return new DynamicRecordInternalSerializer(
+        new RowDataSerializerCache(serializerCache.catalogLoader(), serializerCache.maximumSize()),
+        writeSchemaAndSpec);
+  }
+
+  @Override
+  public DynamicRecordInternal createInstance() {
+    return new DynamicRecordInternal();
+  }
+
+  @Override
+  public void serialize(DynamicRecordInternal toSerialize, DataOutputView dataOutputView)
+      throws IOException {
+    dataOutputView.writeUTF(toSerialize.tableName());
+    dataOutputView.writeUTF(toSerialize.branch());
+    if (writeSchemaAndSpec) {
+      dataOutputView.writeUTF(SchemaParser.toJson(toSerialize.schema()));
+      dataOutputView.writeUTF(PartitionSpecParser.toJson(toSerialize.spec()));
+    } else {
+      dataOutputView.writeInt(toSerialize.schema().schemaId());
+      dataOutputView.writeInt(toSerialize.spec().specId());
+    }
+    dataOutputView.writeInt(toSerialize.writerKey());
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      rowDataSerializer =
+          serializerCache.serializer(
+              toSerialize.tableName(), toSerialize.schema(), toSerialize.spec());
+    } else {
+      // Check that the schema id can be resolved. Not strictly necessary for serialization.
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializer =
+          serializerCache.serializerWithSchemaAndSpec(
+              toSerialize.tableName(),
+              toSerialize.schema().schemaId(),
+              toSerialize.spec().specId());
+      rowDataSerializer = serializer.f0;
+    }
+    rowDataSerializer.serialize(toSerialize.rowData(), dataOutputView);
+    dataOutputView.writeBoolean(toSerialize.upsertMode());
+    dataOutputView.writeInt(toSerialize.equalityFields().size());
+    for (Integer equalityField : toSerialize.equalityFields()) {
+      dataOutputView.writeInt(equalityField);
+    }
+  }
+
+  @Override
+  public DynamicRecordInternal deserialize(DataInputView dataInputView) throws IOException {
+    String tableName = dataInputView.readUTF();
+    String branch = dataInputView.readUTF();
+
+    final Schema schema;
+    final PartitionSpec spec;
+    final RowDataSerializer rowDataSerializer;
+    if (writeSchemaAndSpec) {
+      schema = SchemaParser.fromJson(dataInputView.readUTF());
+      spec = PartitionSpecParser.fromJson(schema, dataInputView.readUTF());
+      rowDataSerializer = serializerCache.serializer(tableName, schema, spec);
+    } else {
+      Integer schemaId = dataInputView.readInt();
+      Integer specId = dataInputView.readInt();
+      Tuple3<RowDataSerializer, Schema, PartitionSpec> serializerWithSchemaAndSpec =
+          serializerCache.serializerWithSchemaAndSpec(tableName, schemaId, specId);
+      schema = serializerWithSchemaAndSpec.f1;
+      spec = serializerWithSchemaAndSpec.f2;
+      rowDataSerializer = serializerWithSchemaAndSpec.f0;
+    }
+
+    int writerKey = dataInputView.readInt();
+    RowData rowData = rowDataSerializer.deserialize(dataInputView);
+    boolean upsertMode = dataInputView.readBoolean();
+    int numEqualityFields = dataInputView.readInt();
+    final List<Integer> equalityFieldIds;
+    if (numEqualityFields > 0) {
+      equalityFieldIds = Lists.newArrayList();
+    } else {
+      equalityFieldIds = Collections.emptyList();
+    }
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java
user: login: mxm
body: Addressed.
created_at: 2025-05-12T14:26:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  DynamicRecordInternal() {}
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: mxm
body: Added.
created_at: 2025-05-12T14:26:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.flink.table.data.RowData;
+import org.apache.iceberg.DistributionMode;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/** A DynamicRecord contains RowData alongside with the Iceberg table metadata. */
+public class DynamicRecord {
+
+  private TableIdentifier tableIdentifier;
+  private String branch;
+  private Schema schema;
+  private RowData rowData;
+  private PartitionSpec partitionSpec;
+  private DistributionMode distributionMode;
+  private int writeParallelism;
+  private boolean upsertMode;
+  @Nullable private List<String> equalityFields;
+
+  public DynamicRecord(
+      TableIdentifier tableIdentifier,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec partitionSpec,
+      DistributionMode distributionMode,
+      int writeParallelism) {
+    this.tableIdentifier = tableIdentifier;
+    this.branch = branch;
+    this.schema = schema;
+    this.partitionSpec = partitionSpec;
+    this.rowData = rowData;
+    this.distributionMode = distributionMode;
+    this.writeParallelism = writeParallelism;
+  }
+
+  public TableIdentifier tableIdentifier() {
+    return tableIdentifier;
+  }
+
+  public void setTableIdentifier(TableIdentifier tableIdentifier) {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecord.java
user: login: mxm
body: We wouldn't. I'm not sure though we should remove these methods, as they allow DynamicRecord to be reused. If we add the builder, that won't be possible anymore.
created_at: 2025-05-12T14:34:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink.dynamic;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+
+@Internal
+class DynamicRecordInternal {
+
+  private String tableName;
+  private String branch;
+  private Schema schema;
+  private PartitionSpec spec;
+  private int writerKey;
+  private RowData rowData;
+  private boolean upsertMode;
+  private List<Integer> equalityFieldIds;
+
+  // Required for serialization instantiation
+  DynamicRecordInternal() {}
+
+  DynamicRecordInternal(
+      String tableName,
+      String branch,
+      Schema schema,
+      RowData rowData,
+      PartitionSpec spec,
+      int writerKey,
+      boolean upsertMode,
+      List<Integer> equalityFieldsIds) {
+    this.tableName = tableName;
+    this.branch = branch;
+    this.schema = schema;
+    this.spec = spec;
+    this.writerKey = writerKey;
+    this.rowData = rowData;
+    this.upsertMode = upsertMode;
+    this.equalityFieldIds = equalityFieldsIds;
+  }
+
+  public String tableName() {
+    return tableName;
+  }
+
+  public void setTableName(String tableName) {
path: flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternal.java
user: login: mxm
body: We currently use these setters here to allow for Flink's object reuse mode: https://github.com/apache/iceberg/blob/bf02ad6395e4e64e9c369bb6f43c12e16d2c1702/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java#L204
created_at: 2025-05-12T14:35:44Z
author_association: CONTRIBUTOR

PR comments:
user: login: mxm
created_at: 2025-05-08T06:15:03Z
body: Thanks for the review @gyfora! I think it makes sense to rename the API-facing fields / getters / setters to avoid confusion for users. 
author_association: CONTRIBUTOR
user: login: gyfora
created_at: 2025-05-08T06:26:12Z
body: > Thanks for the review @gyfora! I think it makes sense to rename the API-facing fields / getters / setters to avoid confusion for users.

I am not yet aware of all the conventions here, @pvary maybe you could chime in related to the naming and then I will learn once and for all :D 
author_association: CONTRIBUTOR
user: login: pvary
created_at: 2025-05-09T14:46:25Z
body: > > Thanks for the review @gyfora! I think it makes sense to rename the API-facing fields / getters / setters to avoid confusion for users.
> 
> I am not yet aware of all the conventions here, @pvary maybe you could chime in related to the naming and then I will learn once and for all :D

It might be strange for new developers, but we always omit `get`, `set`, `is` from the method names.
Here is the guide: https://iceberg.apache.org/contribute/#iceberg-code-contribution-guidelines
The only exceptions are the overrides for external APIs
author_association: CONTRIBUTOR
user: login: gyfora
created_at: 2025-05-09T14:49:02Z
body: > > > Thanks for the review @gyfora! I think it makes sense to rename the API-facing fields / getters / setters to avoid confusion for users.
> > 
> > 
> > I am not yet aware of all the conventions here, @pvary maybe you could chime in related to the naming and then I will learn once and for all :D
> 
> It might be strange for new developers, but we always omit `get`, `set`, `is` from the method names. Here is the guide: https://iceberg.apache.org/contribute/#iceberg-code-contribution-guidelines The only exceptions are the overrides for external APIs

In general I get the idea, but my particular concern was related to `upgradeMode` the convention clearly doesn't work well with a name like this as it's immediately confusing when you have other `xxMode` fields that are enums etc.
author_association: CONTRIBUTOR
user: login: pvary
created_at: 2025-05-09T15:16:19Z
body: > > > > Thanks for the review @gyfora! I think it makes sense to rename the API-facing fields / getters / setters to avoid confusion for users.
> > > 
> > > 
> > > I am not yet aware of all the conventions here, @pvary maybe you could chime in related to the naming and then I will learn once and for all :D
> > 
> > 
> > It might be strange for new developers, but we always omit `get`, `set`, `is` from the method names. Here is the guide: https://iceberg.apache.org/contribute/#iceberg-code-contribution-guidelines The only exceptions are the overrides for external APIs
> 
> In general I get the idea, but my particular concern was related to `upgradeMode` the convention clearly doesn't work well with a name like this as it's immediately confusing when you have other `xxMode` fields that are enums etc.

I assume this is `upsertMode`?
While I understand your concern, the IcebergSink contains `upsertMode`, and this convention is used throughout the Flink code, so I would stick to it.
author_association: CONTRIBUTOR
user: login: mxm
created_at: 2025-05-09T15:17:18Z
body: I've pushed an update to address the comments.

On the name discussion: I think this is all just convention. Every community has its own styles. I don't think either way makes more sense. `upsertMode` makes perfect sense to me, `isUpsert` not so much because not every record produces an upsert, but `isUpsertMode` makes just as much sense, even though the boolean type is next to the name. 

The most important reason is consistency. All existing Flink Iceberg sinks use that name. I don't see a strong case to deviate from it. 

I did rename `mode` to `distributionMode` and `spec` to `partitionSpec`. 
author_association: CONTRIBUTOR
user: login: mxm
created_at: 2025-05-09T16:29:50Z
body: (Spark compile failed in build https://github.com/apache/iceberg/actions/runs/14932704968/job/41952736107?pr=12996)
author_association: CONTRIBUTOR
user: login: mxm
created_at: 2025-05-12T11:11:49Z
body: (rebased and squashed commits)
author_association: CONTRIBUTOR
user: login: pvary
created_at: 2025-05-14T20:09:29Z
body: Merged to main.
Thanks @mxm for the PR and @gyfora for the review!
author_association: CONTRIBUTOR
user: login: mxm
created_at: 2025-05-15T08:46:08Z
body: Thanks @pvary @gyfora for reviewing! Thanks @pvary for the merge!
author_association: CONTRIBUTOR