Repository: apache/beam
Pull request #34140: [Java] Added Metrics Configuration Support to Iceberg Data Writers
Author: login: Suvrat1629
Labels: name: java
name: io
name: Next Action: Reviewers
Created: 2025-03-02T09:29:06Z | Merged: 2025-03-12T16:51:49Z

This PR adds support for configuring metrics in Iceberg data writers.Also added tests for the same.

Fixes #34112

------------------------

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

 - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
 - [ ] Update `CHANGES.md` with noteworthy changes.
 - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).

To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)

GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)

See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.


Linked issue #34112: [Feature Request]: [IcebergIO] Configure data writers to track metrics
### What would you like to happen?

We're configuring our Iceberg data writers without a metrics config! So we're losing out on writing important statistics for each data file (statistics that help with query planning).

This can be easily added over [here](https://github.com/apache/beam/blob/6fbdaa23b3b14c26d378c35915c4a1790cf92dc6/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java#L93) with:
```java
   .metricsConfig(metricsConfig)
```

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Linked issue comments: user: login: Suvrat1629
created_at: 2025-03-01T13:02:30Z
body: @ahmedabu98 Is this issue available to work on?
author_association: CONTRIBUTOR
user: login: ahmedabu98
created_at: 2025-03-01T13:18:12Z
body: Hey @Suvrat1629! Yeah give it a shot and tag me in the PR to review it. Let's make sure we add some unit testing that columns marked for statistic collection are respected.

Take a look at the following table options for enabling metrics on table columns:
- write.metadata.metrics.default
- write.metadata.metrics.column.<column-name>

author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-01T13:40:42Z
body: @ahmedabu98 
Thank you i will give it a shot but since I am new to the codebase so before i start working I would like some guidance and like to verify my approach:
- first we add the `metricsConfig(metricsConfig)` to the `RecordWriter.java` file
- then update all the corresponding `Write*.java` files.

does this approach fit right? Do I need to look into anymore files?
author_association: CONTRIBUTOR
user: login: ahmedabu98
created_at: 2025-03-01T13:46:26Z
body: I think it'll end up being a lot simpler than that. `RecordWriter` is the essential piece that actually writes data. The other `Write*` files just take care of moving data around. The essential piece that actually writes data to files is `RecordWriter`. We just need to create a metrics config (something like `MetricsConfig metricsConfig = MetricsConfig.forTable(table);`) and pass it into the data writers
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-01T13:56:49Z
body: @ahmedabu98 So I need to add `MetricsConfig metricsConfig = MetricsConfig.forTable(table);` and then add test cases for the same?
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-01T19:16:03Z
body: @ahmedabu98 Since now that I have made the necessary change as you told me, do i add a new Test case file for the tests or make changes to and existing one?
author_association: CONTRIBUTOR
user: login: ahmedabu98
created_at: 2025-03-01T23:27:52Z
body: You can add a new test case to an existing class: RecordWriterManagerTest.java
author_association: CONTRIBUTOR
user: login: liferoad
created_at: 2025-03-05T14:31:06Z
body: @ahmedabu98 is this the release blocker? If so, can we get this done before the cut?
author_association: CONTRIBUTOR

Diff:
diff --git a/.github/trigger_files/IO_Iceberg_Integration_Tests.json b/.github/trigger_files/IO_Iceberg_Integration_Tests.json
index 7ab7bcd9a9c6..37dd25bf9029 100644
--- a/.github/trigger_files/IO_Iceberg_Integration_Tests.json
+++ b/.github/trigger_files/IO_Iceberg_Integration_Tests.json
@@ -1,4 +1,4 @@
 {
     "comment": "Modify this file in a trivial way to cause this test suite to run.",
-    "modification": 2
+    "modification": 3
 }
diff --git a/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java b/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java
index 0d81d868c49f..6d53c41f980d 100644
--- a/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java
+++ b/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java
@@ -22,6 +22,7 @@
 import org.apache.beam.sdk.metrics.Metrics;
 import org.apache.iceberg.DataFile;
 import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
 import org.apache.iceberg.PartitionKey;
 import org.apache.iceberg.Table;
 import org.apache.iceberg.avro.Avro;
@@ -59,6 +60,9 @@ class RecordWriter {
       throws IOException {
     this.table = table;
     this.fileFormat = fileFormat;
+
+    MetricsConfig metricsConfig = MetricsConfig.forTable(table);
+
     if (table.spec().isUnpartitioned()) {
       absoluteFilename =
           fileFormat.addExtension(table.locationProvider().newDataLocation(filename));
@@ -80,6 +84,7 @@ class RecordWriter {
                 .schema(table.schema())
                 .withSpec(table.spec())
                 .withPartition(partitionKey)
+                .metricsConfig(metricsConfig)
                 .overwrite()
                 .build();
         break;
@@ -90,6 +95,7 @@ class RecordWriter {
                 .schema(table.schema())
                 .withSpec(table.spec())
                 .withPartition(partitionKey)
+                .metricsConfig(metricsConfig)
                 .overwrite()
                 .build();
         break;
diff --git a/sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java b/sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
index eb10722c263f..5d93f10a43d1 100644
--- a/sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
+++ b/sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.io.iceberg;
 
+import static java.nio.charset.StandardCharsets.UTF_8;
 import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.containsInAnyOrder;
@@ -24,12 +25,13 @@
 import static org.hamcrest.Matchers.either;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
 import static org.junit.Assert.assertThrows;
 import static org.junit.Assert.assertTrue;
 
 import java.io.IOException;
 import java.net.URLEncoder;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.time.LocalDate;
 import java.time.LocalDateTime;
 import java.util.ArrayList;
@@ -46,15 +48,19 @@
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
 import org.apache.commons.lang3.RandomStringUtils;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
 import org.apache.iceberg.DataFile;
 import org.apache.iceberg.FileFormat;
 import org.apache.iceberg.PartitionField;
 import org.apache.iceberg.PartitionKey;
 import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Snapshot;
 import org.apache.iceberg.Table;
 import org.apache.iceberg.catalog.Catalog;
 import org.apache.iceberg.catalog.TableIdentifier;
 import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.types.Conversions;
+import org.apache.iceberg.types.Types;
 import org.checkerframework.checker.nullness.qual.Nullable;
 import org.joda.time.DateTime;
 import org.joda.time.DateTimeZone;
@@ -153,7 +159,8 @@ public void testCreateNewWriterForEachDestination() throws IOException {
     assertEquals(3, writerManager.openWriters);
 
     // dest4
-    // This is a new destination, but the writer manager is saturated with 3 writers. reject the
+    // This is a new destination, but the writer manager is saturated with 3
+    // writers. reject the
     // record
     row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
     writeSuccess = writerManager.write(dest4, row);
@@ -161,7 +168,8 @@ public void testCreateNewWriterForEachDestination() throws IOException {
     assertEquals(3, writerManager.openWriters);
 
     // dest3, partition: [aaa, false]
-    // new partition, but the writer manager is saturated with 3 writers. reject the record
+    // new partition, but the writer manager is saturated with 3 writers. reject the
+    // record
     row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", false).build();
     writeSuccess = writerManager.write(dest3, row);
     assertFalse(writeSuccess);
@@ -214,7 +222,8 @@ public void testCreateNewWriterForEachPartition() throws IOException {
     assertEquals(3, writerManager.openWriters);
 
     // partition: [aaa, false].
-    // The writerManager is already saturated with three writers. This record is rejected.
+    // The writerManager is already saturated with three writers. This record is
+    // rejected.
     row = Row.withSchema(BEAM_SCHEMA).addValues(5, "aaa123", false).build();
     writeSuccess = writerManager.write(windowedDestination, row);
     assertFalse(writeSuccess);
@@ -508,7 +517,7 @@ public void testIdentityPartitioning() throws IOException {
     for (Schema.Field field : primitiveTypeSchema.getFields()) {
       Object val = checkStateNotNull(row.getValue(field.getName()));
       if (dateTypes.contains(field.getName())) {
-        val = URLEncoder.encode(val.toString(), StandardCharsets.UTF_8.toString());
+        val = URLEncoder.encode(val.toString(), UTF_8.toString());
       }
       expectedPartitions.add(field.getName() + "=" + val);
     }
@@ -719,4 +728,169 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA); // Unpartitioned table
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    IcebergDestination destination =
+        IcebergDestination.builder()
+            .setTableIdentifier(tableId)
+            .setFileFormat(FileFormat.PARQUET)
+            .build();
+    WindowedValue<IcebergDestination> singleDestination =
+        WindowedValue.valueInGlobalWindow(destination);
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row3));
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+    assertEquals(
+        "Should have exactly one data file for an unpartitioned table", 1, dataFiles.size());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      AppendFiles appendFiles = table.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
+      }
+      appendFiles.commit();
+    }
+    table.refresh();
+
+    Snapshot snapshot = table.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
+
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    assertNotNull("Value counts should not be null", valueCounts);
+    assertTrue("Value counts should exist for id (column 1)", valueCounts.containsKey(1));
+    assertEquals("Value count for id should be 3 in this file", 3L, valueCounts.get(1).longValue());
+    assertTrue("Value counts should exist for name (column 2)", valueCounts.containsKey(2));
+    assertEquals(
+        "Value count for name should be 3 in this file", 3L, valueCounts.get(2).longValue());
+
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+    Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
+    Map<Integer, Long> columnSizes = dataFile.columnSizes();
+    for (int i = 1; i <= ICEBERG_SCHEMA.columns().size(); i++) {
+      assertTrue(
+          "Null value counts should be collected for column " + i, nullValueCounts.containsKey(i));
+      assertTrue("Column sizes should be collected for column " + i, columnSizes.containsKey(i));
+    }
+  }
+
+  @Test
+  public void testDefaultMetrics() throws IOException {
+    TableIdentifier tableId = TableIdentifier.of("default", "test_default_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA);
+    table.updateProperties().set("write.metadata.metrics.default", "full").commit();
+
+    IcebergDestination destination =
+        IcebergDestination.builder()
+            .setTableIdentifier(tableId)
+            .setFileFormat(FileFormat.PARQUET)
+            .build();
+    WindowedValue<IcebergDestination> singleDestination =
+        WindowedValue.valueInGlobalWindow(destination);
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(singleDestination, row3));
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+    assertEquals(
+        "Should have exactly one data file for an unpartitioned table", 1, dataFiles.size());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      AppendFiles appendFiles = table.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
+      }
+      appendFiles.commit();
+    }
+    table.refresh();
+
+    Snapshot snapshot = table.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
+
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+    assertNotNull(dataFile.lowerBounds());
+    assertNotNull(dataFile.upperBounds());
+
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
+    Map<Integer, Long> columnSizes = dataFile.columnSizes();
+    Map<Integer, ByteBuffer> lowerBounds = dataFile.lowerBounds();
+    Map<Integer, ByteBuffer> upperBounds = dataFile.upperBounds();
+
+    for (int i = 1; i <= ICEBERG_SCHEMA.columns().size(); i++) {
+      assertTrue("Value counts should be collected for column " + i, valueCounts.containsKey(i));
+      long expectedCount = 3L;
+      assertEquals(
+          "Value count for column " + i + " should match total values in this file",
+          expectedCount,
+          valueCounts.get(i).longValue());
+
+      assertTrue(
+          "Null value counts should be collected for column " + i, nullValueCounts.containsKey(i));
+      assertTrue("Column sizes should be collected for column " + i, columnSizes.containsKey(i));
+      assertTrue("Lower bounds should be collected for column " + i, lowerBounds.containsKey(i));
+      assertTrue("Upper bounds should be collected for column " + i, upperBounds.containsKey(i));
+
+      ByteBuffer lower = lowerBounds.get(i);
+      ByteBuffer upper = upperBounds.get(i);
+      assertNotNull("Lower bound for column " + i + " should not be null", lower);
+      assertNotNull("Upper bound for column " + i + " should not be null", upper);
+
+      switch (i) {
+        case 1:
+          Integer lowerId = Conversions.fromByteBuffer(Types.IntegerType.get(), lower);
+          Integer upperId = Conversions.fromByteBuffer(Types.IntegerType.get(), upper);
+          assertEquals("Lower bound for id should be min value", 1, lowerId.intValue());
+          assertEquals("Upper bound for id should be max value", 3, upperId.intValue());
+          break;
+        case 2:
+          CharSequence lowerName = Conversions.fromByteBuffer(Types.StringType.get(), lower);
+          CharSequence upperName = Conversions.fromByteBuffer(Types.StringType.get(), upper);
+          assertEquals("Lower bound for name should be min value", "aaa", lowerName.toString());
+          assertEquals("Upper bound for name should be max value", "ccc", upperName.toString());
+          break;
+        case 3:
+          Boolean lowerBool = Conversions.fromByteBuffer(Types.BooleanType.get(), lower);
+          Boolean upperBool = Conversions.fromByteBuffer(Types.BooleanType.get(), upper);
+          assertEquals("Lower bound for bool should be min value", false, lowerBool);
+          assertEquals("Upper bound for bool should be max value", true, upperBool);
+          break;
+        default:
+          throw new IllegalStateException("Unexpected column index: " + i);
+      }
+    }
+  }
 }


Reviews:
user: login: ahmedabu98
body: thanks @Suvrat1629

the fix looks good, but i have some nitpicks for the tests
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T21:22:47Z
user: login: ahmedabu98
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T12:30:51Z
user: login: ahmedabu98
body: Thanks, this LGTM

Last thing would be to trigger integration tests by pushing a new commit with a trivial change to this file:
https://github.com/apache/beam/blob/master/.github/trigger_files/IO_Iceberg_Integration_Tests.json
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T14:16:20Z

Review comments:
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: Btw we can continue using the same `table` instance above -- we don't have to loading it again or refresh
created_at: 2025-03-04T21:15:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable =
+        catalog.loadTable(
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: similar to previous comment, just use the same `table` 
created_at: 2025-03-04T21:15:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable =
+        catalog.loadTable(
+            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected only for specified columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: can we also check the count for each column exists and is correct?
created_at: 2025-03-04T21:18:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable =
+        catalog.loadTable(
+            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected only for specified columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+  }
+
+  @Test
+  public void testDefaultMetrics() throws IOException {
+    // Set up table with default metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_default_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table.updateProperties().set("write.metadata.metrics.default", "full").commit();
+
+    // Create a RecordWriterManager
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    // Write a row
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    // Manually commit the data files to the Iceberg table
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable = catalog.loadTable(TableIdentifier.of("default", "test_default_metrics"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected for all columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+
+    // Verify metrics are collected for all columns
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
+    Map<Integer, Long> columnSizes = dataFile.columnSizes();
+
+    for (int i = 1; i <= ICEBERG_SCHEMA.columns().size(); i++) {
+      assertTrue("Value counts should be collected for column " + i, valueCounts.containsKey(i));
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: we can also check the value of valueCounts.get(i) 
created_at: 2025-03-04T21:20:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable =
+        catalog.loadTable(
+            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected only for specified columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+  }
+
+  @Test
+  public void testDefaultMetrics() throws IOException {
+    // Set up table with default metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_default_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table.updateProperties().set("write.metadata.metrics.default", "full").commit();
+
+    // Create a RecordWriterManager
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    // Write a row
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: for both of these tests, can we write multiple rows for good measure?
created_at: 2025-03-04T21:21:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -719,4 +725,113 @@ public void close() throws IOException {
       throw new IOException("I am failing!");
     }
   }
+
+  @Test
+  public void testColumnSpecificMetricsCollection() throws IOException {
+    // Set up table with column-specific metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_column_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table
+        .updateProperties()
+        .set("write.metadata.metrics.column.id", "counts")
+        .set("write.metadata.metrics.column.name", "counts")
+        .commit();
+
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable =
+        catalog.loadTable(
+            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected only for specified columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+  }
+
+  @Test
+  public void testDefaultMetrics() throws IOException {
+    // Set up table with default metrics enabled
+    TableIdentifier tableId = TableIdentifier.of("default", "test_default_metrics");
+    Table table = warehouse.createTable(tableId, ICEBERG_SCHEMA, PARTITION_SPEC);
+    table.updateProperties().set("write.metadata.metrics.default", "full").commit();
+
+    // Create a RecordWriterManager
+    RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
+
+    // Write a row
+    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    boolean writeSuccess = writerManager.write(windowedDestination, row);
+    assertTrue("Write operation should succeed", writeSuccess);
+    writerManager.close();
+
+    // Manually commit the data files to the Iceberg table
+    Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
+        writerManager.getSerializableDataFiles();
+    assertFalse("Data files should not be empty", dataFiles.isEmpty());
+
+    for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
+        dataFiles.entrySet()) {
+      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
+      AppendFiles appendFiles = tableToCommit.newAppend();
+      for (SerializableDataFile dataFile : entry.getValue()) {
+        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+      }
+      appendFiles.commit();
+      tableToCommit.refresh();
+    }
+
+    // Verify the correct table (where data was written)
+    Table dataTable = catalog.loadTable(TableIdentifier.of("default", "test_default_metrics"));
+    dataTable.refresh();
+
+    // Verify that a snapshot exists
+    Snapshot snapshot = dataTable.currentSnapshot();
+    assertNotNull("Table should have a snapshot after writing data", snapshot);
+
+    // Verify metrics are collected for all columns
+    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    assertNotNull(dataFile.valueCounts());
+    assertNotNull(dataFile.nullValueCounts());
+    assertNotNull(dataFile.columnSizes());
+
+    // Verify metrics are collected for all columns
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
+    Map<Integer, Long> columnSizes = dataFile.columnSizes();
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: also check `.lowerBounds()` and `.upperBounds()` and their values
created_at: 2025-03-04T21:21:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -80,6 +84,7 @@ class RecordWriter {
                 .schema(table.schema())
                 .withSpec(table.spec())
                 .withPartition(partitionKey)
+                .metricsConfig(metricsConfig) // Pass the MetricsConfig
path: sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java
user: login: ahmedabu98
body: nit: the comment is unnecessary
created_at: 2025-03-04T21:22:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -739,39 +740,45 @@ public void testColumnSpecificMetricsCollection() throws IOException {
 
     RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
 
-    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
-    boolean writeSuccess = writerManager.write(windowedDestination, row);
-    assertTrue("Write operation should succeed", writeSuccess);
+    // Write multiple rows
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row3));
     writerManager.close();
 
     Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
         writerManager.getSerializableDataFiles();
     assertFalse("Data files should not be empty", dataFiles.isEmpty());
 
+    // Commit using the same table instance
     for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
         dataFiles.entrySet()) {
-      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
-      AppendFiles appendFiles = tableToCommit.newAppend();
+      AppendFiles appendFiles = table.newAppend();
       for (SerializableDataFile dataFile : entry.getValue()) {
-        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
       }
       appendFiles.commit();
-      tableToCommit.refresh();
     }
+    table.refresh(); // Single refresh after all commits
 
-    // Verify the correct table (where data was written)
-    Table dataTable =
-        catalog.loadTable(
-            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
-    dataTable.refresh();
-
-    // Verify that a snapshot exists
-    Snapshot snapshot = dataTable.currentSnapshot();
+    // Verify using the same table
+    Snapshot snapshot = table.currentSnapshot();
     assertNotNull("Table should have a snapshot after writing data", snapshot);
 
-    // Verify metrics are collected only for specified columns
-    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
-    assertNotNull(dataFile.valueCounts());
+    // Verify metrics are collected for specified columns in one file
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    assertNotNull("Value counts should not be null", valueCounts);
+    assertTrue("Value counts should exist for id (column 1)", valueCounts.containsKey(1));
+    assertEquals("Value count for id should be 1 in this file", 1L, valueCounts.get(1).longValue());
+    assertTrue("Value counts should exist for name (column 2)", valueCounts.containsKey(2));
+    assertEquals(
+        "Value count for name should be 1 in this file", 1L, valueCounts.get(2).longValue());
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: We're writing 3 records, so shouldn't these value counts equal 3? 
created_at: 2025-03-12T12:25:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -739,39 +740,45 @@ public void testColumnSpecificMetricsCollection() throws IOException {
 
     RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
 
-    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
-    boolean writeSuccess = writerManager.write(windowedDestination, row);
-    assertTrue("Write operation should succeed", writeSuccess);
+    // Write multiple rows
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row3));
     writerManager.close();
 
     Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
         writerManager.getSerializableDataFiles();
     assertFalse("Data files should not be empty", dataFiles.isEmpty());
 
+    // Commit using the same table instance
     for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
         dataFiles.entrySet()) {
-      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
-      AppendFiles appendFiles = tableToCommit.newAppend();
+      AppendFiles appendFiles = table.newAppend();
       for (SerializableDataFile dataFile : entry.getValue()) {
-        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
       }
       appendFiles.commit();
-      tableToCommit.refresh();
     }
+    table.refresh(); // Single refresh after all commits
 
-    // Verify the correct table (where data was written)
-    Table dataTable =
-        catalog.loadTable(
-            TableIdentifier.of("default", "table_testColumnSpecificMetricsCollection"));
-    dataTable.refresh();
-
-    // Verify that a snapshot exists
-    Snapshot snapshot = dataTable.currentSnapshot();
+    // Verify using the same table
+    Snapshot snapshot = table.currentSnapshot();
     assertNotNull("Table should have a snapshot after writing data", snapshot);
 
-    // Verify metrics are collected only for specified columns
-    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
-    assertNotNull(dataFile.valueCounts());
+    // Verify metrics are collected for specified columns in one file
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
+    Map<Integer, Long> valueCounts = dataFile.valueCounts();
+    assertNotNull("Value counts should not be null", valueCounts);
+    assertTrue("Value counts should exist for id (column 1)", valueCounts.containsKey(1));
+    assertEquals("Value count for id should be 1 in this file", 1L, valueCounts.get(1).longValue());
+    assertTrue("Value counts should exist for name (column 2)", valueCounts.containsKey(2));
+    assertEquals(
+        "Value count for name should be 1 in this file", 1L, valueCounts.get(2).longValue());
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: I think it's because we're creating the table with a partition spec above, so this test is actually writing 3 different data files, with one row in each.

Can we remove the partition spec so we only have one data file?
created_at: 2025-03-12T12:26:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -786,52 +793,69 @@ public void testDefaultMetrics() throws IOException {
     // Create a RecordWriterManager
     RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
 
-    // Write a row
-    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
-    boolean writeSuccess = writerManager.write(windowedDestination, row);
-    assertTrue("Write operation should succeed", writeSuccess);
+    // Write multiple rows
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row3));
     writerManager.close();
 
     // Manually commit the data files to the Iceberg table
     Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
         writerManager.getSerializableDataFiles();
     assertFalse("Data files should not be empty", dataFiles.isEmpty());
 
+    // Commit using the same table instance
     for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
         dataFiles.entrySet()) {
-      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
-      AppendFiles appendFiles = tableToCommit.newAppend();
+      AppendFiles appendFiles = table.newAppend();
       for (SerializableDataFile dataFile : entry.getValue()) {
-        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
       }
       appendFiles.commit();
-      tableToCommit.refresh();
     }
+    table.refresh(); // Single refresh after all commits
 
-    // Verify the correct table (where data was written)
-    Table dataTable = catalog.loadTable(TableIdentifier.of("default", "test_default_metrics"));
-    dataTable.refresh();
-
-    // Verify that a snapshot exists
-    Snapshot snapshot = dataTable.currentSnapshot();
+    // Verify using the same table
+    Snapshot snapshot = table.currentSnapshot();
     assertNotNull("Table should have a snapshot after writing data", snapshot);
 
-    // Verify metrics are collected for all columns
-    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    // Verify metrics are collected for all columns in one file
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
     assertNotNull(dataFile.valueCounts());
     assertNotNull(dataFile.nullValueCounts());
     assertNotNull(dataFile.columnSizes());
+    assertNotNull(dataFile.lowerBounds());
+    assertNotNull(dataFile.upperBounds());
 
-    // Verify metrics are collected for all columns
     Map<Integer, Long> valueCounts = dataFile.valueCounts();
     Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
     Map<Integer, Long> columnSizes = dataFile.columnSizes();
+    Map<Integer, ByteBuffer> lowerBounds = dataFile.lowerBounds();
+    Map<Integer, ByteBuffer> upperBounds = dataFile.upperBounds();
 
     for (int i = 1; i <= ICEBERG_SCHEMA.columns().size(); i++) {
       assertTrue("Value counts should be collected for column " + i, valueCounts.containsKey(i));
+      assertEquals(
+          "Value count for column " + i + " should be 1 in this file",
+          1L,
+          valueCounts.get(i).longValue());
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: See previous comment for this as well
created_at: 2025-03-12T12:27:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -786,52 +793,69 @@ public void testDefaultMetrics() throws IOException {
     // Create a RecordWriterManager
     RecordWriterManager writerManager = new RecordWriterManager(catalog, "test_file_name", 1000, 3);
 
-    // Write a row
-    Row row = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
-    boolean writeSuccess = writerManager.write(windowedDestination, row);
-    assertTrue("Write operation should succeed", writeSuccess);
+    // Write multiple rows
+    Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1, "aaa", true).build();
+    Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2, "bbb", false).build();
+    Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3, "ccc", true).build();
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row1));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row2));
+    assertTrue("Write operation should succeed", writerManager.write(windowedDestination, row3));
     writerManager.close();
 
     // Manually commit the data files to the Iceberg table
     Map<WindowedValue<IcebergDestination>, List<SerializableDataFile>> dataFiles =
         writerManager.getSerializableDataFiles();
     assertFalse("Data files should not be empty", dataFiles.isEmpty());
 
+    // Commit using the same table instance
     for (Map.Entry<WindowedValue<IcebergDestination>, List<SerializableDataFile>> entry :
         dataFiles.entrySet()) {
-      Table tableToCommit = catalog.loadTable(entry.getKey().getValue().getTableIdentifier());
-      AppendFiles appendFiles = tableToCommit.newAppend();
+      AppendFiles appendFiles = table.newAppend();
       for (SerializableDataFile dataFile : entry.getValue()) {
-        appendFiles.appendFile(dataFile.createDataFile(tableToCommit.specs()));
+        appendFiles.appendFile(dataFile.createDataFile(table.specs()));
       }
       appendFiles.commit();
-      tableToCommit.refresh();
     }
+    table.refresh(); // Single refresh after all commits
 
-    // Verify the correct table (where data was written)
-    Table dataTable = catalog.loadTable(TableIdentifier.of("default", "test_default_metrics"));
-    dataTable.refresh();
-
-    // Verify that a snapshot exists
-    Snapshot snapshot = dataTable.currentSnapshot();
+    // Verify using the same table
+    Snapshot snapshot = table.currentSnapshot();
     assertNotNull("Table should have a snapshot after writing data", snapshot);
 
-    // Verify metrics are collected for all columns
-    DataFile dataFile = snapshot.addedDataFiles(dataTable.io()).iterator().next();
+    // Verify metrics are collected for all columns in one file
+    DataFile dataFile = snapshot.addedDataFiles(table.io()).iterator().next();
     assertNotNull(dataFile.valueCounts());
     assertNotNull(dataFile.nullValueCounts());
     assertNotNull(dataFile.columnSizes());
+    assertNotNull(dataFile.lowerBounds());
+    assertNotNull(dataFile.upperBounds());
 
-    // Verify metrics are collected for all columns
     Map<Integer, Long> valueCounts = dataFile.valueCounts();
     Map<Integer, Long> nullValueCounts = dataFile.nullValueCounts();
     Map<Integer, Long> columnSizes = dataFile.columnSizes();
+    Map<Integer, ByteBuffer> lowerBounds = dataFile.lowerBounds();
+    Map<Integer, ByteBuffer> upperBounds = dataFile.upperBounds();
 
     for (int i = 1; i <= ICEBERG_SCHEMA.columns().size(); i++) {
       assertTrue("Value counts should be collected for column " + i, valueCounts.containsKey(i));
+      assertEquals(
+          "Value count for column " + i + " should be 1 in this file",
+          1L,
+          valueCounts.get(i).longValue());
       assertTrue(
           "Null value counts should be collected for column " + i, nullValueCounts.containsKey(i));
       assertTrue("Column sizes should be collected for column " + i, columnSizes.containsKey(i));
+      assertTrue("Lower bounds should be collected for column " + i, lowerBounds.containsKey(i));
+      assertTrue("Upper bounds should be collected for column " + i, upperBounds.containsKey(i));
+
+      // Verify bounds are non-null and equal (single-row file)
+      ByteBuffer lower = lowerBounds.get(i);
+      ByteBuffer upper = upperBounds.get(i);
+      assertNotNull("Lower bound for column " + i + " should not be null", lower);
+      assertNotNull("Upper bound for column " + i + " should not be null", upper);
+      assertTrue(
+          "Lower and upper bounds for column " + i + " should be equal in single-row file",
+          lower.equals(upper));
path: sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
user: login: ahmedabu98
body: We can use Iceberg utils to convert ByteBuffer to a value and verify:
```java
Conversions.fromByteBuffer(
              Types.IntegerType.get(), lowerBounds.get(<columnId>))
```
created_at: 2025-03-12T12:30:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -22,6 +22,7 @@
 import org.apache.beam.sdk.metrics.Metrics;
 import org.apache.iceberg.DataFile;
 import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig; // Add this import
path: sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriter.java
user: login: ahmedabu98
body: nit: cleanup comment
created_at: 2025-03-12T14:15:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: Suvrat1629
created_at: 2025-03-02T09:32:22Z
body: Hey @ahmedabu98 I have opened a pr please take a look and provide comments for further imporvement also regarding the tests I have added a `testDefaultMetrics()` which seems to be failing due to some `snapshot` error please provide me comments to fix that too.
Thank you :)
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-02T10:36:23Z
body: Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-06T11:13:35Z
body: @ahmedabu98 I have made the changes as you have asked. Please take a look
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-06T12:10:57Z
body: Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:

R: @damondouglas for label java.
R: @chamikaramj for label io.

Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-12T03:03:21Z
body: @chamikaramj @damondouglas Gentle ping! Any more fixes that I need to make for this pr? Could you please review this.
Thank you.
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-12T13:28:30Z
body: @ahmedabu98 I have made the changes you have requested for, please take a look.
Thank you.
author_association: CONTRIBUTOR
user: login: Suvrat1629
created_at: 2025-03-12T16:04:19Z
body: @ahmedabu98 
> Last thing would be to trigger integration tests by pushing a new commit with a trivial change to this file: https://github.com/apache/beam/blob/master/.github/trigger_files/IO_Iceberg_Integration_Tests.json

I changed the `"modification": 2` to `"modification": 3` hope that does the job. 


author_association: CONTRIBUTOR
user: login: ahmedabu98
created_at: 2025-03-12T16:51:33Z
body: Yep that ran it, thanks!
Merging now
author_association: CONTRIBUTOR