Repository: GoogleCloudDataproc/hadoop-connectors
Pull request #1349: Using Apiary client for isHnBucket checks 
Author: login: animesh-g
Created: 2025-05-12T06:01:28Z | Merged: 2025-05-19T13:48:20Z

getStorageLayout call from Apiary client results in faster responses than StorageControlClient so using the same. Here are the results from our experiments,

Using Apiary call,
Before initializing connection - 1202 ms
After initializing connection - 66 ms

Using Storage control client,
Without initializing connection - 2969 ms
After initializing connection - 83 ms

There is around 20% improvement in post-initialization calls and 60% improvement in the pre-initialization calls. This will also helps us reducing the tax for non-HNS users when we enable the HNS flag by default in connector. One can find code changes for benchmarking these APIs [here](https://github.com/animesh-g/hadoop-connectors/pull/4/commits/5db42b44749244c1af155e5d885334e3570b1423).

Older PR from a fork - https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1334



Diff:
diff --git a/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java b/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
index 55fd71d99d..279efeaf15 100644
--- a/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
+++ b/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
@@ -46,6 +46,7 @@
 import com.google.api.services.storage.Storage;
 import com.google.api.services.storage.StorageRequest;
 import com.google.api.services.storage.model.Bucket;
+import com.google.api.services.storage.model.BucketStorageLayout;
 import com.google.api.services.storage.model.Buckets;
 import com.google.api.services.storage.model.ComposeRequest;
 import com.google.api.services.storage.model.Objects;
@@ -1824,7 +1825,7 @@ private String listStorageFoldersAndPrefixesPage(
     checkNotNull(listedFolder, "Must provide a non-null container for listedFolder.");
 
     ListFoldersPagedResponse listFolderRespose =
-        storageControlClient.listFolders(listFoldersRequest);
+        lazyGetStorageControlClient().listFolders(listFoldersRequest);
     try (ITraceOperation op = TraceOperation.addToExistingTrace("gcs.folders.list")) {
       Iterator<Folder> itemsIterator = listFolderRespose.getPage().getValues().iterator();
       while (itemsIterator.hasNext()) {
@@ -2414,20 +2415,11 @@ public boolean isHnBucket(URI src) throws IOException {
       return isEnabled;
     }
 
-    String prefix = src.getPath().substring(1);
-
-    StorageControlClient storageControlClient = lazyGetStorageControlClient();
-    GetStorageLayoutRequest request =
-        GetStorageLayoutRequest.newBuilder()
-            .setPrefix(prefix)
-            .setName(StorageLayoutName.format("_", bucketName))
-            .build();
-
+    Storage.Buckets.GetStorageLayout request =
+        initializeRequest(storage.buckets().getStorageLayout(bucketName), bucketName);
     try (ITraceOperation to = TraceOperation.addToExistingTrace("getStorageLayout.HN")) {
-      StorageLayout storageLayout = storageControlClient.getStorageLayout(request);
-      boolean result =
-          storageLayout.hasHierarchicalNamespace()
-              && storageLayout.getHierarchicalNamespace().getEnabled();
+      BucketStorageLayout layout = request.execute();
+      boolean result = layout.getHierarchicalNamespace().getEnabled();
 
       logger.atInfo().log("Checking if %s is HN enabled returned %s", src, result);
 
@@ -2461,7 +2453,7 @@ public void renameHnFolder(URI src, URI dst) throws IOException {
 
     try (ITraceOperation to = TraceOperation.addToExistingTrace("renameHnFolder")) {
       logger.atFine().log("Renaming HN folder (%s -> %s)", src, dst);
-      this.storageControlClient.renameFolderOperationCallable().call(request);
+      lazyGetStorageControlClient().renameFolderOperationCallable().call(request);
     } catch (Throwable t) {
       logger.atSevere().withCause(t).log("Renaming %s to %s failed", src, dst);
       throw t;
diff --git a/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java b/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java
index ee855282ae..335f939a2c 100644
--- a/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java
+++ b/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java
@@ -29,6 +29,7 @@
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.deleteBucketRequestString;
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.deleteRequestString;
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.getBucketRequestString;
+import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.getBucketStorageLayoutRequestString;
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.getMediaRequestString;
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.getRequestString;
 import static com.google.cloud.hadoop.gcsio.TrackingHttpRequestInitializer.listBucketsRequestString;
@@ -61,6 +62,8 @@
 import com.google.api.client.util.Sleeper;
 import com.google.api.services.storage.Storage;
 import com.google.api.services.storage.model.Bucket;
+import com.google.api.services.storage.model.BucketStorageLayout;
+import com.google.api.services.storage.model.BucketStorageLayout.HierarchicalNamespace;
 import com.google.api.services.storage.model.Buckets;
 import com.google.api.services.storage.model.Objects;
 import com.google.api.services.storage.model.StorageObject;
@@ -84,6 +87,7 @@
 import java.io.InputStream;
 import java.math.BigInteger;
 import java.net.SocketTimeoutException;
+import java.net.URI;
 import java.nio.ByteBuffer;
 import java.nio.channels.ClosedChannelException;
 import java.nio.channels.SeekableByteChannel;
@@ -2240,6 +2244,46 @@ public void testListBucketInfo() throws IOException {
         .inOrder();
   }
 
+  /** Test for GoogleCloudStorage.isHnBucket(1). */
+  @Test
+  public void testIsHnBucket_enabled() throws Exception {
+    BucketStorageLayout layout =
+        new BucketStorageLayout()
+            .setHierarchicalNamespace(new HierarchicalNamespace().setEnabled(true));
+    MockHttpTransport transport = mockTransport(jsonDataResponse(layout));
+    GoogleCloudStorage gcs =
+        mockedGcsImpl(GCS_OPTIONS, transport, trackingRequestInitializerWithRetries);
+
+    String testHnsBucket = "hns-bucket-enabled";
+    URI bucketUri = new URI("gs://" + testHnsBucket);
+    boolean result = gcs.isHnBucket(bucketUri);
+
+    assertThat(trackingRequestInitializerWithRetries.getAllRequestStrings())
+        .containsExactly(getBucketStorageLayoutRequestString(testHnsBucket))
+        .inOrder();
+    assertThat(result).isTrue();
+  }
+
+  /** Test for GoogleCloudStorage.isHnBucket(1). */
+  @Test
+  public void testIsHnBucket_disabled() throws Exception {
+    BucketStorageLayout layout =
+        new BucketStorageLayout()
+            .setHierarchicalNamespace(new HierarchicalNamespace().setEnabled(false));
+    MockHttpTransport transport = mockTransport(jsonDataResponse(layout));
+    GoogleCloudStorage gcs =
+        mockedGcsImpl(GCS_OPTIONS, transport, trackingRequestInitializerWithRetries);
+
+    String testHnsBucket = "hns-bucket-disabled";
+    URI bucketUri = new URI("gs://" + testHnsBucket);
+    boolean result = gcs.isHnBucket(bucketUri);
+
+    assertThat(trackingRequestInitializerWithRetries.getAllRequestStrings())
+        .containsExactly(getBucketStorageLayoutRequestString(testHnsBucket))
+        .inOrder();
+    assertThat(result).isFalse();
+  }
+
   @Test
   public void listObjectInfo_objects() throws IOException {
     String prefix = "foo/bar/baz/";
diff --git a/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/TrackingHttpRequestInitializer.java b/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/TrackingHttpRequestInitializer.java
index 929e3b436f..28cdce3926 100644
--- a/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/TrackingHttpRequestInitializer.java
+++ b/gcsio/src/test/java/com/google/cloud/hadoop/gcsio/TrackingHttpRequestInitializer.java
@@ -50,6 +50,9 @@ public class TrackingHttpRequestInitializer implements HttpRequestInitializer {
   private static final String GET_BUCKET_REQUEST_FORMAT =
       "GET:" + GOOGLEAPIS_ENDPOINT + "/storage/v1/b/%s";
 
+  private static final String GET_BUCKET_STORAGE_LAYOUT_REQUEST_FORMAT =
+      "GET:" + GOOGLEAPIS_ENDPOINT + "/storage/v1/b/%s/storageLayout";
+
   private static final String POST_REQUEST_FORMAT =
       "POST:" + GOOGLEAPIS_ENDPOINT + "/storage/v1/b/%s/o/%s";
 
@@ -264,6 +267,10 @@ public static String getBucketRequestString(String bucketName) {
     return String.format(GET_BUCKET_REQUEST_FORMAT, bucketName);
   }
 
+  public static String getBucketStorageLayoutRequestString(String bucketName) {
+    return String.format(GET_BUCKET_STORAGE_LAYOUT_REQUEST_FORMAT, bucketName);
+  }
+
   public static String postRequestString(String bucketName, String object) {
     return String.format(POST_REQUEST_FORMAT, bucketName, urlEncode(object));
   }


Reviews:
user: login: singhravidutt
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T08:41:08Z
user: login: animesh-g
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T09:49:03Z
user: login: singhravidutt
body: LGTM, just wondering why flipping the client didn't cause any Unit test failure.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T04:21:34Z
user: login: animesh-g
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T10:24:44Z
user: login: animesh-g
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T10:25:14Z
user: login: singhravidutt
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T10:29:58Z

Review comments:
diff_hunk: @@ -2461,7 +2453,7 @@ public void renameHnFolder(URI src, URI dst) throws IOException {
 
     try (ITraceOperation to = TraceOperation.addToExistingTrace("renameHnFolder")) {
       logger.atFine().log("Renaming HN folder (%s -> %s)", src, dst);
-      this.storageControlClient.renameFolderOperationCallable().call(request);
+      lazyGetStorageControlClient().renameFolderOperationCallable().call(request);
path: gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
user: login: singhravidutt
body: what about this API? have we benchmarked it as well?
created_at: 2025-05-13T08:41:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2461,7 +2453,7 @@ public void renameHnFolder(URI src, URI dst) throws IOException {
 
     try (ITraceOperation to = TraceOperation.addToExistingTrace("renameHnFolder")) {
       logger.atFine().log("Renaming HN folder (%s -> %s)", src, dst);
-      this.storageControlClient.renameFolderOperationCallable().call(request);
+      lazyGetStorageControlClient().renameFolderOperationCallable().call(request);
path: gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
user: login: animesh-g
body: As discussed over chat, it is also benchmarked and this performs significantly better than the alternative APIs.
created_at: 2025-05-13T09:49:03Z
author_association: COLLABORATOR
diff_hunk: @@ -2414,20 +2415,11 @@ public boolean isHnBucket(URI src) throws IOException {
       return isEnabled;
     }
 
-    String prefix = src.getPath().substring(1);
path: gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
user: login: singhravidutt
body: How come there are no UnitTest failure caused by this change?
created_at: 2025-05-14T04:19:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2240,6 +2243,38 @@ public void testListBucketInfo() throws IOException {
         .inOrder();
   }
 
+  /** Test for GoogleCloudStorage.isHnBucket(1). */
+  @Test
+  public void testIsHnBucket_enabled() throws Exception {
+    BucketStorageLayout layout =
+        new BucketStorageLayout()
+            .setHierarchicalNamespace(new HierarchicalNamespace().setEnabled(true));
+    MockHttpTransport transport = mockTransport(jsonDataResponse(layout));
+    GoogleCloudStorage gcs =
+        mockedGcsImpl(GCS_OPTIONS, transport, trackingRequestInitializerWithRetries);
+
+    URI bucketUri = new URI("gs://hns-enabled-bucket");
+    boolean result = gcs.isHnBucket(bucketUri);
+
+    assertThat(result).isTrue();
path: gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java
user: login: singhravidutt
body: shall we also verifying if mockedTransport was is even called?
created_at: 2025-05-14T04:20:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2240,6 +2243,38 @@ public void testListBucketInfo() throws IOException {
         .inOrder();
   }
 
+  /** Test for GoogleCloudStorage.isHnBucket(1). */
+  @Test
+  public void testIsHnBucket_enabled() throws Exception {
+    BucketStorageLayout layout =
+        new BucketStorageLayout()
+            .setHierarchicalNamespace(new HierarchicalNamespace().setEnabled(true));
+    MockHttpTransport transport = mockTransport(jsonDataResponse(layout));
+    GoogleCloudStorage gcs =
+        mockedGcsImpl(GCS_OPTIONS, transport, trackingRequestInitializerWithRetries);
+
+    URI bucketUri = new URI("gs://hns-enabled-bucket");
+    boolean result = gcs.isHnBucket(bucketUri);
+
+    assertThat(result).isTrue();
path: gcsio/src/test/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageTest.java
user: login: animesh-g
body: Done
created_at: 2025-05-14T10:24:44Z
author_association: COLLABORATOR
diff_hunk: @@ -2414,20 +2415,11 @@ public boolean isHnBucket(URI src) throws IOException {
       return isEnabled;
     }
 
-    String prefix = src.getPath().substring(1);
path: gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorageImpl.java
user: login: animesh-g
body: There was no unit test written for this method. I have added some in this PR.
created_at: 2025-05-14T10:25:14Z
author_association: COLLABORATOR

PR comments:
user: login: animesh-g
created_at: 2025-05-12T06:01:44Z
body: /gcbrun
author_association: COLLABORATOR
user: login: codecov[bot]
created_at: 2025-05-12T06:10:48Z
body: ## [Codecov](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/pull/1349?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 82.65%. Comparing base [(`95f39f5`)](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/commit/95f39f573c8edf1ac51c8f716c207e0b246e50cf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc) to head [(`dd7eb17`)](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/commit/dd7eb176b3d6c42d8d7a53b47fa3e48549261395?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc).
> Report is 1 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #1349      +/-   ##
============================================
+ Coverage     72.91%   82.65%   +9.74%     
- Complexity     1948     2191     +243     
============================================
  Files           115      115              
  Lines          9665     9659       -6     
  Branches       1172     1170       -2     
============================================
+ Hits           7047     7984     +937     
+ Misses         2084     1187     -897     
+ Partials        534      488      -46     
```

| [Flag](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/pull/1349/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc) | Coverage Δ | |
|---|---|---|
| [integrationtest](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/pull/1349/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc) | `68.30% <100.00%> (?)` | |
| [unittest](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/pull/1349/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc) | `73.06% <66.66%> (+0.14%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/GoogleCloudDataproc/hadoop-connectors/pull/1349?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudDataproc).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: animesh-g
created_at: 2025-05-12T09:39:04Z
body: /gcbrun
author_association: COLLABORATOR
user: login: animesh-g
created_at: 2025-05-14T10:24:33Z
body: /gcbrun
author_association: COLLABORATOR
user: login: animesh-g
created_at: 2025-05-18T18:41:30Z
body: /gcbrun
author_association: COLLABORATOR
user: login: animesh-g
created_at: 2025-05-19T13:13:35Z
body: /gcbrun
author_association: COLLABORATOR