Repository: DataBiosphere/jade-data-repo
Pull request #1931: DT-1330: On enable inherit steward, grant ServiceUsageConsumer role to the dataset custodian group on the child snapshot
Author: login: snf2ye
Created: 2025-03-10T18:54:45Z | Merged: 2025-03-12T14:44:56Z

__Jira ticket__: https://broadworkbench.atlassian.net/browse/DT-1330

## Addresses

When the inherit steward flag is set, we want for dataset custodians to be able to perform all of the same operations as members directly on the snapshot. One additional permission we need to grant the dataset custodian group is the `roles/serviceusage.serviceUsageConsumer` role in GCP. This enables snapshot users to be able to resolve DRS URLS when the dataset is self-hosted with a requester pays bucket.

## Summary of changes
- On enable inherit steward, grant the data custodian group the `roles/serviceusage.serviceUsageConsumer` role
- Add new service method that updates multiple roles in Google IAM with a single call
- Update the SetAuthBqJobUserStep by renaming it and having it update both the BQ Job User role and the Service Usage Consumer roles in one call per snapshot/google project
- On undo, revoke this access

## Testing Strategy
- Unit tests
- Full manual test with https://broadworkbench.atlassian.net/browse/DT-1317

<!-- Reminder -->
<!-- Two CODEOWNERS will be automatically assigned to review this pull request. If you otherwise have two reviewers, you do not need to wait for their review. -->



Diff:
diff --git a/src/main/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlight.java b/src/main/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlight.java
index a8d18546b0..e11ee287ae 100644
--- a/src/main/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlight.java
+++ b/src/main/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlight.java
@@ -44,7 +44,7 @@ public EnableInheritStewardFlight(FlightMap inputParameters, Object applicationC
     addStep(new GetSnapshotIdsStep(snapshotDao, datasetId));
     addStep(new SetParentOnSnapshotsStep(iamService, datasetId, userReq));
     addStep(new GetSnapshotGoogleProjectIdsStep(snapshotService, datasetId));
-    addStep(new SetAuthBqJobUserStep(resourceService, custodianEmail, inheritSteward));
+    addStep(new SetAuthGcpUserRolesStep(resourceService, custodianEmail, inheritSteward));
     addStep(
         new SetAuthTabularAclStep(
             bigQuerySnapshotPdao, snapshotService, custodianEmail, inheritSteward));
diff --git a/src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStep.java b/src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStep.java
similarity index 88%
rename from src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStep.java
rename to src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStep.java
index d24be2297b..dd479dad75 100644
--- a/src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStep.java
+++ b/src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStep.java
@@ -11,7 +11,7 @@
 import java.util.List;
 import java.util.Objects;
 
-public record SetAuthBqJobUserStep(
+public record SetAuthGcpUserRolesStep(
     ResourceService resourceService, String custodianEmail, boolean inheritSteward)
     implements Step {
 
@@ -22,9 +22,9 @@ private StepResult setAuth(FlightContext flightContext, boolean inheritSteward)
         workingMap.get(DatasetWorkingMapKeys.SNAPSHOT_GOOGLE_PROJECT_IDS, new TypeReference<>() {});
     for (var projectId : Objects.requireNonNull(projectIds)) {
       if (inheritSteward) {
-        resourceService.grantPoliciesBqJobUser(projectId, List.of(custodianEmail));
+        resourceService.assignRolesForSnapshot(projectId, List.of(custodianEmail));
       } else {
-        resourceService.revokePoliciesBqJobUser(projectId, List.of(custodianEmail));
+        resourceService.revokeRolesForSnapshot(projectId, List.of(custodianEmail));
       }
     }
     return StepResult.getStepResultSuccess();
diff --git a/src/main/java/bio/terra/service/resourcemanagement/ResourceService.java b/src/main/java/bio/terra/service/resourcemanagement/ResourceService.java
index e22e8d339e..51ae690cc4 100644
--- a/src/main/java/bio/terra/service/resourcemanagement/ResourceService.java
+++ b/src/main/java/bio/terra/service/resourcemanagement/ResourceService.java
@@ -53,6 +53,9 @@ public class ResourceService {
   public static final String SERVICE_USAGE_CONSUMER_ROLE =
       "roles/serviceusage.serviceUsageConsumer";
 
+  public static final List<String> SNAPSHOT_GCP_IAM_ROLES =
+      List.of(BQ_JOB_USER_ROLE, SERVICE_USAGE_CONSUMER_ROLE);
+
   private final AzureDataLocationSelector azureDataLocationSelector;
   private final GoogleProjectService projectService;
   private final GoogleBucketService bucketService;
@@ -599,6 +602,16 @@ public AzureApplicationDeploymentResource getApplicationDeploymentResource(UUID
     return applicationDeploymentService.getApplicationDeploymentResourceById(applicationId);
   }
 
+  public void assignRolesForSnapshot(String dataProject, Collection<String> policyEmails)
+      throws InterruptedException {
+    modifyRoles(dataProject, policyEmails, SNAPSHOT_GCP_IAM_ROLES, ENABLE_PERMISSIONS);
+  }
+
+  public void revokeRolesForSnapshot(String dataProject, Collection<String> policyEmails)
+      throws InterruptedException {
+    modifyRoles(dataProject, policyEmails, SNAPSHOT_GCP_IAM_ROLES, REVOKE_PERMISSIONS);
+  }
+
   public void grantPoliciesBqJobUser(String dataProject, Collection<String> policyEmails)
       throws InterruptedException {
     modifyRoles(dataProject, policyEmails, List.of(BQ_JOB_USER_ROLE), ENABLE_PERMISSIONS);
diff --git a/src/test/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlightTest.java b/src/test/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlightTest.java
index 84386240d8..9c2a8337fa 100644
--- a/src/test/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlightTest.java
+++ b/src/test/java/bio/terra/service/dataset/flight/inheritsteward/EnableInheritStewardFlightTest.java
@@ -81,7 +81,7 @@ void allSteps() {
             "GetSnapshotIdsStep",
             "SetParentOnSnapshotsStep",
             "GetSnapshotGoogleProjectIdsStep",
-            "SetAuthBqJobUserStep",
+            "SetAuthGcpUserRolesStep",
             "SetAuthTabularAclStep",
             "UnlockDatasetStep"));
   }
@@ -186,10 +186,10 @@ void getSnapshotGoogleProjectIdsStep() {
   }
 
   @Test
-  void setAuthBqJobUserStep() {
+  void setAuthGcpUserRolesStep() {
     try (var mockStep =
         mockConstruction(
-            SetAuthBqJobUserStep.class,
+            SetAuthGcpUserRolesStep.class,
             (mock, context) -> {
               assertThat((ResourceService) context.arguments().get(0), equalTo(resourceService));
               assertThat(
diff --git a/src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStepTest.java b/src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStepTest.java
similarity index 84%
rename from src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStepTest.java
rename to src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStepTest.java
index 84a76a80c1..ef79132bc1 100644
--- a/src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthBqJobUserStepTest.java
+++ b/src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStepTest.java
@@ -22,7 +22,7 @@
 
 @ExtendWith(MockitoExtension.class)
 @Tag(Unit.TAG)
-class SetAuthBqJobUserStepTest {
+class SetAuthGcpUserRolesStepTest {
 
   @Mock private ResourceService resourceService;
   @Mock private FlightContext flightContext;
@@ -35,15 +35,16 @@ interface DoOrUndo {
 
   private void verifySetAuth(DoOrUndo doOrUndo, boolean grantPolicy) throws Exception {
     var projectIds = Arrays.asList("project1", "project2");
+    var emails = List.of(custodianEmail);
     FlightMap workingMap = new FlightMap();
     workingMap.put(DatasetWorkingMapKeys.SNAPSHOT_GOOGLE_PROJECT_IDS, projectIds);
     when(flightContext.getWorkingMap()).thenReturn(workingMap);
     assertThat(doOrUndo.apply(flightContext), is(StepResult.getStepResultSuccess()));
     for (var projectId : projectIds) {
       if (grantPolicy) {
-        verify(resourceService).grantPoliciesBqJobUser(projectId, List.of(custodianEmail));
+        verify(resourceService).assignRolesForSnapshot(projectId, emails);
       } else {
-        verify(resourceService).revokePoliciesBqJobUser(projectId, List.of(custodianEmail));
+        verify(resourceService).revokeRolesForSnapshot(projectId, emails);
       }
     }
   }
@@ -51,8 +52,8 @@ private void verifySetAuth(DoOrUndo doOrUndo, boolean grantPolicy) throws Except
   @ParameterizedTest
   @ValueSource(booleans = {true, false})
   void doStep(boolean inheritSteward) throws Exception {
-    SetAuthBqJobUserStep step =
-        new SetAuthBqJobUserStep(resourceService, custodianEmail, inheritSteward);
+    SetAuthGcpUserRolesStep step =
+        new SetAuthGcpUserRolesStep(resourceService, custodianEmail, inheritSteward);
     verifySetAuth(step::doStep, inheritSteward);
     verifySetAuth(step::undoStep, !inheritSteward);
   }
diff --git a/src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceUnitTest.java b/src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceTest.java
similarity index 77%
rename from src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceUnitTest.java
rename to src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceTest.java
index 531984d018..7f180aefa1 100644
--- a/src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceUnitTest.java
+++ b/src/test/java/bio/terra/service/resourcemanagement/google/ResourceServiceTest.java
@@ -1,10 +1,14 @@
 package bio.terra.service.resourcemanagement.google;
 
+import static bio.terra.service.resourcemanagement.ResourceService.SNAPSHOT_GCP_IAM_ROLES;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.is;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.argThat;
+import static org.mockito.ArgumentMatchers.eq;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 
 import bio.terra.app.configuration.SamConfiguration;
@@ -40,7 +44,7 @@
 
 @ExtendWith(MockitoExtension.class)
 @Tag(Unit.TAG)
-class ResourceServiceUnitTest {
+class ResourceServiceTest {
 
   private ResourceService resourceService;
 
@@ -52,6 +56,8 @@ class ResourceServiceUnitTest {
 
   @Mock private AzureApplicationDeploymentService applicationDeploymentService;
 
+  @Mock private GoogleResourceManagerService resourceManagerService;
+
   private final UUID billingProfileId = UUID.randomUUID();
 
   private final UUID datasetId = UUID.randomUUID();
@@ -100,7 +106,7 @@ void setup() {
             mock(SamConfiguration.class),
             datasetStorageAccountDao,
             mock(SnapshotStorageAccountDao.class),
-            mock(GoogleResourceManagerService.class),
+            resourceManagerService,
             mock(AzureContainerPdao.class),
             mock(ProfileDao.class));
   }
@@ -134,4 +140,40 @@ void testGetOrCreateStorageAccount() throws Exception {
         resourceService.getOrCreateDatasetStorageAccount(dataset, profileModel, "flightId");
     assertThat(createdStorageAccount, is(storageAccountResource));
   }
+
+  @Test
+  void assignRolesForSnapshot() throws InterruptedException {
+    String dataProject = "test-project";
+    List<String> policyEmails = List.of("test-email@example.com");
+
+    resourceService.assignRolesForSnapshot(dataProject, policyEmails);
+
+    // Verify that the IAM permissions were updated
+    verify(resourceManagerService)
+        .updateIamPermissions(
+            argThat(
+                arg ->
+                    arg.containsKey(SNAPSHOT_GCP_IAM_ROLES.get(0))
+                        && arg.containsKey(SNAPSHOT_GCP_IAM_ROLES.get(1))),
+            eq(dataProject),
+            eq(GoogleProjectService.PermissionOp.ENABLE_PERMISSIONS));
+  }
+
+  @Test
+  void revokeRolesForSnapshot() throws InterruptedException {
+    String dataProject = "test-project";
+    List<String> policyEmails = List.of("test-email@example.com");
+
+    resourceService.revokeRolesForSnapshot(dataProject, policyEmails);
+
+    // Verify that the IAM permissions were updated
+    verify(resourceManagerService)
+        .updateIamPermissions(
+            argThat(
+                arg ->
+                    arg.containsKey(SNAPSHOT_GCP_IAM_ROLES.get(0))
+                        && arg.containsKey(SNAPSHOT_GCP_IAM_ROLES.get(1))),
+            eq(dataProject),
+            eq(GoogleProjectService.PermissionOp.REVOKE_PERMISSIONS));
+  }
 }


Reviews:
user: login: pshapiro4broad
body: Looks good to me.

With my PR https://github.com/DataBiosphere/jade-data-repo/pull/1929/files#diff-1eb13a8740fb5de2d58862596788fd7d8a65e12416e93f92daea6506b9b8c2d6 we now have three steps that use the same pattern. What do you think about combining them into a single step? I'm thinking the order here doesn't matter so combining into a single step could be OK.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-10T19:02:06Z
user: login: s-rubenstein
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T15:26:52Z
user: login: pshapiro4broad
body: Looks OK, but the use of `ResourceService` constants outside the class makes me wonder if there's a higher level concept for the operation which would avoid the need to expose these values.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-11T15:42:33Z
user: login: fboulnois
body: 👍 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T13:54:12Z

Review comments:
diff_hunk: @@ -11,20 +14,25 @@
 import java.util.List;
 import java.util.Objects;
 
-public record SetAuthBqJobUserStep(
+public record SetAuthGcpUserRolesStep(
     ResourceService resourceService, String custodianEmail, boolean inheritSteward)
     implements Step {
 
+  public static final List<String> SNAPSHOT_GCP_IAM_ROLES =
+      List.of(BQ_JOB_USER_ROLE, SERVICE_USAGE_CONSUMER_ROLE);
path: src/main/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStep.java
user: login: pshapiro4broad
body: Would it make sense to push this list into `ResourceService`? Or, hide it completely from the caller in a method like `ResourceService.grantSnapshotPolicies(projectId, List<String>)`?

created_at: 2025-03-11T15:32:00Z
author_association: MEMBER
diff_hunk: @@ -41,18 +41,22 @@ private void verifySetAuth(DoOrUndo doOrUndo, boolean grantPolicy) throws Except
     assertThat(doOrUndo.apply(flightContext), is(StepResult.getStepResultSuccess()));
     for (var projectId : projectIds) {
       if (grantPolicy) {
-        verify(resourceService).grantPoliciesBqJobUser(projectId, List.of(custodianEmail));
+        verify(resourceService)
+            .grantPoliciesForRoles(
+                projectId, List.of(custodianEmail), SetAuthGcpUserRolesStep.SNAPSHOT_GCP_IAM_ROLES);
path: src/test/java/bio/terra/service/dataset/flight/inheritsteward/SetAuthGcpUserRolesStepTest.java
user: login: pshapiro4broad
body: You could create this `List.of()` outside the loop and share it with both calls.
created_at: 2025-03-11T15:33:23Z
author_association: MEMBER

PR comments:
user: login: snf2ye
created_at: 2025-03-10T20:25:09Z
body: > With my PR https://github.com/DataBiosphere/jade-data-repo/pull/1929/files#diff-1eb13a8740fb5de2d58862596788fd7d8a65e12416e93f92daea6506b9b8c2d6 we now have three steps that use the same pattern. What do you think about combining them into a single step? I'm thinking the order here doesn't matter so combining into a single step could be OK.

@pshapiro4broad , Great point - Since the BQ Job user and the Service Usage Consumer roles are both added by the same mechanism, we can actually make both of these changes in one call to Google. So, I've combined those two steps into one, with a single call to google per project. I would leave it up to your discretion as to whether or not we want to add in the BQ ACL change from your latest PR into this same step or leave it separate since it can't actually be combined into the same call to google. I think either would be fine. 
author_association: CONTRIBUTOR
user: login: sonarqubecloud[bot]
created_at: 2025-03-11T19:19:14Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=DataBiosphere_jade-data-repo&pullRequest=1931) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=DataBiosphere_jade-data-repo&pullRequest=1931&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=DataBiosphere_jade-data-repo&pullRequest=1931&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=DataBiosphere_jade-data-repo&pullRequest=1931&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=DataBiosphere_jade-data-repo&pullRequest=1931&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=DataBiosphere_jade-data-repo&pullRequest=1931&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=DataBiosphere_jade-data-repo&pullRequest=1931)


author_association: NONE