Repository: DataBiosphere/jade-data-repo
Pull request #1966: DT-1534 - Add request/response info to logs; Add python script to collect and format logs
Author: login: snf2ye
Created: 2025-04-28T17:58:11Z | Merged: 2025-05-05T16:01:06Z

__Jira ticket__: https://broadworkbench.atlassian.net/browse/DT-1534

## Addresses
- Add additional information to logs
- Collect and format logs

## Summary of changes
- Pull more information from existing request/response objects and add to logger interceptor log message 
- Python script to pull more information from data warehouse, TDR fields and format fields

## Testing Strategy
Deployed on a BEE and ran some jobs to get logs. Example [logs](https://cloudlogging.app.goo.gl/jk6VYWAkRpKWkjvT7). 

Example:
```
{
  "jsonPayload": {
    "message": "userId: <redacted>, email: <redacted>, institute: broadinstitute.org, url: https://datarepo-datarepo-api/api/repository/v1/datasets/05b98250-86ee-4e79-8dcf-f2e93b0f8796/data/domain, method: POST, params: {}, status: 200, duration: 1086, srcIP: <redacted>, destIP: <redacted>, destPort: 8080, sessionId: <redacted>, userAgent: <redacted>, contentType: application/json, bytes: 80",
  "timestamp": "2025-04-28T19:32:44.415Z"
}
```

<!-- Reminder -->
<!-- Two CODEOWNERS will be automatically assigned to review this pull request. If you otherwise have two reviewers, you do not need to wait for their review. -->



Diff:
diff --git a/.gitignore b/.gitignore
index 4bd5767d60..9a3c33a728 100644
--- a/.gitignore
+++ b/.gitignore
@@ -66,6 +66,9 @@ gha-creds-*.json
 #tools output
 tools/setupResourceScripts/*_outputs.json
 tools/profileEndpoints/results_*.csv
+tools/logCollection/rawLogs/
+tools/logCollection/output/
 
 
 gha-creds-*.json
+
diff --git a/src/main/java/bio/terra/app/logging/LoggerInterceptor.java b/src/main/java/bio/terra/app/logging/LoggerInterceptor.java
index ad5abf6fbe..a5537f26bb 100644
--- a/src/main/java/bio/terra/app/logging/LoggerInterceptor.java
+++ b/src/main/java/bio/terra/app/logging/LoggerInterceptor.java
@@ -27,7 +27,6 @@ public class LoggerInterceptor implements HandlerInterceptor {
   private static final String UNAUTHED_INSTITUTE = "N/A";
   // Don't log requests for URLs that end with any of   the following paths
   private static final Set<String> LOG_EXCLUDE_LIST = Set.of("/status");
-
   private static final String REQUEST_START_ATTRIBUTE = "x-request-start";
   private static final long NOT_FOUND_DURATION = -1;
 
@@ -68,8 +67,15 @@ public void afterCompletion(
       userEmail = UNAUTHED_EMAIL;
       institute = UNAUTHED_INSTITUTE;
     }
-
+    String srcIpAddress = request.getRemoteAddr();
+    String destIPAddress = request.getLocalAddr();
+    String destPort = Integer.toString(request.getLocalPort());
+    String sessionId = request.getSession().getId();
+    String userAgent = request.getHeader("User-Agent");
     String url = request.getRequestURL().toString();
+    String contentType = request.getContentType();
+    int bytes = request.getContentLength();
+
     String method = request.getMethod();
     Map<String, String[]> paramMap = request.getParameterMap();
     Gson gson = new Gson();
@@ -106,7 +112,7 @@ public void afterCompletion(
       // https://github.com/DataBiosphere/terra-common-lib/blob/develop/src/main/java/bio/terra/common/logging/GoogleJsonLayout.java
       // for the layout implementation.
       logger.info(
-          "userId: {}, email: {}, institute: {}, url: {}, method: {}, params: {}, status: {}, duration: {}",
+          "userId: {}, email: {}, institute: {}, url: {}, method: {}, params: {}, status: {}, duration: {}, srcIP: {}, destIP: {}, destPort: {}, sessionId: {}, userAgent: {}, contentType: {}, bytes: {}",
           userId,
           userEmail,
           institute,
@@ -115,6 +121,13 @@ public void afterCompletion(
           paramString,
           responseStatus,
           requestDuration,
+          srcIpAddress,
+          destIPAddress,
+          destPort,
+          sessionId,
+          userAgent,
+          contentType,
+          bytes,
           stackDriverPayload);
     }
 
diff --git a/tools/logCollection/README.md b/tools/logCollection/README.md
new file mode 100644
index 0000000000..f6e0cb8713
--- /dev/null
+++ b/tools/logCollection/README.md
@@ -0,0 +1,45 @@
+The `collect-logs.py` script can be used to transform logs from TDR GCP logging into a tabular format
+with more details.
+
+### Usage:
+**NOTE:** Must first collect logs from gcloud console, download them in JSON format and put the JSON file in the
+`logCollection/rawLogs` folder. This file is what needs to be referenced in the `--raw_logs_location` argument
+for the script.
+
+1. If you need to set up a virtual python environment, you can run the following commands:
+  * `python3 -m venv c:/path/to/myenv`
+  * `source c:/path/to/myenv/bin/activate`
+2. `cd jade-data-repo/tools/logCollection`
+3. `pip3 install -r requirements.txt --upgrade`
+4. `gcloud auth application-default login <user>@firecloud.org` (Need firecloud account to access production user data)
+5. Run the command pointed at the raw log location:
+```
+python3 collect-logs.py --raw_logs_location=<path to raw log file>
+```
+Or the help command to see all options:
+`python3 collect-logs.py --help to see all flags used.`
+6. Formatted logs will output in the `logCollection/output` folder
+
+#### Example:
+`python3 collect_logs.py --raw_logs_location=sample_tdr_logs.json`
+
+#### Testing:
+Run all tests with the following command:
+`python3 -m unittest src/tests/*`
+
+### Notes about output:
+There are a couple of fields that need to be manually populated based on the dataset/snapshot
+that we are logging:
+* `nih_ico`
+* `cadr_name`
+
+There are a couple of hardcoded fields:
+* `user_id_provider` is set to `Terra`
+* `app` is set to `TDR`
+
+Not relevant fields:
+* `eRA_commons_id ` - until RAS is supported in production, this is empty. It will require work to populate this field.
+
+Notes:
+* `associated_study` - This will populate with a list of PHS IDs and/or DUOS ids if available
+
diff --git a/tools/logCollection/collect_logs.py b/tools/logCollection/collect_logs.py
new file mode 100644
index 0000000000..c1cf876ac9
--- /dev/null
+++ b/tools/logCollection/collect_logs.py
@@ -0,0 +1,66 @@
+import argparse
+import os
+import csv
+import datetime
+
+from src.parse_gcp_log import parse_gcp_log
+from src.populate_user_details_from_data_warehouse import populate_user_details
+from src.populate_tdr_details import populate_snapshot_user_permission_group_and_studies, populate_dataset_studies
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--host",
+        required=False,
+        default="https://data.terra.bio",
+        help="The data repo root URL to point to. Defaults to production TDR (https://data.terra.bio).`",
+    )
+    parser.add_argument(
+        "--raw_logs_location",
+        required=True,
+        help="A file pointer to the raw logs to convert. This is required flag. Available Options: "
+             + ", ".join(os.listdir("./rawLogs/")),
+    )
+    args = parser.parse_args()
+    raw_logs_location = args.raw_logs_location
+    host = args.host
+
+    emails = set()
+    dataset_ids = set()
+    snapshot_ids = set()
+
+    populated_new_logs = parse_gcp_log(raw_logs_location, emails, dataset_ids, snapshot_ids)
+    # Get user details from Thurloe/data warehouse
+    populate_user_details(populated_new_logs, emails)
+
+    # For Snapshots, get auth domains from TDR to populate the `user_permission_group` field
+    # and get PHSIds and DUOS Ids from TDR to populate the `associated_study` field
+    populate_snapshot_user_permission_group_and_studies(populated_new_logs, snapshot_ids, host)
+
+    # For Datasets, get PHSIds from TDR
+    populate_dataset_studies(populated_new_logs, dataset_ids, host)
+
+    # Pull ERA commmons id
+    eRA_commons_id = ""
+    # Not yet relevant
+
+
+    # Write to desired format
+    outputs = []
+    outputs.append(["_time", "src_ip", "dest_ip", "dest_port", "user_name", "user_id", "user_id_provider", "session_id", "method", "url", "app", "http_user_agent", "status", "http_content_type", "bytes", "duration", "nih_ico", "cadr_name", "user_country_name", "user_org", "user_email", "associated_study", "eRA_commons_id", "user_permission_group", "event_type", "dataset_id", "snapshot_id"])
+    for log in populated_new_logs:
+        outputs.append([log._time, log.src_ip, log.dest_ip, log.dest_port, log.user_name, log.user_id, log.user_id_provider, log.session_id, log.method, log.url, log.app, log.http_user_agent, log.status, log.http_content_type, log.bytes, log.duration, log.nih_ico, log.cadr_name, log.user_country_name, log.user_org, log.user_email, log.associated_study, log.eRA_commons_id, log.user_permission_group, log.event_type, log.dataset_id, log.snapshot_id])
+
+    output_filename = f"output/{os.path.basename(args.raw_logs_location).split('.')[0]}_{datetime.datetime.now()}.csv"
+
+    with open(output_filename, 'w', newline='') as csvfile:
+        writer = csv.writer(csvfile)
+        for row in outputs:
+            writer.writerow(row)
+
+
+    print(f"\n\n\nDONE. Logs written to {output_filename}")
+
+if __name__ == "__main__":
+    main()
diff --git a/tools/logCollection/rawLogs/sample_tdr_logs.json b/tools/logCollection/rawLogs/sample_tdr_logs.json
new file mode 100644
index 0000000000..f513525c0f
--- /dev/null
+++ b/tools/logCollection/rawLogs/sample_tdr_logs.json
@@ -0,0 +1,30 @@
+[
+  {
+    "insertId": "2fre66rgygzjb2zs",
+    "jsonPayload": {
+      "message": "userId: a0d98b19-e92e-4236-af7a-67858f252f1d, email: sholden@broadinstitute.org, institute: broadinstitute.org, url: https://datarepo-datarepo-api/api/repository/v1/datasets/1e81193f-73f7-4923-91ed-e302a4b4eea5/data/visit_occurrence, method: POST, params: {}, status: 200, duration: 4168, srcIP: 10.64.112.28, destIP: 10.76.0.167, destPort: 8080, sessionId: 4975D83E81046B72F272D5DCE366B41F, userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36, contentType: application/json, bytes: 80",
+      "logger": "bio.terra.app.logging.LoggerInterceptor",
+      "context": "default",
+      "requestId": "2e89443aa86116dccc41e3dd61e5b66e",
+      "requestBody": "{\"offset\":0,\"limit\":100,\"sort\":\"datarepo_row_id\",\"direction\":\"desc\",\"filter\":\"\"}",
+      "thread": "http-nio-8080-exec-9",
+      "serviceContext": {
+        "version": null,
+        "service": null
+      }
+    },
+    "resource": {
+      "type": "k8s_container",
+      "labels": {
+        "project_id": "broad-dsde-qa",
+        "cluster_name": "terra-qa-bees",
+        "namespace_name": "terra-shelbee",
+        "pod_name": "datarepo-datarepo-api-5458b84cf8-v9z2b",
+        "location": "us-central1-a",
+        "container_name": "datarepo-datarepo-api"
+      }
+    },
+    "timestamp": "2025-04-28T19:32:35.636Z",
+    "severity": "INFO"
+  }
+]
diff --git a/tools/logCollection/requirements.txt b/tools/logCollection/requirements.txt
new file mode 100644
index 0000000000..325a738fc7
--- /dev/null
+++ b/tools/logCollection/requirements.txt
@@ -0,0 +1,4 @@
+data-repo-client>=2.280.0
+google-cloud-bigquery
+google-cloud-secret-manager
+parameterized
diff --git a/tools/logCollection/src/categorize_event_type.py b/tools/logCollection/src/categorize_event_type.py
new file mode 100644
index 0000000000..54ff20c966
--- /dev/null
+++ b/tools/logCollection/src/categorize_event_type.py
@@ -0,0 +1,73 @@
+from enum import Enum
+import re
+class EventType(Enum):
+    DATA_ACCESS = 1
+    DATA_UPLOAD = 2
+    DATA_DELETION = 3
+    OTHER = 4
+def event_type_from_url(url, method):
+    """
+    Given a URL, determine the event type.
+    Data Access:
+    :check_green: /api/repository/v1/datasets/{id}/data/{table}
+    :check_green: /api/repository/v1/datasets/{id}/data/{table}/statistics/{column}
+    GET /api/repository/v1/datasets/{id}/files
+    GET /api/repository/v1/datasets/{id}/files/{fileid}
+    /api/repository/v1/datasets/{id}/filesystem/objects
+
+    :check_green: /api/repository/v1/snapshots/{id}/data/{table}
+    /api/repository/v1/snapshots/{id}/files
+    /api/repository/v1/snapshots/{id}/export
+    /api/repository/v1/snapshots/{id}/files/{fileid}
+    /api/repository/v1/snapshots/{id}/filesystem/objects
+
+
+    /ga4gh/drs/v1/objects/{object_id}
+    /ga4gh/drs/v1/objects/{object_id}/access/{access_id}
+
+    Data Uploads:
+    /api/repository/v1/datasets/{id}/ingest
+    POST /api/repository/v1/datasets/{id}/files
+    /api/repository/v1/datasets/{id}/files/bulk
+    /api/repository/v1/datasets/{id}/files/bulk/array
+
+    Data Deletion:
+    DELETE /api/repository/v1/snapshots/{id}
+    POST /api/repository/v1/datasets/{id}/deletes
+    DELETE /api/repository/v1/datasets/{id}
+    DELETE /api/repository/v1/datasets/{id}/files/{fileid}
+    """
+    dataset_regex = re.compile(r'datasets/[0-9a-fA-F-]{36}$')
+    snapshot_regex = re.compile(r'snapshots/[0-9a-fA-F-]{36}$')
+
+    if "/data/" in url:
+        return EventType.DATA_ACCESS
+    elif "/ga4gh/drs/v1/objects/" in url and (method == "GET" or method == "POST"):
+        return EventType.DATA_ACCESS
+    elif "/files" in url:
+        if "/bulk/array" in url:
+            return EventType.DATA_UPLOAD
+        # don't match on load tag endpoints
+        regexp = re.compile(r'/files/bulk/[A-Za-z]+$')
+        if regexp.search(url):
+            return "Other"
+        elif method == "DELETE":
+            return EventType.DATA_DELETION
+        elif method == "POST":
+            return EventType.DATA_UPLOAD
+        elif method == "GET":
+            return EventType.DATA_ACCESS
+    elif "filesystem/objects" in url:
+        return EventType.DATA_ACCESS
+    elif "/export" in url:
+        return EventType.DATA_ACCESS
+    elif "/ingest" in url:
+        return EventType.DATA_UPLOAD
+    elif dataset_regex.search(url) and method == "DELETE":
+        return EventType.DATA_DELETION
+    elif snapshot_regex.search(url) and method == "DELETE":
+        return EventType.DATA_DELETION
+    elif "/deletes" in url and method == "POST":
+        return EventType.DATA_DELETION
+    else:
+        return EventType.OTHER
diff --git a/tools/logCollection/src/log_class.py b/tools/logCollection/src/log_class.py
new file mode 100644
index 0000000000..243d99b4a6
--- /dev/null
+++ b/tools/logCollection/src/log_class.py
@@ -0,0 +1,32 @@
+class LogClass:
+    def __init__(self):
+        self._time = None
+        self.src_ip = None
+        self.dest_ip = None
+        self.dest_port = None
+        self.user_name = None
+        self.user_id = None
+        self.user_id_provider = "Terra"
+        self.session_id = None
+        self.url = None
+        self.app = "TDR"
+        self.http_user_agent = None
+        self.status = None
+        self.http_content_type = None
+        self.bytes = None
+        self.duration = None
+        self.nih_ico = None
+        self.cadr_name = None
+        self.user_country_name = None
+        self.user_org = None
+        self.user_email = None
+        self.associated_study = None
+        self.eRA_commons_id = "N/A"
+        self.user_permission_group = None
+        self.event_type = None
+        self.dataset_id = None
+        self.snapshot_id = None
+        self.method = None
+
+    def __str__(self):
+        return f"{self._time}, {self.src_ip}, {self.dest_ip}, {self.dest_port}, {self.user_name}, {self.user_id}, {self.user_id_provider}, {self.session_id}, {self.url}, {self.app}, {self.http_user_agent}, {self.status}, {self.http_content_type}, {self.bytes}, {self.duration}, {self.nih_ico}, {self.cadr_name}, {self.user_country_name}, {self.user_org}, {self.user_email}, {self.associated_study}, {self.eRA_commons_id}, {self.user_permission_group}, {self.event_type}"
diff --git a/tools/logCollection/src/parse_gcp_log.py b/tools/logCollection/src/parse_gcp_log.py
new file mode 100644
index 0000000000..9f591aa967
--- /dev/null
+++ b/tools/logCollection/src/parse_gcp_log.py
@@ -0,0 +1,65 @@
+import json
+import os
+import re
+
+from .log_class import LogClass
+from .categorize_event_type import event_type_from_url
+
+def get_raw_logs(filename):
+    with open(os.path.join("rawLogs", filename)) as f:
+        return json.load(f)
+
+def parse_gcp_log(raw_logs_location, emails, dataset_ids, snapshot_ids):
+    populated_new_logs = []
+    for log in get_raw_logs(raw_logs_location):
+        new_log = LogClass()
+        new_log._time = log["timestamp"]
+        log_message = log["jsonPayload"]["message"]
+        for item in log_message.split(","):
+            item = item.strip()
+            if item.startswith("userId"):
+                new_log.user_id = item.split(":")[1].strip()
+            elif item.startswith("url"):
+                url = item.split(":", 1)[1].strip()
+                new_log.url = url
+                # regex that matches /datasets/{UUID}
+                regexp = re.compile(r'datasets/[0-9a-fA-F-]{36}')
+                if regexp.search(url):
+                    dataset_id = regexp.search(url).group(0).split("/")[1]
+                    dataset_ids.add(dataset_id)
+                    new_log.dataset_id = dataset_id
+                # regex that matches /snapshots/{UUID}
+                regexp = re.compile(r'snapshots/[0-9a-fA-F-]{36}')
+                if regexp.search(url):
+                    snapshot_id = regexp.search(url).group(0).split("/")[1]
+                    snapshot_ids.add(snapshot_id)
+                    new_log.snapshot_id = snapshot_id
+            elif item.startswith("email"):
+                email = item.split(":")[1].strip()
+                new_log.user_email = email
+                emails.add(email)
+            elif item.startswith("institute"):
+                new_log.user_org = item.split(":")[1].strip()
+            elif item.startswith("status"):
+                new_log.status = item.split(":")[1].strip()
+            elif item.startswith("srcIP"):
+                new_log.src_ip = item.split(":")[1].strip()
+            elif item.startswith("destIP"):
+                new_log.dest_ip = item.split(":")[1].strip()
+            elif item.startswith("destPort"):
+                new_log.dest_port = item.split(":")[1].strip()
+            elif item.startswith("sessionId"):
+                new_log.session_id = item.split(":")[1].strip()
+            elif item.startswith("userAgent"):
+                new_log.http_user_agent = item.split(":")[1].strip()
+            elif item.startswith("contentType"):
+                new_log.http_content_type = item.split(":")[1].strip()
+            elif item.startswith("bytes"):
+                new_log.bytes = item.split(":")[1].strip()
+            elif item.startswith("duration"):
+                new_log.duration = item.split(":")[1].strip()
+            elif item.startswith("method"):
+                new_log.method = item.split(":")[1].strip()
+        new_log.event_type = event_type_from_url(new_log.url, new_log.method).name
+        populated_new_logs.append(new_log)
+    return populated_new_logs
diff --git a/tools/logCollection/src/populate_tdr_details.py b/tools/logCollection/src/populate_tdr_details.py
new file mode 100644
index 0000000000..594694bc89
--- /dev/null
+++ b/tools/logCollection/src/populate_tdr_details.py
@@ -0,0 +1,67 @@
+import subprocess
+from data_repo_client import (
+    Configuration,
+    ApiClient,
+    ProfilesApi,
+    DatasetsApi,
+    SnapshotsApi,
+    JobsApi,
+    SnapshotAccessRequestApi,
+)
+
+
+class TDRClients:
+    def __init__(self, host):
+        config = Configuration()
+        config.host = host
+        token_output = subprocess.run(
+            ["gcloud", "auth", "print-access-token"], capture_output=True
+        )
+        config.access_token = token_output.stdout.decode("UTF-8").strip()
+        self.api_client = ApiClient(configuration=config)
+
+        self.datasets_api = DatasetsApi(api_client=self.api_client)
+        self.snapshots_api = SnapshotsApi(api_client=self.api_client)
+
+
+def populate_snapshot_user_permission_group_and_studies(populated_new_logs, snapshot_ids, host):
+    clients = TDRClients(host)
+    for snapshot_id in snapshot_ids:
+        user_permission_group = retrieve_snapshot_auth_domain(clients, snapshot_id)
+        snapshot = retrieve_snapshot_via_enumerate(clients, snapshot_id)
+        for log in populated_new_logs:
+            if log.snapshot_id == snapshot_id and snapshot is not None:
+                log.user_permission_group = user_permission_group
+                log.associated_study = list(filter(None,[snapshot.phs_id, snapshot.duos_id]))
+
+def populate_dataset_studies(populated_new_logs, dataset_ids, host):
+    clients = TDRClients(host)
+    for dataset_id in dataset_ids:
+        dataset = retrieve_dataset_via_enumerate(clients, dataset_id)
+        for log in populated_new_logs:
+            if log.dataset_id == dataset_id and dataset is not None:
+                log.associated_study = dataset.phs_id
+
+### TDR API calls ###
+def retrieve_snapshot_auth_domain(clients, snapshot_id):
+    policy_response = clients.snapshots_api.retrieve_snapshot_policies(snapshot_id)
+    return policy_response.auth_domain
+
+
+# We want a TDR admin to be able to run this on any snapshot
+# So, we can't directly use the retrieve snapshot endpoint
+# We need to use the enumerate snapshot endpoint with a filter for the snapshot id
+def retrieve_snapshot_via_enumerate(clients, snapshot_id):
+    snapshots_response = clients.snapshots_api.enumerate_snapshots(filter=snapshot_id)
+    for snapshot in snapshots_response.items:
+        if snapshot.id == snapshot_id:
+            return snapshot
+    return None
+
+
+def retrieve_dataset_via_enumerate(clients, dataset_id):
+    datasets_response = clients.datasets_api.enumerate_datasets(filter=dataset_id)
+    for dataset in datasets_response.items:
+        if dataset.id == dataset_id:
+            return dataset
+    return None
diff --git a/tools/logCollection/src/populate_user_details_from_data_warehouse.py b/tools/logCollection/src/populate_user_details_from_data_warehouse.py
new file mode 100644
index 0000000000..b68f5722ef
--- /dev/null
+++ b/tools/logCollection/src/populate_user_details_from_data_warehouse.py
@@ -0,0 +1,62 @@
+from google.cloud import bigquery
+from google.cloud import secretmanager
+
+def populate_user_details(populated_new_logs, emails):
+    # Get user details from Thurloe/data warehouse
+    bq_client = bigquery.Client()
+    sm_client = secretmanager.SecretManagerServiceClient()
+    # Build the resource name of the secret.
+    name = f"projects/terra-datarepo-production/secrets/warehouse-db-name/versions/latest"
+    response = sm_client.access_secret_version(request={"name": name})
+    WAREHOUSE_DB_NAME = response.payload.data.decode("UTF-8")
+
+    # # Perform a query.
+    QUERY = (
+        f"""
+        SELECT t.USER_ID, e.email, t.Key, t.Value FROM `{WAREHOUSE_DB_NAME}.thurloe` t
+       JOIN `{WAREHOUSE_DB_NAME}.email` e ON t.USER_ID = e.USER_ID
+        WHERE t.KEY IN ("firstName", "lastName", "institute", "programLocationCountry")
+        AND t.USER_ID IN(
+            SELECT USER_ID FROM `{WAREHOUSE_DB_NAME}.email`
+            WHERE email IN UNNEST(@emails)
+        )
+        ORDER BY t.USER_ID
+        """
+    )
+
+    job_config = bigquery.QueryJobConfig(
+        query_parameters=[
+            bigquery.ArrayQueryParameter("emails", "STRING", emails)
+        ]
+    )
+
+    query_job = bq_client.query(QUERY, job_config=job_config)  # API request
+    rows = query_job.result()  # Waits for query to finish
+
+    first_names = {}
+    last_names = {}
+    institutes = {}
+    countries = {}
+    user_ids = {}
+    for row in rows:
+        user_ids[row.email] = row.USER_ID
+        if (row.Key == "firstName"):
+            first_names[row.email] = row.Value
+        elif (row.Key == "lastName"):
+            last_names[row.email] = row.Value
+        elif (row.Key == "institute"):
+            institutes[row.email] = row.Value
+        elif (row.Key == "programLocationCountry"):
+            countries[row.email] = row.Value
+
+    for log in populated_new_logs:
+        if log.user_email != "N/A":
+            if log.user_email in first_names:
+                log.user_name = first_names[log.user_email] + " " + last_names[log.user_email]
+                log.user_country_name = countries[log.user_email]
+                log.user_id = user_ids[log.user_email]
+                if institutes[log.user_email] is not None:
+                    log.user_org = institutes[log.user_email]
+            else:
+                log.user_name = ""
+                log.user_country_name = ""
diff --git a/tools/logCollection/src/tests/event_type_test.py b/tools/logCollection/src/tests/event_type_test.py
new file mode 100644
index 0000000000..2c248f5f58
--- /dev/null
+++ b/tools/logCollection/src/tests/event_type_test.py
@@ -0,0 +1,50 @@
+import unittest
+from ..categorize_event_type import event_type_from_url, EventType
+import uuid
+from parameterized import parameterized
+
+# Run tests with the following command:
+# `python3 -m unittest src/tests/event_type_test.py`
+
+id = uuid.uuid4()
+table_name = "tableName123"
+column_name = "columnName123"
+object_id = f'v1_{id}_{id}'
+access_id = "accessId123"
+class EventTypeTest(unittest.TestCase):
+    @parameterized.expand([
+        # Data Access
+        (f'/api/repository/v1/datasets/{id}/data/{table_name}', "POST", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/datasets/{id}/data/{table_name}/statistics/{column_name}', "POST", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/datasets/{id}/files', "GET", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/datasets/{id}/files/{id}', "GET", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/datasets/{id}/filesystem/objects', "GET", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/snapshots/{id}/data/{table_name}', "POST", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/snapshots/{id}/data/{table_name}/statistics/{column_name}', "POST", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/snapshots/{id}/files', "GET", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/snapshots/{id}/export', "GET", EventType.DATA_ACCESS),
+        (f'/api/repository/v1/snapshots/{id}/files/{id}', "GET", EventType.DATA_ACCESS),
+        (f'/ga4gh/drs/v1/objects/{object_id}', "GET", EventType.DATA_ACCESS),
+        (f'/ga4gh/drs/v1/objects/{object_id}/access/{access_id}', "GET", EventType.DATA_ACCESS),
+        (f'/ga4gh/drs/v1/objects/{object_id}', "POST", EventType.DATA_ACCESS),
+        (f'/ga4gh/drs/v1/objects/{object_id}/access/{access_id}', "POST", EventType.DATA_ACCESS),
+         # Data Uploads
+        (f'/api/repository/v1/datasets/{id}/ingest', "POST", EventType.DATA_UPLOAD),
+        (f'/api/repository/v1/datasets/{id}/files', "POST", EventType.DATA_UPLOAD),
+        (f'/api/repository/v1/datasets/{id}/files/bulk', "POST", EventType.DATA_UPLOAD),
+        (f'/api/repository/v1/datasets/{id}/files/bulk/array', "POST", EventType.DATA_UPLOAD),
+        # Data Deletions
+        (f'/api/repository/v1/snapshots/{id}', "DELETE", EventType.DATA_DELETION),
+        (f'/api/repository/v1/datasets/{id}', "DELETE", EventType.DATA_DELETION),
+        (f'/api/repository/v1/datasets/{id}/deletes', "POST", EventType.DATA_DELETION),
+        (f'/api/repository/v1/datasets/{id}/files/{id}', "DELETE", EventType.DATA_DELETION),
+        # Other
+        (f'/api/repository/v1/snapshots/{id}', "GET", EventType.OTHER),
+        (f'/api/repository/v1/datasets/{id}', "GET", EventType.OTHER)
+    ])
+
+    def test_determines_data_access_event_for_data_url(self, url, method, expectedEventType):
+        self.assertEqual(event_type_from_url(url, method), expectedEventType)
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/tools/logCollection/src/tests/populate_tdr_details_test.py b/tools/logCollection/src/tests/populate_tdr_details_test.py
new file mode 100644
index 0000000000..deaf98f85e
--- /dev/null
+++ b/tools/logCollection/src/tests/populate_tdr_details_test.py
@@ -0,0 +1,56 @@
+import unittest
+from ..populate_tdr_details import retrieve_snapshot_auth_domain, TDRClients, populate_snapshot_user_permission_group_and_studies, retrieve_snapshot_via_enumerate, retrieve_dataset_via_enumerate, populate_dataset_studies
+from ..log_class import LogClass
+# Run tests with the following command:
+# `python3 -m unittest src/tests/populate_tdr_details_test.py`
+
+# example snapshot that firecloud accounts do not have direct access to,
+# but still can access the policies endpoint
+# Full_View_Snapshot_of_NonStewardDataset_1746106084627
+# has auth domain
+prod_example_snapshot_id = "49c010f6-1112-4839-b87d-8dfcfe5407a7"
+prod_example_snapshot_auth_domain = "AXIN_Testing_Group"
+prod_example_dataset_id = "1e81193f-73f7-4923-91ed-e302a4b4eea5"
+host = "https://data.terra.bio"
+clients = TDRClients(host)
+class PopulateTdrTest(unittest.TestCase):
+    def test_retrieve_snapshot_auth_domain(self):
+        self.assertEqual(retrieve_snapshot_auth_domain(clients, prod_example_snapshot_id), [prod_example_snapshot_auth_domain])
+
+    def test_retrieve_snapshot_via_enumerate(self):
+        snapshot = retrieve_snapshot_via_enumerate(clients, prod_example_snapshot_id)
+        self.assertEqual(snapshot.id, prod_example_snapshot_id)
+
+    def test_retrieve_dataset_via_enumerate(self):
+        dataset = retrieve_dataset_via_enumerate(clients, prod_example_dataset_id)
+        self.assertEqual(dataset.id, prod_example_dataset_id)
+
+
+    def test_populate_user_permission_group(self):
+        # populate test data
+        populated_new_logs = []
+        example_log = LogClass()
+        example_log.snapshot_id = prod_example_snapshot_id
+        populated_new_logs.append(example_log)
+        snapshot_ids = {prod_example_snapshot_id}
+        # run populate method
+        populate_snapshot_user_permission_group_and_studies(populated_new_logs, snapshot_ids, host)
+        # confirm auth domains were added to the logs
+        self.assertEqual(populated_new_logs[0].user_permission_group, [prod_example_snapshot_auth_domain])
+        # empty list since we're testing against prod, but manually confirmed duos id gets added
+        self.assertEqual(populated_new_logs[0].associated_study, [])
+
+    def test_populate_dataset_studies(self):
+        # populate test data
+        populated_new_logs = []
+        example_log = LogClass()
+        example_log.dataset_id = prod_example_dataset_id
+        populated_new_logs.append(example_log)
+        dataset_ids = {prod_example_dataset_id}
+        # run populate method
+        populate_dataset_studies(populated_new_logs, dataset_ids, host)
+        # None since we're testing against prod
+        self.assertIsNone(populated_new_logs[0].associated_study)
+
+if __name__ == '__main__':
+    unittest.main()


Reviews:
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-28T18:34:09Z
user: login: snf2ye
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T19:20:50Z
user: login: fboulnois
body: looks good, thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T13:19:02Z
user: login: s-rubenstein
body: This looks reasonable to me. One comment about the output files.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T20:25:39Z
user: login: rjohanek
body: looks good to me!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T17:33:33Z
user: login: fboulnois
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T13:13:53Z
user: login: fboulnois
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T13:14:19Z
user: login: snf2ye
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T13:57:14Z
user: login: snf2ye
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T14:12:56Z

Review comments:
diff_hunk: @@ -68,8 +67,15 @@
       userEmail = UNAUTHED_EMAIL;
       institute = UNAUTHED_INSTITUTE;
     }
-
+    String srcIpAddress = request.getRemoteAddr();
+    String destIPAddress = request.getLocalAddr();
+    String destPort = Integer.toString(request.getLocalPort());
+    String sessionId = request.getSession().getId();
path: src/main/java/bio/terra/app/logging/LoggerInterceptor.java
user: login: github-advanced-security[bot]
body: ## "HttpServletRequest.getRequestedSessionId()" should not be used

<!--SONAR_ISSUE_KEY:AZZ9kgtXKCN7UpBrGu36-->Remove use of this unsecured "getRequestedSessionId()" method <p>See more on <a href="https://sonarcloud.io/project/issues?id=DataBiosphere_jade-data-repo&issues=AZZ9kgtXKCN7UpBrGu36&open=AZZ9kgtXKCN7UpBrGu36&pullRequest=1966">SonarQube Cloud</a></p>

[Show more details](https://github.com/DataBiosphere/jade-data-repo/security/code-scanning/5)
created_at: 2025-04-28T18:34:09Z
author_association: NONE
diff_hunk: @@ -68,8 +67,15 @@
       userEmail = UNAUTHED_EMAIL;
       institute = UNAUTHED_INSTITUTE;
     }
-
+    String srcIpAddress = request.getRemoteAddr();
+    String destIPAddress = request.getLocalAddr();
+    String destPort = Integer.toString(request.getLocalPort());
+    String sessionId = request.getSession().getId();
path: src/main/java/bio/terra/app/logging/LoggerInterceptor.java
user: login: snf2ye
body: I think this is old - I already switched to the compliant solution.
created_at: 2025-04-28T19:20:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,2 @@
+_time,src_ip,dest_ip,dest_port,user_name,user_id,user_id_provider,session_id,method,url,app,http_user_agent,status,http_content_type,bytes,duration,nih_ico,cadr_name,user_country_name,user_org,user_email,associated_study,eRA_commons_id,user_permission_group,event_type,dataset_id,snapshot_id
path: tools/logCollection/output/sample_tdr_logs_2025-05-01 10:51:09.527812.csv
user: login: s-rubenstein
body: Do you need this output? Should we mark output as ignored?
created_at: 2025-05-01T20:19:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,30 @@
+[
path: tools/logCollection/rawLogs/sample_tdr_logs.json
user: login: s-rubenstein
body: Same thing, do we want to include this or mark as ignored
created_at: 2025-05-01T20:20:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,2 @@
+_time,src_ip,dest_ip,dest_port,user_name,user_id,user_id_provider,session_id,method,url,app,http_user_agent,status,http_content_type,bytes,duration,nih_ico,cadr_name,user_country_name,user_org,user_email,associated_study,eRA_commons_id,user_permission_group,event_type,dataset_id,snapshot_id
path: tools/logCollection/output/sample_tdr_logs_2025-05-01 10:51:09.527812.csv
user: login: fboulnois
body: Indeed, I think this should be excluded!
created_at: 2025-05-05T13:13:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,30 @@
+[
path: tools/logCollection/rawLogs/sample_tdr_logs.json
user: login: fboulnois
body: Ditto!
created_at: 2025-05-05T13:14:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,2 @@
+_time,src_ip,dest_ip,dest_port,user_name,user_id,user_id_provider,session_id,method,url,app,http_user_agent,status,http_content_type,bytes,duration,nih_ico,cadr_name,user_country_name,user_org,user_email,associated_study,eRA_commons_id,user_permission_group,event_type,dataset_id,snapshot_id
path: tools/logCollection/output/sample_tdr_logs_2025-05-01 10:51:09.527812.csv
user: login: snf2ye
body: Oh, okay. I thought a sample could be helpful to understanding the desired output if someone runs into issues with the script, but it would be nice to include this whole folder in gitignore. I can do that. 
created_at: 2025-05-05T13:57:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,30 @@
+[
path: tools/logCollection/rawLogs/sample_tdr_logs.json
user: login: snf2ye
body: I think the example input would be helpful for debugging if the users run into issues with the scripts, like if the tdr logs change format. But, I can add the rest of the folder to gitignore
created_at: 2025-05-05T14:12:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-05-05T14:25:30Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=DataBiosphere_jade-data-repo&pullRequest=1966) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=DataBiosphere_jade-data-repo&pullRequest=1966&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=DataBiosphere_jade-data-repo&pullRequest=1966&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=DataBiosphere_jade-data-repo&pullRequest=1966&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=DataBiosphere_jade-data-repo&pullRequest=1966&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=DataBiosphere_jade-data-repo&pullRequest=1966&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=DataBiosphere_jade-data-repo&pullRequest=1966)


author_association: NONE