Repository: google/cloud-forensics-utils

Issue:
GCP log query should handle searching more than one resource
It would be useful if we were able to execute a single query against all available resources (projects, organizations etc.).

`GoogleCloudLog` currently accepts a single project ID and uses that for where to perform its search. This is probably fine for most use cases, but fails to address:

- A user wanting to perform a search across all available resources

- A user wanting to search across some combination of 1+ resource (e.g. 10 out of their 50 projects)

Based on the API docs, it looks like the [`resourceNames`](https://github.com/google/cloud-forensics-utils/blob/master/libcloudforensics/providers/gcp/internal/log.py#L102) field has the ability to accept a list of 1+ resource to query. 

If that's not the case, and while not ideal, it may be useful to at least have the ability to loop through X resources and issue said query on behalf of the user.

This improvement would help relieve some of the burden users face when using other options where they're limited to searching a single resource (`gcloud`, `Logs Viewer` etc.)... which usually results in something along the lines of looping through each project and executing said query (not ideal if you have hundreds of projects).

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/google/cloud-forensics-utils/pull/335?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google) Report
> :exclamation: No coverage uploaded for pull request base (`main@943ebc9`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google#section-missing-base-commit).
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/google/cloud-forensics-utils/pull/335/graphs/tree.svg?width=650&height=150&src=pr&token=zZkPno3ah0&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google)](https://codecov.io/gh/google/cloud-forensics-utils/pull/335?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google)

```diff
@@           Coverage Diff           @@
##             main     #335   +/-   ##
=======================================
  Coverage        ?   63.74%           
=======================================
  Files           ?       33           
  Lines           ?     2452           
  Branches        ?        0           
=======================================
  Hits            ?     1563           
  Misses          ?      889           
  Partials        ?        0           
```

| Flag | Coverage Δ | |
|---|---|---|
| nosetests | `63.74% <0.00%> (?)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google#carryforward-flags-in-the-pull-request-comment) to find out more.


------

[Continue to review full report at Codecov](https://codecov.io/gh/google/cloud-forensics-utils/pull/335?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/google/cloud-forensics-utils/pull/335?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google). Last update [943ebc9...e84afd2](https://codecov.io/gh/google/cloud-forensics-utils/pull/335?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=google).


Patch:
diff --git a/libcloudforensics/providers/gcp/internal/log.py b/libcloudforensics/providers/gcp/internal/log.py
index 1a63b7d..3d2dc97 100644
--- a/libcloudforensics/providers/gcp/internal/log.py
+++ b/libcloudforensics/providers/gcp/internal/log.py
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """Google Cloud Logging functionalities."""
-
+from typing import Optional
 from typing import TYPE_CHECKING, List, Dict, Any
 
 from libcloudforensics.providers.gcp.internal import common
@@ -26,7 +26,7 @@ class GoogleCloudLog:
   """Class representing a Google Cloud Logs interface.
 
   Attributes:
-    project_id: Google Cloud project ID.
+    project_ids: List of Google Cloud project IDs.
     gcl_api_client: Client to interact with GCP logging API.
 
   Example use:
@@ -39,14 +39,13 @@ class GoogleCloudLog:
 
   LOGGING_API_VERSION = 'v2'
 
-  def __init__(self, project_id: str) -> None:
+  def __init__(self, project_ids: List[str]) -> None:
     """Initialize the GoogleCloudProject object.
 
     Args:
-      project_id (str): The name of the project.
+      project_ids (List[str]): List of project IDs.
     """
-
-    self.project_id = project_id
+    self.project_ids = project_ids
     self.gcl_api_client = None
 
   def GclApi(self) -> 'googleapiclient.discovery.Resource':
@@ -75,20 +74,23 @@ class GoogleCloudLog:
 
     logs = []
     gcl_instance_client = self.GclApi().logs()
-    responses = common.ExecuteRequest(
-        gcl_instance_client, 'list', {'parent': 'projects/' + self.project_id})
-
-    for response in responses:
-      for logtypes in response.get('logNames', []):
-        logs.append(logtypes)
+    for project_id in self.project_ids:
+      responses = common.ExecuteRequest(
+          gcl_instance_client,
+          'list',
+          {'parent': 'projects/' + project_id})
+      for response in responses:
+        for logtypes in response.get('logNames', []):
+          logs.append(logtypes)
 
     return logs
 
-  def ExecuteQuery(self, qfilter: str) -> List[Dict[str, Any]]:
+  def ExecuteQuery(
+      self, qfilter: Optional[List[str]] = None) -> List[Dict[str, Any]]:
     """Query logs in GCP project.
 
     Args:
-      qfilter (str): The query filter to use.
+      qfilter (List[str]): Optional. A list of query filters to use.
 
     Returns:
       List[Dict]: Log entries returned by the query, e.g. [{'projectIds':
@@ -96,21 +98,27 @@ class GoogleCloudLog:
 
     Raises:
       RuntimeError: If API call failed.
+      ValueError: If the number of project IDs being queried doesn't match
+          the number of provided filters.
     """
 
-    body = {
-        'resourceNames': 'projects/' + self.project_id,
-        'filter': qfilter,
-        'orderBy': 'timestamp desc',
-    }
-
     entries = []
     gcl_instance_client = self.GclApi().entries()
-    responses = common.ExecuteRequest(
-        gcl_instance_client, 'list', {'body': body}, throttle=True)
-
-    for response in responses:
-      for entry in response.get('entries', []):
-        entries.append(entry)
 
+    if qfilter and len(self.project_ids) != len(qfilter):
+      raise ValueError(
+          'Several project IDs detected ({0:d}) but only {1:d} query filters '
+          'provided.'.format(len(self.project_ids), len(qfilter)))
+
+    for idx, project_id in enumerate(self.project_ids):
+      body = {
+          'resourceNames': 'projects/' + project_id,
+          'filter': qfilter[idx] if qfilter else '',
+          'orderBy': 'timestamp desc',
+      }
+      responses = common.ExecuteRequest(
+          gcl_instance_client, 'list', {'body': body}, throttle=True)
+      for response in responses:
+        for entry in response.get('entries', []):
+          entries.append(entry)
     return entries
diff --git a/libcloudforensics/providers/gcp/internal/project.py b/libcloudforensics/providers/gcp/internal/project.py
index 13601cf..877708d 100644
--- a/libcloudforensics/providers/gcp/internal/project.py
+++ b/libcloudforensics/providers/gcp/internal/project.py
@@ -145,7 +145,7 @@ class GoogleCloudProject:
     if self._log:
       return self._log
     self._log = log_module.GoogleCloudLog(
-        self.project_id)
+        [self.project_id])
     return self._log
 
   @property
diff --git a/tools/cli.py b/tools/cli.py
index c38b463..aeaeecf 100644
--- a/tools/cli.py
+++ b/tools/cli.py
@@ -324,7 +324,9 @@ def Main() -> None:
   gcp_parser.add_argument(
       '--project', help='GCP project ID. If not provided, the library will look'
                         ' for a project ID configured with your gcloud SDK. If '
-                        'none found, errors.')
+                        'none found, errors. For GCP logs operations, a list of'
+                        ' project IDs can be passed, as a comma-separated '
+                        'string: project_id1,project_id2,...')
   gcp_subparsers = gcp_parser.add_subparsers()
   AddParser('gcp', gcp_subparsers, 'listinstances',
             'List GCE instances in GCP project.')
@@ -369,7 +371,10 @@ def Main() -> None:
             ])
   AddParser('gcp', gcp_subparsers, 'querylogs', 'Query GCP logs.',
             args=[
-                ('--filter', 'Query filter', None),
+                ('--filter', 'Query filter. If querying multiple logs / '
+                             'multiple project IDs, enter each filter in a '
+                             'single string that is comma-separated: '
+                             '--filter="filter1,filter2,..."', None),
                 ('--start', 'Start date for query (2020-05-01T11:13:00Z)',
                  None),
                 ('--end', 'End date for query (2020-05-01T11:13:00Z)', None)
diff --git a/tools/gcp_cli.py b/tools/gcp_cli.py
index 6b70826..291e683 100644
--- a/tools/gcp_cli.py
+++ b/tools/gcp_cli.py
@@ -140,7 +140,7 @@ def ListLogs(args: 'argparse.Namespace') -> None:
 
   AssignProjectID(args)
 
-  logs = gcp_log.GoogleCloudLog(args.project)
+  logs = gcp_log.GoogleCloudLog(args.project.split(','))
   results = logs.ListLogs()
   logger.info('Found {0:d} available log types:'.format(len(results)))
   for line in results:
@@ -161,7 +161,7 @@ def QueryLogs(args: 'argparse.Namespace') -> None:
 
   AssignProjectID(args)
 
-  logs = gcp_log.GoogleCloudLog(args.project)
+  logs = gcp_log.GoogleCloudLog(args.project.split(','))
 
   try:
     if args.start:
@@ -186,7 +186,7 @@ def QueryLogs(args: 'argparse.Namespace') -> None:
   elif args.filter:
     qfilter += args.filter
 
-  results = logs.ExecuteQuery(qfilter)
+  results = logs.ExecuteQuery(qfilter.split(',') if qfilter else None)
   logger.info('Found {0:d} log entries:'.format(len(results)))
   for line in results:
     logger.info(json.dumps(line))


Test patch:
diff --git a/tests/providers/gcp/gcp_mocks.py b/tests/providers/gcp/gcp_mocks.py
index 364b3c4..f702f0a 100644
--- a/tests/providers/gcp/gcp_mocks.py
+++ b/tests/providers/gcp/gcp_mocks.py
@@ -58,7 +58,7 @@ FAKE_SNAPSHOT_LONG_NAME = compute.GoogleComputeSnapshot(
 FAKE_DISK_COPY = compute.GoogleComputeDisk(
     FAKE_SOURCE_PROJECT.project_id, 'fake-zone', 'fake-disk-copy')
 
-FAKE_LOGS = gcp_log.GoogleCloudLog('fake-target-project')
+FAKE_LOGS = gcp_log.GoogleCloudLog(['fake-target-project'])
 
 FAKE_LOG_LIST = [
     'projects/fake-target-project/logs/GCEGuestAgent',
diff --git a/tests/providers/gcp/internal/test_log.py b/tests/providers/gcp/internal/test_log.py
index 9df0e01..eeb77ed 100644
--- a/tests/providers/gcp/internal/test_log.py
+++ b/tests/providers/gcp/internal/test_log.py
@@ -41,7 +41,7 @@ class GoogleCloudLogTest(unittest.TestCase):
     """Test that logs of project are correctly queried."""
     query = mock_gcl_api.return_value.entries.return_value.list
     query.return_value.execute.return_value = gcp_mocks.MOCK_LOG_ENTRIES
-    qfilter = '*'
+    qfilter = ['*']
     query_logs = gcp_mocks.FAKE_LOGS.ExecuteQuery(qfilter)
     self.assertEqual(2, len(query_logs))
     self.assertEqual(gcp_mocks.FAKE_LOG_ENTRIES[0], query_logs[0])