Repository: sosreport/sos

Issue:
[gcp] New plugin for Google Cloud Platform
Hi!

I'm planning to write a new plugin to better support Virtual Machines running on Google Cloud Platform. Similar to what is already implemented for [Azure](https://github.com/sosreport/sos/blob/main/sos/report/plugins/azure.py).

I plan to capture following info:
 * All information that can be extracted from [Metadata server](https://cloud.google.com/compute/docs/metadata/overview).
 * Get output of `gcloud auth list` - if it's available
 * Grab journal logs for any Google-related services 

When would it run?
Only on VMs running on GCP - meaning a `curl` to `http://metadata.google.internal/computeMetadata/v1/` is successful.

My questions:
 * Is `gcp` a good name for such plugin, or would `google-cloud` be better?
 * Should all this data be collected inside one plugin, or is it better to divide those into for example: `gcp-metadata` and `gcp-services`?


Discussion:
That plugin can have sense.

> When would it run?
> Only on VMs running on GCP - meaning a `curl` to `http://metadata.google.internal/computeMetadata/v1/` is successful.

You mean the plugin enabledness condition would be the curl request? Assume a system (not gCloud VM) with DNS issues where such curl would hang - then sos report would hang even during startup phase. Can't be a test like "is `gcloud` executable?" work well?

> My questions:
> 
>     * Is `gcp` a good name for such plugin, or would `google-cloud` be better?

No strong opinion here. On one side, "gcp" is quite concise such that one must know the acronym to understand it. On the other side, the acronym is well(?) known.

>     * Should all this data be collected inside one plugin, or is it better to divide those into for example: `gcp-metadata` and `gcp-services`?

If the data will be collected everytime altogether (under the same enablement conditions), I would stick with one plugin.
> > When would it run?
> > Only on VMs running on GCP - meaning a `curl` to `http://metadata.google.internal/computeMetadata/v1/` is successful.
> 
> You mean the plugin enabledness condition would be the curl request? Assume a system (not gCloud VM) with DNS issues where such curl would hang - then sos report would hang even during startup phase. Can't be a test like "is `gcloud` executable?" work well?
> 

Since this is for VMs in GCP, `gcloud` presence isn't a great indicator. You don't have to have `gcloud` installed on GCP VMs, and it can be installed on systems outside GCP to leverage GCP resources.

A `curl` while not ideal, could use a short timeout to avoiding hanging our setup phase.

Alternatively, is there some intrinsic attribute on the VM that could be used? E.G. something GCP specific from something like `dmesg` or `dmidecode`?

> > My questions:
> > ```
> > * Is `gcp` a good name for such plugin, or would `google-cloud` be better?
> > ```
> 
> No strong opinion here. On one side, "gcp" is quite concise such that one must know the acronym to understand it. On the other side, the acronym is well(?) known.
> 
> > ```
> > * Should all this data be collected inside one plugin, or is it better to divide those into for example: `gcp-metadata` and `gcp-services`?
> > ```
> 
> If the data will be collected everytime altogether (under the same enablement conditions), I would stick with one plugin.

Agree with single plugin. `gcp` seems fine for me. 

I agree that checking for `gcloud` being available is not a good indicator - developers install gcloud on their local machines to control cloud instances etc. And `gcloud` is definitely not guaranteed to be installed on VMs.

I have checked `dmesg` of an instance, and I like what I'm seeing. There is this line for example:
```
[    0.000000] DMI: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
```
Should be relatively easy to check.

As for using `dmidecode` - can we count on it being available? Is is part of every Linux distribution?

If `dmidecode` is really common, then it also has good indicators I guess:

```
~$ sudo dmidecode
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.4 present.
14 structures occupying 546 bytes.
Table at 0xBE9CC000.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Google
        Version: Google
        Release Date: 01/01/2011
        Address: 0xE8000
        Runtime Size: 96 kB
        ROM Size: 64 kB
        Characteristics:
                BIOS characteristics not supported
                Targeted content distribution is supported
        BIOS Revision: 1.0
Handle 0x0097, DMI type 1, 27 bytes
System Information
        Manufacturer: Google
        Product Name: Google Compute Engine
        Version: Not Specified
        Serial Number: GoogleCloud-98E621D0FDC0576BB1A81BDCC7E66F61
        UUID: 98e621d0-fdc0-576b-b1a8-1bdcc7e66f61
        Wake-up Type: Power Switch
        SKU Number: Not Specified
        Family: Not Specified
Handle 0x0098, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Google
        Product Name: Google Compute Engine
        Version: Not Specified
        Serial Number: Board-GoogleCloud-98E621D0FDC0576BB1A81BDCC7E66F61
        Asset Tag: 98E621D0-FDC0-576B-B1A8-1BDCC7E66F61
        Features:
                Board is a hosting board
        Location In Chassis: Not Specified
        Chassis Handle: 0x0099
        Type: Motherboard
        Contained Object Handles: 0
```

> I have checked `dmesg` of an instance, and I like what I'm seeing. There is this line for example:
> 
> ```
> [    0.000000] DMI: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> ```
> 
> Should be relatively easy to check.
> 
> As for using `dmidecode` - can we count on it being available? Is is part of every Linux distribution?
> 

It's part of every Linux distro we currently support. Beyond that, I would honestly be beside myself if there was a distro out there not shipping either/both of these.

Resolving patch:
diff --git a/sos/report/plugins/gcp.py b/sos/report/plugins/gcp.py
new file mode 100644
--- /dev/null
+++ b/sos/report/plugins/gcp.py
@@ -0,0 +1,145 @@
+# This file is part of the sos project: https://github.com/sosreport/sos
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# version 2 of the GNU General Public License.
+#
+# See the LICENSE file in the source distribution for further information.
+import json
+from http.client import HTTPResponse
+from typing import Any
+from urllib import request
+from urllib.error import URLError
+
+from sos.report.plugins import Plugin, IndependentPlugin, PluginOpt
+
+
+class GCP(Plugin, IndependentPlugin):
+
+    short_desc = 'Google Cloud Platform'
+    plugin_name = 'gcp'
+    profiles = ('virt',)
+
+    option_list = [
+        PluginOpt('keep-pii', default=False,
+                  desc="Stop the plugin from removing PIIs like project name "
+                       "or organization ID from the metadata retrieved from "
+                       "Metadata server.")
+    ]
+
+    METADATA_ROOT = "http://metadata.google.internal/computeMetadata/v1/"
+    METADATA_QUERY = "http://metadata.google.internal/computeMetadata/v1/" \
+                     "?recursive=true"
+    REDACTED = "[--REDACTED--]"
+
+    # A line we will be looking for in the dmesg output. If it's there,
+    # that means we're running on a Google Cloud Compute instance.
+    GOOGLE_DMI = "DMI: Google Google Compute Engine/Google " \
+                 "Compute Engine, BIOS Google"
+
+    def check_enabled(self):
+        """
+        Checks if this plugin should be executed at all. In this case, it
+        will check the `dmesg` command output to see if the system is
+        running on a Google Cloud Compute instance.
+        """
+        dmesg = self.exec_cmd("dmesg")
+        if dmesg['status'] != 0:
+            return False
+        return self.GOOGLE_DMI in dmesg['output']
+
+    def setup(self):
+        """
+        Collect the following info:
+         * Metadata from the Metadata server
+         * `gcloud auth list` output
+         * Any google services output from journal
+        """
+
+        # Capture gcloud auth list
+        self.add_cmd_output("gcloud auth list", tags=['gcp'])
+
+        # Get and store Metadata
+        try:
+            self.metadata = self.get_metadata()
+            self.scrub_metadata()
+            self.add_string_as_file(json.dumps(self.metadata, indent=4),
+                                    "metadata.json", plug_dir=True,
+                                    tags=['gcp'])
+        except RuntimeError as err:
+            self.add_string_as_file(str(err), 'metadata.json',
+                                    plug_dir=True, tags=['gcp'])
+
+        # Add journal entries
+        self.add_journal(units="google*", tags=['gcp'])
+
+    def get_metadata(self) -> dict:
+        """
+        Retrieves metadata from the Metadata Server and transforms it into a
+        dictionary object.
+        """
+        response = self._query_address(self.METADATA_QUERY)
+        response_body = response.read().decode()
+        return json.loads(response_body)
+
+    @staticmethod
+    def _query_address(url: str) -> HTTPResponse:
+        """
+        Query the given url address with headers required by Google Metadata
+        Server.
+        """
+        try:
+            req = request.Request(url, headers={'Metadata-Flavor': 'Google'})
+            response = request.urlopen(req)
+        except URLError as err:
+            raise RuntimeError(
+                "Failed to communicate with Metadata Server: " + str(err))
+        if response.code != 200:
+            raise RuntimeError(
+                f"Failed to communicate with Metadata Server "
+                f"(code: {response.code}): " + response.read().decode())
+        return response
+
+    def scrub_metadata(self):
+        """
+        Remove all PII information from metadata, unless a keep-pii option
+        is specified.
+
+        Note: PII information collected by this plugin, like
+        project number, account names etc. might be required by Google
+        Cloud Support for faster issue resolution.
+        """
+        if self.get_option('keep-pii'):
+            return
+
+        project_id = self.metadata['project']['projectId']
+        project_number_int = self.metadata['project']['numericProjectId']
+        project_number = str(project_number_int)
+
+        def scrub(data: Any) -> Any:
+            if isinstance(data, dict):
+                if 'token' in data:
+                    # Data returned for recursive query shouldn't contain
+                    # tokens, but you can't be too careful.
+                    data['token'] = self.REDACTED
+                return {scrub(k): scrub(v) for k, v in data.items()}
+            elif isinstance(data, list):
+                return [scrub(value) for value in data]
+            elif isinstance(data, str):
+                return data.replace(project_number, self.REDACTED)\
+                           .replace(project_id, self.REDACTED)
+            elif isinstance(data, int):
+                return self.REDACTED if data == project_number_int else data
+            return data
+
+        self.metadata = scrub(self.metadata)
+
+        self.safe_redact_key(self.metadata['project']['attributes'],
+                             'ssh-keys')
+        self.safe_redact_key(self.metadata['project']['attributes'],
+                             'sshKeys')
+
+    @classmethod
+    def safe_redact_key(cls, dict_obj: dict, key: str):
+        if key in dict_obj:
+            dict_obj[key] = cls.REDACTED


Test patch: