Repository: googleapis/google-auth-library-python

Issue:
Getting project ID using Application Default Credentials fails when gcloud command writes anything to stderr
  - OS: Ubuntu 20.04
  - Python version: 3.8
  - pip version: 20.0.2
  - `google-auth` version: 1.19.2

#### Steps to reproduce

  1. Arrange for gcloud to throw a warning. For example I'm suffering from this https://github.com/GoogleCloudPlatform/gsutil/issues/999
  2. Attempt to use ADC e.g. `credentials, project = google.auth.default()`
  3. Note that project always comes back at None even if `gcloud config set project` is correctly set
  4. Root cause seems to be that in _cloud_sdk.py/get_project_id() the subprocess.check_output command merges stderr and stdout. So in the case that stderr is not empty and the subprocess does not fail, you might get badly formed JSON on which json.loads a few lines later chokes.

For example, my raw gcloud output is like:

/snap/google-cloud-sdk/165/lib/third_party/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.\n  warnings.warn(warning, RequestsDependencyWarning)\n{\n  "configuration": {\n    "active_configuration": "default",\n    "properties": {\n      "core": {\n        "account": "me@myemailaddress.com",\n        "disable_usage_reporting": "False",\n        "project": "my-test-project"\n      },\n      "deployment_manager": {\n        "glob_imports": "True"\n      }\n    }\n  },\n  "credential": {\n    "access_token".... etc etc.

Expected behaviour: non-fatal errors or warnings from gcloud should not corrupt the output and cause the project ID lookup to fail.

Discussion:

Patch:
diff --git a/google/auth/_cloud_sdk.py b/google/auth/_cloud_sdk.py
index e772fe9..40e6aec 100644
--- a/google/auth/_cloud_sdk.py
+++ b/google/auth/_cloud_sdk.py
@@ -84,6 +84,13 @@ def get_application_default_credentials_path():
     return os.path.join(config_path, _CREDENTIALS_FILENAME)
 
 
+def _run_subprocess_ignore_stderr(command):
+    """ Return subprocess.check_output with the given command and ignores stderr."""
+    with open(os.devnull, "w") as devnull:
+        output = subprocess.check_output(command, stderr=devnull)
+    return output
+
+
 def get_project_id():
     """Gets the project ID from the Cloud SDK.
 
@@ -96,9 +103,9 @@ def get_project_id():
         command = _CLOUD_SDK_POSIX_COMMAND
 
     try:
-        output = subprocess.check_output(
-            (command,) + _CLOUD_SDK_CONFIG_COMMAND, stderr=subprocess.STDOUT
-        )
+        # Ignore the stderr coming from gcloud, so it won't be mixed into the output.
+        # https://github.com/googleapis/google-auth-library-python/issues/673
+        output = _run_subprocess_ignore_stderr((command,) + _CLOUD_SDK_CONFIG_COMMAND)
     except (subprocess.CalledProcessError, OSError, IOError):
         return None
 


Test patch:
diff --git a/tests/test__cloud_sdk.py b/tests/test__cloud_sdk.py
index 3377604..31cb6c2 100644
--- a/tests/test__cloud_sdk.py
+++ b/tests/test__cloud_sdk.py
@@ -71,6 +71,25 @@ def test_get_project_id_call_error(check_output):
     assert check_output.called
 
 
+def test__run_subprocess_ignore_stderr():
+    command = [
+        "python",
+        "-c",
+        "from __future__ import print_function;"
+        + "import sys;"
+        + "print('error', file=sys.stderr);"
+        + "print('output', file=sys.stdout)",
+    ]
+
+    # If we ignore stderr, then the output only has stdout
+    output = _cloud_sdk._run_subprocess_ignore_stderr(command)
+    assert output == b"output\n"
+
+    # If we pipe stderr to stdout, then the output is mixed with stdout and stderr.
+    output = subprocess.check_output(command, stderr=subprocess.STDOUT)
+    assert output == b"output\nerror\n" or output == b"error\noutput\n"
+
+
 @mock.patch("os.name", new="nt")
 def test_get_project_id_windows():
     check_output_patch = mock.patch(