Repository: boto/botocore
Pull request #1316 (2017-11-21 02:20:26)

Add support for process-based credential providers
This allows users to configure an external process to provide
credentials using the `credential_process` config variable.

Note that this is built on top of the assume role provider since
there is rebasing necessary.

codecov-io: # [Codecov](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=h1) Report
> Merging [#1316](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=desc) into [develop](https://codecov.io/gh/boto/botocore/commit/01b5f9315114b5e7676e971e18d2dbcab28d9a5b?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/boto/botocore/pull/1316/graphs/tree.svg?src=pr&width=650&token=QOLXw6XSza&height=150)](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1316      +/-   ##
===========================================
+ Coverage    98.06%   98.11%   +0.04%     
===========================================
  Files           46       46              
  Lines         7460     7644     +184     
===========================================
+ Hits          7316     7500     +184     
  Misses         144      144
```


| [Impacted Files](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [botocore/credentials.py](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=tree#diff-Ym90b2NvcmUvY3JlZGVudGlhbHMucHk=) | `98.31% <100%> (+0.58%)` | :arrow_up: |
| [botocore/exceptions.py](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=tree#diff-Ym90b2NvcmUvZXhjZXB0aW9ucy5weQ==) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=footer). Last update [01b5f93...1221ae5](https://codecov.io/gh/boto/botocore/pull/1316?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/.changes/next-release/feature-credentials-59288.json b/.changes/next-release/feature-credentials-59288.json
new file mode 100644
index 000000000..6e9bf2080
--- /dev/null
+++ b/.changes/next-release/feature-credentials-59288.json
@@ -0,0 +1,5 @@
+{
+  "category": "credentials", 
+  "type": "feature", 
+  "description": "Adds support for the process credential provider, allowing users to specify a process to call to get credentials."
+}
diff --git a/botocore/credentials.py b/botocore/credentials.py
index b53707dda..fcee6cd79 100644
--- a/botocore/credentials.py
+++ b/botocore/credentials.py
@@ -17,6 +17,8 @@ import logging
 import os
 import getpass
 import threading
+import subprocess
+import shlex
 from collections import namedtuple
 from copy import deepcopy
 from hashlib import sha256
@@ -82,6 +84,8 @@ def create_credential_resolver(session):
             creds_filename=credential_file,
             profile_name=profile_name
         ),
+        ProcessProvider(profile_name=profile_name,
+                        load_config=lambda: session.full_config),
         # The new config file has precedence over the legacy
         # config file.
         ConfigProvider(config_filename=config_file, profile_name=profile_name),
@@ -701,6 +705,76 @@ class CredentialProvider(object):
         return found
 
 
+class ProcessProvider(CredentialProvider):
+
+    METHOD = 'custom-process'
+
+    def __init__(self, profile_name, load_config, popen=subprocess.Popen):
+        self._profile_name = profile_name
+        self._load_config = load_config
+        self._loaded_config = None
+        self._popen = popen
+
+    def load(self):
+        credential_process = self._credential_process
+        if credential_process is None:
+            return
+
+        creds_dict = self._retrieve_credentials_using(credential_process)
+        if creds_dict.get('expiry_time') is not None:
+            return RefreshableCredentials.create_from_metadata(
+                creds_dict,
+                lambda: self._retrieve_credentials_using(credential_process),
+                self.METHOD
+            )
+
+        return Credentials(
+            access_key=creds_dict['access_key'],
+            secret_key=creds_dict['secret_key'],
+            token=creds_dict.get('token'),
+            method=self.METHOD
+        )
+
+    def _retrieve_credentials_using(self, credential_process):
+        # We're not using shell=True, so we need to pass the
+        # command and all arguments as a list.
+        process_list = shlex.split(credential_process)
+        p = self._popen(process_list,
+                        stdout=subprocess.PIPE,
+                        stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        if p.returncode != 0:
+            raise CredentialRetrievalError(
+                provider=self.METHOD, error_msg=stderr.decode('utf-8'))
+        parsed = botocore.compat.json.loads(stdout.decode('utf-8'))
+        version = parsed.get('Version', '<Version key not provided>')
+        if version != 1:
+            raise CredentialRetrievalError(
+                provider=self.METHOD,
+                error_msg=("Unsupported version '%s' for credential process "
+                           "provider, supported versions: 1" % version))
+        try:
+            return {
+                'access_key': parsed['AccessKeyId'],
+                'secret_key': parsed['SecretAccessKey'],
+                'token': parsed.get('SessionToken'),
+                'expiry_time': parsed.get('Expiration'),
+            }
+        except KeyError as e:
+            raise CredentialRetrievalError(
+                provider=self.METHOD,
+                error_msg="Missing required key in response: %s" % e
+            )
+
+    @property
+    def _credential_process(self):
+        if self._loaded_config is None:
+            self._loaded_config = self._load_config()
+        profile_config = self._loaded_config.get(
+            'profiles', {}).get(self._profile_name, {})
+        return profile_config.get('credential_process')
+
+
 class InstanceMetadataProvider(CredentialProvider):
     METHOD = 'iam-role'
     CANONICAL_NAME = 'Ec2InstanceMetadata'


Test patch:
diff --git a/tests/functional/test_credentials.py b/tests/functional/test_credentials.py
index 4a0d3c59e..37ea14041 100644
--- a/tests/functional/test_credentials.py
+++ b/tests/functional/test_credentials.py
@@ -18,11 +18,14 @@ import mock
 import tempfile
 import shutil
 from datetime import datetime, timedelta
+import sys
 
 from botocore.vendored import requests
 from dateutil.tz import tzlocal
+from botocore.exceptions import CredentialRetrievalError
 
 from tests import unittest, IntegerRefresher, BaseEnvVar, random_chars
+from tests import temporary_file
 from botocore.credentials import EnvProvider, ContainerProvider
 from botocore.credentials import InstanceMetadataProvider
 from botocore.credentials import Credentials, ReadOnlyCredentials
@@ -400,3 +403,65 @@ class TestAssumeRole(BaseEnvVar):
         actual_creds = session.get_credentials()
         self.assert_creds_equal(actual_creds, expected_creds)
         stubber.assert_no_pending_responses()
+
+
+class TestProcessProvider(unittest.TestCase):
+    def setUp(self):
+        current_dir = os.path.dirname(os.path.abspath(__file__))
+        credential_process = os.path.join(
+            current_dir, 'utils', 'credentialprocess.py'
+        )
+        self.credential_process = '%s %s' % (
+            sys.executable, credential_process
+        )
+        self.environ = os.environ.copy()
+        self.environ_patch = mock.patch('os.environ', self.environ)
+        self.environ_patch.start()
+
+    def tearDown(self):
+        self.environ_patch.stop()
+
+    def test_credential_process(self):
+        config = (
+            '[profile processcreds]\n'
+            'credential_process = %s\n'
+        )
+        config = config % self.credential_process
+        with temporary_file('w') as f:
+            f.write(config)
+            f.flush()
+            self.environ['AWS_CONFIG_FILE'] = f.name
+
+            credentials = Session(profile='processcreds').get_credentials()
+            self.assertEqual(credentials.access_key, 'spam')
+            self.assertEqual(credentials.secret_key, 'eggs')
+
+    def test_credential_process_returns_error(self):
+        config = (
+            '[profile processcreds]\n'
+            'credential_process = %s --raise-error\n'
+        )
+        config = config % self.credential_process
+        with temporary_file('w') as f:
+            f.write(config)
+            f.flush()
+            self.environ['AWS_CONFIG_FILE'] = f.name
+
+            session = Session(profile='processcreds')
+
+            # This regex validates that there is no substring: b'
+            # The reason why we want to validate that is that we want to
+            # make sure that stderr is actually decoded so that in
+            # exceptional cases the error is properly formatted.
+            # As for how the regex works:
+            # `(?!b').` is a negative lookahead, meaning that it will only
+            # match if it is not followed by the pattern `b'`. Since it is
+            # followed by a `.` it will match any character not followed by
+            # that pattern. `((?!hede).)*` does that zero or more times. The
+            # final pattern adds `^` and `$` to anchor the beginning and end
+            # of the string so we can know the whole string is consumed.
+            # Finally `(?s)` at the beginning makes dots match newlines so
+            # we can handle a multi-line string.
+            reg = r"(?s)^((?!b').)*$"
+            with self.assertRaisesRegexp(CredentialRetrievalError, reg):
+                session.get_credentials()
diff --git a/tests/functional/utils/__init__.py b/tests/functional/utils/__init__.py
new file mode 100644
index 000000000..53a960891
--- /dev/null
+++ b/tests/functional/utils/__init__.py
@@ -0,0 +1,12 @@
+# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
diff --git a/tests/functional/utils/credentialprocess.py b/tests/functional/utils/credentialprocess.py
new file mode 100644
index 000000000..722b1f23f
--- /dev/null
+++ b/tests/functional/utils/credentialprocess.py
@@ -0,0 +1,35 @@
+# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""This is a dummy implementation of a credential provider process."""
+import argparse
+import json
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--raise-error', action='store_true', help=(
+        'If set, this will cause the process to return a non-zero exit code '
+        'and print to stderr.'
+    ))
+    args = parser.parse_args()
+    if args.raise_error:
+        raise Exception('Failed to fetch credentials.')
+    print(json.dumps({
+        'AccessKeyId': 'spam',
+        'SecretAccessKey': 'eggs',
+        'Version': 1
+    }))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/unit/test_credentials.py b/tests/unit/test_credentials.py
index c8951d6f3..e81c7d22a 100644
--- a/tests/unit/test_credentials.py
+++ b/tests/unit/test_credentials.py
@@ -12,6 +12,7 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 from datetime import datetime, timedelta
+import subprocess
 import mock
 import os
 import json
@@ -21,6 +22,7 @@ from dateutil.tz import tzlocal, tzutc
 
 from botocore import credentials
 from botocore.utils import ContainerMetadataFetcher
+from botocore.compat import json
 from botocore.credentials import EnvProvider, create_assume_role_refresher
 from botocore.credentials import CredentialProvider, AssumeRoleProvider
 from botocore.credentials import ConfigProvider, SharedCredentialProvider
@@ -2397,3 +2399,258 @@ class TestContainerProvider(BaseEnvVar):
         self.assertEqual(creds.secret_key, 'secret_key')
         self.assertEqual(creds.token, 'token')
         self.assertEqual(creds.method, 'container-role')
+
+
+class TestProcessProvider(BaseEnvVar):
+    def setUp(self):
+        super(TestProcessProvider, self).setUp()
+        self.loaded_config = {}
+        self.load_config = mock.Mock(return_value=self.loaded_config)
+        self.invoked_process = mock.Mock()
+        self.popen_mock = mock.Mock(return_value=self.invoked_process,
+                                    spec=subprocess.Popen)
+
+    def create_process_provider(self, profile_name='default'):
+        provider = credentials.ProcessProvider(profile_name, self.load_config,
+                                               popen=self.popen_mock)
+        return provider
+
+    def _get_output(self, stdout, stderr=''):
+        return json.dumps(stdout).encode('utf-8'), stderr.encode('utf-8')
+
+    def _set_process_return_value(self, stdout, stderr='', rc=0):
+        output = self._get_output(stdout, stderr)
+        self.invoked_process.communicate.return_value = output
+        self.invoked_process.returncode = rc
+
+    def test_process_not_invoked_if_profile_does_not_exist(self):
+        # self.loaded_config is an empty dictionary with no profile
+        # information.
+        provider = self.create_process_provider()
+        self.assertIsNone(provider.load())
+
+    def test_process_not_invoked_if_not_configured_for_empty_config(self):
+        # No credential_process configured so we skip this provider.
+        self.loaded_config['profiles'] = {'default': {}}
+        provider = self.create_process_provider()
+        self.assertIsNone(provider.load())
+
+    def test_can_retrieve_via_process(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.assertEqual(creds.access_key, 'foo')
+        self.assertEqual(creds.secret_key, 'bar')
+        self.assertEqual(creds.token, 'baz')
+        self.assertEqual(creds.method, 'custom-process')
+        self.popen_mock.assert_called_with(
+            ['my-process'],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE
+        )
+
+    def test_can_pass_arguments_through(self):
+        self.loaded_config['profiles'] = {
+            'default': {
+                'credential_process': 'my-process --foo --bar "one two"'
+            }
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.popen_mock.assert_called_with(
+            ['my-process', '--foo', '--bar', 'one two'],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE
+        )
+
+    def test_can_refresh_credentials(self):
+        # We given a time that's already expired so .access_key
+        # will trigger the refresh worfklow.  We just need to verify
+        # that the refresh function gives the same result as the
+        # initial retrieval.
+        expired_date = '2016-01-01T00:00:00Z'
+        future_date = str(datetime.now(tzlocal()) + timedelta(hours=24))
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        old_creds = self._get_output({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': expired_date,
+        })
+        new_creds = self._get_output({
+            'Version': 1,
+            'AccessKeyId': 'foo2',
+            'SecretAccessKey': 'bar2',
+            'SessionToken': 'baz2',
+            'Expiration': future_date,
+        })
+        self.invoked_process.communicate.side_effect = [old_creds, new_creds]
+        self.invoked_process.returncode = 0
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.assertEqual(creds.access_key, 'foo2')
+        self.assertEqual(creds.secret_key, 'bar2')
+        self.assertEqual(creds.token, 'baz2')
+        self.assertEqual(creds.method, 'custom-process')
+
+    def test_non_zero_rc_raises_exception(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value('', 'Error Message', 1)
+
+        provider = self.create_process_provider()
+        exception = botocore.exceptions.CredentialRetrievalError
+        with self.assertRaisesRegexp(exception, 'Error Message'):
+            provider.load()
+
+    def test_unsupported_version_raises_mismatch(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        bad_version = 100
+        self._set_process_return_value({
+            'Version': bad_version,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        exception = botocore.exceptions.CredentialRetrievalError
+        with self.assertRaisesRegexp(exception, 'Unsupported version'):
+            provider.load()
+
+    def test_missing_version_in_payload_returned_raises_exception(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            # Let's say they forget a 'Version' key.
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        exception = botocore.exceptions.CredentialRetrievalError
+        with self.assertRaisesRegexp(exception, 'Unsupported version'):
+            provider.load()
+
+    def test_missing_access_key_raises_exception(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            # Missing access key.
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        exception = botocore.exceptions.CredentialRetrievalError
+        with self.assertRaisesRegexp(exception, 'Missing required key'):
+            provider.load()
+
+    def test_missing_secret_key_raises_exception(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            # Missing secret key.
+            'SessionToken': 'baz',
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        exception = botocore.exceptions.CredentialRetrievalError
+        with self.assertRaisesRegexp(exception, 'Missing required key'):
+            provider.load()
+
+    def test_missing_session_token(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            # Missing session token.
+            'Expiration': '2020-01-01T00:00:00Z',
+        })
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.assertEqual(creds.access_key, 'foo')
+        self.assertEqual(creds.secret_key, 'bar')
+        self.assertIsNone(creds.token)
+        self.assertEqual(creds.method, 'custom-process')
+
+    def test_missing_expiration(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            'SessionToken': 'baz',
+            # Missing expiration.
+        })
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.assertEqual(creds.access_key, 'foo')
+        self.assertEqual(creds.secret_key, 'bar')
+        self.assertEqual(creds.token, 'baz')
+        self.assertEqual(creds.method, 'custom-process')
+
+    def test_missing_expiration_and_session_token(self):
+        self.loaded_config['profiles'] = {
+            'default': {'credential_process': 'my-process'}
+        }
+        self._set_process_return_value({
+            'Version': 1,
+            'AccessKeyId': 'foo',
+            'SecretAccessKey': 'bar',
+            # Missing session token and expiration
+        })
+
+        provider = self.create_process_provider()
+        creds = provider.load()
+        self.assertIsNotNone(creds)
+        self.assertEqual(creds.access_key, 'foo')
+        self.assertEqual(creds.secret_key, 'bar')
+        self.assertIsNone(creds.token)
+        self.assertEqual(creds.method, 'custom-process')