Repository: aws/aws-cli

Issue:
Proposal: Integrate better with the shared credentials file
Version 1.3.13 of the CLI added support for loading credentials from the `~/.aws/credentials` file.  With this change, if credentials are found in the `~/.aws/credentials` file, those credentials will be used over the credentials in the `~/.aws/config` file.

The benefit of using the `~/.aws/credentials` file is two fold:
1. It’s a file that only contains credentials (i.e secrets).  This makes a clean separation between credentials and configuration data.
2. _All_ of the AWS SDKs support the `~/.aws/credentials` file, whereas the `~/.aws/config` is specific to the AWS CLI.

So by using the `~/.aws/credentials` file, you won’t have to worry about configuration credentials for any other AWS SDKs, which makes interoperability nice.

However, none of the `aws configure` commands (including `aws configure get/set`) were updated.  This means that all of our control plane commands for dealing with configuration operated solely on the `~/.aws/config` file.  This is simple and easy to understand.

This issue is broadly about what we can do with our `aws configure` suite of commands to integrate better with the `~/.aws/credentials` file.  On one of the spectrum, we can completely switch over to `~/.aws/credentials` and not allow credentials to be written into the `~/.aws/config` via the `configure` commands (though we’d still support loading credentials from `~/.aws/config` for backwards compatibility).  On the other end we can leave things the way they are and users need to manually create/edit the `~/.aws/credentials` file.  Below are a few proposals.  Feedback would be much appreciated.
### Option 1: Do Nothing

Simple, don’t make any changes.  Keep everything as it is now.
### Option 2: Add options to use shared credentials

Add a `--use-shared-credentials/--no-use-shared-credentials` option that allows the user to explicitly specify which file they want to use. So for example:

```
aws configure —use-shared-credentials
AWS Access Key Id []: k1           <—— written to ~/.aws/credentials
AWS Secret Access Key []: k2       <—- written to ~/.aws/credentials
Default region name []: us-west-2  <—- written to ~/.aws/config
Default output format: json        <—- written to ~/.aws/config
```

```
aws configure --no-use-shared-credentials

AWS Access Key Id []: k1           <—— written to ~/.aws/config
AWS Secret Access Key []: k2       <—- written to ~/.aws/config
Default region name []: us-west-2  <—- written to ~/.aws/config
Default output format: json        <—- written to ~/.aws/config 
```

There’s also the question of what to do about the default.  Having a default of `--no-use-shared-credentials` would make using the shared credentials file completely opt in.  Defaulting to `--use-shared-credentials` would be a change in behavior and would mean you’d need to opt _out_ of using the shared credentials file.
### Option 3: Automatically Switch Over

This option would always use used the shared credentials file whenever possible for access/secret/token vars.  This is basically `—use-shared-credentials`, but always enabled.  With this option the user doesn’t have to do anything, we’ll just automatically always write out access/secret/session key information to `~/.aws/credentials`.  If a user wants credentials in `~/.aws/config` they will need to manually edit their config file.

It would be really helpful to get feedback about which (if any) of these options people prefer.



Discussion:
Option 3 :+1:. Using a single set of credentials between the CLI and my favorite SDK is a no-brainer.

+1 for option 3.

3++

Makes for a small risk of confusion short term, but that's better than long term inconsistency and confusion, which is what we've lived with for too many years with AWS command line tools.

Another :thumbsup: for option 3.

Just want to make sure I understand.

Option 3 would automatically move credentials to ~/.aws/credentials but there would still be a ~/.aws/config file and it would still contain any non-credential attributes defined in ~/.aws/config, right?    And those non-credentail attributes would still be accessible if I did a get_scoped_config() call? Would the credential also appear in the dict returned by a get_scoped_config() call?

+1 option 3

@garnaat Yes that's correct.  Ideally the change is only to the `aws configure` commands so that anytime you access credential variables it will use the `~/.aws/credentials` file.  Botocore should remain unchanged.

So it sounds like option 3 is the option we should go with.  To flesh this out a bit more:
- The list of variables considered to be credentials are: aws_access_key_id/aws_secret_access_key/aws_session_token
- Anytime you **set**  credential variables (either interactively via `aws configure` or via `aws configure set`, any credentials variables will _always_ be written to `~/.aws/credentials`).  We will no longer write credential variables to `~/.aws/config`.
- Anytime you **get** credential variables (either if you're rerunning `aws configure` where in the prompt we show you the current value or if you run the `aws configure get` command), we will first look in the `~/.aws/credentials` for credentials before then falling back to the `~/.aws/config` file.

Thanks everyone for the feedback, it was really helpful.


Resolving patch:
diff --git a/awscli/compat.py b/awscli/compat.py
--- a/awscli/compat.py
+++ b/awscli/compat.py
@@ -17,6 +17,8 @@
     import locale
     import urllib.parse as urlparse
 
+    raw_input = input
+
     def get_stdout_text_writer():
         return sys.stdout
 
@@ -41,6 +43,8 @@ def compat_open(filename, mode='r', encoding=None):
     import io
     import urlparse
 
+    raw_input = raw_input
+
     def get_stdout_text_writer():
         # In python3, all the sys.stdout/sys.stderr streams are in text
         # mode.  This means they expect unicode, and will encode the
diff --git a/awscli/customizations/configure.py b/awscli/customizations/configure.py
--- a/awscli/customizations/configure.py
+++ b/awscli/customizations/configure.py
@@ -18,12 +18,7 @@
 from botocore.exceptions import ProfileNotFound
 
 from awscli.customizations.commands import BasicCommand
-
-
-try:
-    raw_input = raw_input
-except NameError:
-    raw_input = input
+from awscli.compat import raw_input
 
 
 logger = logging.getLogger(__name__)
@@ -55,7 +50,7 @@ def _mask_value(current_value):
     if current_value is None:
         return 'None'
     else:
-        return ('*' * 16) +  current_value[-4:]
+        return ('*' * 16) + current_value[-4:]
 
 
 class InteractivePrompter(object):
@@ -80,6 +75,32 @@ class ConfigFileWriter(object):
     )
 
     def update_config(self, new_values, config_filename):
+        """Update config file with new values.
+
+        This method will update a section in a config file with
+        new key value pairs.
+
+        This method provides a few conveniences:
+
+        * If the ``config_filename`` does not exist, it will
+          be created.  Any parent directories will also be created
+          if necessary.
+        * If the section to update does not exist, it will be created.
+        * Any existing lines that specified by ``new_values``
+          **will not be touched**.  This ensures that commented out
+          values are left unaltered.
+
+        :type new_values: dict
+        :param new_values: The values to update.  There is a special
+            key ``__section__``, that specifies what section in the INI
+            file to update.  If this key is not present, then the
+            ``default`` section will be updated with the new values.
+
+        :type config_filename: str
+        :param config_filename: The config filename where values will be
+            written.
+
+        """
         section_name = new_values.pop('__section__', 'default')
         if not os.path.isfile(config_filename):
             self._create_file(config_filename)
@@ -98,11 +119,11 @@ def update_config(self, new_values, config_filename):
 
     def _create_file(self, config_filename):
         # Create the file as well as the parent dir if needed.
-        dirname, basename = os.path.split(config_filename)
+        dirname = os.path.split(config_filename)[0]
         if not os.path.isdir(dirname):
             os.makedirs(dirname)
         with os.fdopen(os.open(config_filename,
-                               os.O_WRONLY|os.O_CREAT, 0o600), 'w'):
+                               os.O_WRONLY | os.O_CREAT, 0o600), 'w'):
             pass
 
     def _write_new_section(self, section_name, new_values, config_filename):
@@ -124,15 +145,15 @@ def _find_section_start(self, contents, section_name):
             if match is not None and self._matches_section(match,
                                                            section_name):
                 return i
-        else:
-            raise SectionNotFoundError(section_name)
+        raise SectionNotFoundError(section_name)
 
     def _update_section_contents(self, contents, section_name, new_values):
         # First, find the line where the section_name is defined.
         # This will be the value of i.
         new_values = new_values.copy()
         # ``contents`` is a list of file line contents.
-        section_start_line_num = self._find_section_start(contents, section_name)
+        section_start_line_num = self._find_section_start(contents,
+                                                          section_name)
         # If we get here, then we've found the section.  We now need
         # to figure out if we're updating a value or adding a new value.
         # There's 2 cases.  Either we're setting a normal scalar value
@@ -182,7 +203,8 @@ def _update_subattributes(self, index, contents, values, starting_indent):
             line = contents[i]
             match = self.OPTION_REGEX.search(line)
             if match is not None:
-                current_indent = len(match.group(1)) - len(match.group(1).lstrip())
+                current_indent = len(
+                    match.group(1)) - len(match.group(1).lstrip())
                 key_name = match.group(1).strip()
                 if key_name in values:
                     option_value = values[key_name]
@@ -205,7 +227,8 @@ def _insert_new_values(self, line_number, contents, new_values, indent=''):
                 subindent = indent + '    '
                 new_contents.append('%s%s =\n' % (indent, key))
                 for subkey, subval in list(value.items()):
-                    new_contents.append('%s%s = %s\n' % (subindent, subkey, subval))
+                    new_contents.append('%s%s = %s\n' % (subindent, subkey,
+                                                         subval))
             else:
                 new_contents.append('%s%s = %s\n' % (indent, key, value))
             del new_values[key]
@@ -302,9 +325,9 @@ def _lookup_credentials(self):
                 # the credentials.method is sufficient to show
                 # where the credentials are coming from.
                 access_key = ConfigValue(credentials.access_key,
-                                        credentials.method, '')
+                                         credentials.method, '')
                 secret_key = ConfigValue(credentials.secret_key,
-                                        credentials.method, '')
+                                         credentials.method, '')
                 access_key.mask_value()
                 secret_key.mask_value()
                 return access_key, secret_key
@@ -322,6 +345,7 @@ def _lookup_config(self, name):
         else:
             return ConfigValue(NOT_SET, None, None)
 
+
 class ConfigureSetCommand(BasicCommand):
     NAME = 'set'
     DESCRIPTION = BasicCommand.FROM_FILE('configure', 'set',
@@ -338,6 +362,8 @@ class ConfigureSetCommand(BasicCommand):
          'action': 'store',
          'cli_type_name': 'string', 'positional_arg': True},
     ]
+    _WRITE_TO_CREDS_FILE = ['aws_access_key_id', 'aws_secret_access_key',
+                            'aws_session_token']
 
     def __init__(self, session, config_writer=None):
         super(ConfigureSetCommand, self).__init__(session)
@@ -362,7 +388,6 @@ def _run_main(self, args, parsed_globals):
                 section = 'profile %s' % self._session.profile
         else:
             # First figure out if it's been scoped to a profile.
-            # This will happen if 
             parts = varname.split('.')
             if parts[0] in ('default', 'profile'):
                 # Then we know we're scoped to a profile.
@@ -383,6 +408,12 @@ def _run_main(self, args, parsed_globals):
         config_filename = os.path.expanduser(
             self._session.get_config_variable('config_file'))
         updated_config = {'__section__': section, varname: value}
+        if varname in self._WRITE_TO_CREDS_FILE:
+            config_filename = os.path.expanduser(
+                self._session.get_config_variable('credentials_file'))
+            section_name = updated_config['__section__']
+            if section_name.startswith('profile '):
+                updated_config['__section__'] = section_name[8:]
         self._config_writer.update_config(updated_config, config_filename)
 
 
@@ -456,7 +487,6 @@ def _get_dotted_config_value(self, varname):
         return value
 
 
-
 class ConfigureCommand(BasicCommand):
     NAME = 'configure'
     DESCRIPTION = BasicCommand.FROM_FILE()
@@ -522,7 +552,30 @@ def _run_main(self, parsed_args, parsed_globals):
         config_filename = os.path.expanduser(
             self._session.get_config_variable('config_file'))
         if new_values:
+            self._write_out_creds_file_values(new_values,
+                                              parsed_globals.profile)
             if parsed_globals.profile is not None:
                 new_values['__section__'] = (
                     'profile %s' % parsed_globals.profile)
             self._config_writer.update_config(new_values, config_filename)
+
+    def _write_out_creds_file_values(self, new_values, profile_name):
+        # The access_key/secret_key are now *always* written to the shared
+        # credentials file (~/.aws/credentials), see aws/aws-cli#847.
+        # post-conditions: ~/.aws/credentials will have the updated credential
+        # file values and new_values will have the cred vars removed.
+        credential_file_values = {}
+        if 'aws_access_key_id' in new_values:
+            credential_file_values['aws_access_key_id'] = new_values.pop(
+                'aws_access_key_id')
+        if 'aws_secret_access_key' in new_values:
+            credential_file_values['aws_secret_access_key'] = new_values.pop(
+                'aws_secret_access_key')
+        if credential_file_values:
+            if profile_name is not None:
+                credential_file_values['__section__'] = profile_name
+            shared_credentials_filename = self._session.get_config_variable(
+                'credentials_file')
+            self._config_writer.update_config(
+                credential_file_values,
+                shared_credentials_filename)


Test patch:
diff --git a/tests/unit/customizations/test_configure.py b/tests/unit/customizations/test_configure.py
--- a/tests/unit/customizations/test_configure.py
+++ b/tests/unit/customizations/test_configure.py
@@ -70,6 +70,10 @@ def get_scoped_config(self):
         return self.config
 
     def get_config_variable(self, name, methods=None):
+        if name == 'credentials_file':
+            # The credentials_file var doesn't require a
+            # profile to exist.
+            return 'fake_credentials_file'
         if self.profile_does_not_exist and not name == 'config_file':
             raise ProfileNotFound(profile='foo')
         if methods is not None:
@@ -98,12 +102,22 @@ def setUp(self):
                                                     prompter=self.precanned,
                                                     config_writer=self.writer)
 
+    def assert_credentials_file_updated_with(self, new_values):
+        called_args = self.writer.update_config.call_args_list
+        credentials_file_call = called_args[0]
+        self.assertEqual(credentials_file_call,
+                         mock.call(new_values, 'fake_credentials_file'))
+
     def test_configure_command_sends_values_to_writer(self):
         self.configure(args=[], parsed_globals=self.global_args)
-        self.writer.update_config.assert_called_with(
+        # Credentials are always written to the shared credentials file.
+        self.assert_credentials_file_updated_with(
             {'aws_access_key_id': 'new_value',
-             'aws_secret_access_key': 'new_value',
-             'region': 'new_value',
+             'aws_secret_access_key': 'new_value'})
+
+        # Non-credentials config is written to the config file.
+        self.writer.update_config.assert_called_with(
+            {'region': 'new_value',
              'output': 'new_value'}, 'myconfigfile')
 
     def test_same_values_are_not_changed(self):
@@ -154,10 +168,12 @@ def test_section_name_can_be_changed_for_profiles(self):
         self.global_args.profile = 'myname'
         self.configure(args=[], parsed_globals=self.global_args)
         # Note the __section__ key name.
+        self.assert_credentials_file_updated_with(
+            {'aws_access_key_id': 'new_value',
+             'aws_secret_access_key': 'new_value',
+             '__section__': 'myname'})
         self.writer.update_config.assert_called_with(
             {'__section__': 'profile myname',
-             'aws_access_key_id': 'new_value',
-             'aws_secret_access_key': 'new_value',
              'region': 'new_value',
              'output': 'new_value'}, 'myconfigfile')
 
@@ -173,18 +189,26 @@ def test_session_says_profile_does_not_exist(self):
                                                     config_writer=self.writer)
         self.global_args.profile = 'profile-does-not-exist'
         self.configure(args=[], parsed_globals=self.global_args)
+        self.assert_credentials_file_updated_with(
+            {'aws_access_key_id': 'new_value',
+             'aws_secret_access_key': 'new_value',
+             '__section__': 'profile-does-not-exist'})
         self.writer.update_config.assert_called_with(
             {'__section__': 'profile profile-does-not-exist',
-             'aws_access_key_id': 'new_value',
-             'aws_secret_access_key': 'new_value',
              'region': 'new_value',
              'output': 'new_value'}, 'myconfigfile')
 
 
 class TestInteractivePrompter(unittest.TestCase):
-    @mock.patch('awscli.customizations.configure.raw_input')
-    def test_access_key_is_masked(self, mock_raw_input):
-        mock_raw_input.return_value = 'foo'
+    def setUp(self):
+        self.patch = mock.patch('awscli.customizations.configure.raw_input')
+        self.mock_raw_input = self.patch.start()
+
+    def tearDown(self):
+        self.patch.stop()
+
+    def test_access_key_is_masked(self):
+        self.mock_raw_input.return_value = 'foo'
         prompter = configure.InteractivePrompter()
         response = prompter.get_value(
             current_value='myaccesskey', config_name='aws_access_key_id',
@@ -192,45 +216,54 @@ def test_access_key_is_masked(self, mock_raw_input):
         # First we should return the value from raw_input.
         self.assertEqual(response, 'foo')
         # We should also not display the entire access key.
-        prompt_text = mock_raw_input.call_args[0][0]
+        prompt_text = self.mock_raw_input.call_args[0][0]
         self.assertNotIn('myaccesskey', prompt_text)
         self.assertRegexpMatches(prompt_text, r'\[\*\*\*\*.*\]')
 
-    @mock.patch('awscli.customizations.configure.raw_input')
-    def test_access_key_not_masked_when_none(self, mock_raw_input):
-        mock_raw_input.return_value = 'foo'
+    def test_access_key_not_masked_when_none(self):
+        self.mock_raw_input.return_value = 'foo'
         prompter = configure.InteractivePrompter()
         response = prompter.get_value(
             current_value=None, config_name='aws_access_key_id',
             prompt_text='Access key')
         # First we should return the value from raw_input.
         self.assertEqual(response, 'foo')
-        prompt_text = mock_raw_input.call_args[0][0]
+        prompt_text = self.mock_raw_input.call_args[0][0]
         self.assertIn('[None]', prompt_text)
 
-    @mock.patch('awscli.customizations.configure.raw_input')
-    def test_secret_key_is_masked(self, mock_raw_input):
+    def test_secret_key_is_masked(self):
         prompter = configure.InteractivePrompter()
         prompter.get_value(
             current_value='mysupersecretkey',
             config_name='aws_secret_access_key',
             prompt_text='Secret Key')
         # We should also not display the entire secret key.
-        prompt_text = mock_raw_input.call_args[0][0]
+        prompt_text = self.mock_raw_input.call_args[0][0]
         self.assertNotIn('mysupersecretkey', prompt_text)
         self.assertRegexpMatches(prompt_text, r'\[\*\*\*\*.*\]')
 
-    @mock.patch('awscli.customizations.configure.raw_input')
-    def test_non_secret_keys_are_not_masked(self, mock_raw_input):
+    def test_non_secret_keys_are_not_masked(self):
         prompter = configure.InteractivePrompter()
         prompter.get_value(
             current_value='mycurrentvalue', config_name='not_a_secret_key',
             prompt_text='Enter value')
         # We should also not display the entire secret key.
-        prompt_text = mock_raw_input.call_args[0][0]
+        prompt_text = self.mock_raw_input.call_args[0][0]
         self.assertIn('mycurrentvalue', prompt_text)
         self.assertRegexpMatches(prompt_text, r'\[mycurrentvalue\]')
 
+    def test_user_hits_enter_returns_none(self):
+        # If a user hits enter, then raw_input returns the empty string.
+        self.mock_raw_input.return_value = ''
+
+        prompter = configure.InteractivePrompter()
+        response = prompter.get_value(
+            current_value=None, config_name='aws_access_key_id',
+            prompt_text='Access key')
+        # We convert the empty string to None to indicate that there
+        # was no input.
+        self.assertIsNone(response)
+
 
 class TestConfigFileWriter(unittest.TestCase):
     def setUp(self):
@@ -395,6 +428,22 @@ def test_update_config_with_comments(self):
             'foo = newvalue\n'
         )
 
+    def test_update_config_with_commented_section(self):
+        original = (
+            '#[default]\n'
+            '[default]\n'
+            '#foo = 1\n'
+            'bar = 1\n'
+        )
+        self.assert_update_config(
+            original, {'foo': 'newvalue'},
+            '#[default]\n'
+            '[default]\n'
+            '#foo = 1\n'
+            'bar = 1\n'
+            'foo = newvalue\n'
+        )
+
     def test_spaces_around_key_names(self):
         original = (
             '[default]\n'
@@ -720,7 +769,7 @@ def test_configure_set_triple_dotted(self):
             {'__section__': 'default', 's3': {'signature_version': 's3v4'}},
              'myconfigfile')
 
-    def test_configure_set_with_profile(self):
+    def test_configure_set_with_profile_nested(self):
         # aws configure set default.s3.signature_version s3v4
         set_command = configure.ConfigureSetCommand(self.session, self.config_writer)
         set_command(args=['profile.foo.s3.signature_version', 's3v4'],
@@ -728,3 +777,49 @@ def test_configure_set_with_profile(self):
         self.config_writer.update_config.assert_called_with(
             {'__section__': 'profile foo',
              's3': {'signature_version': 's3v4'}}, 'myconfigfile')
+
+    def test_access_key_written_to_shared_credentials_file(self):
+        set_command = configure.ConfigureSetCommand(self.session, self.config_writer)
+        set_command(args=['aws_access_key_id', 'foo'],
+                    parsed_globals=None)
+        self.config_writer.update_config.assert_called_with(
+            {'__section__': 'default',
+             'aws_access_key_id': 'foo'}, 'fake_credentials_file')
+
+    def test_secret_key_written_to_shared_credentials_file(self):
+        set_command = configure.ConfigureSetCommand(self.session, self.config_writer)
+        set_command(args=['aws_secret_access_key', 'foo'],
+                    parsed_globals=None)
+        self.config_writer.update_config.assert_called_with(
+            {'__section__': 'default',
+             'aws_secret_access_key': 'foo'}, 'fake_credentials_file')
+
+    def test_session_token_written_to_shared_credentials_file(self):
+        set_command = configure.ConfigureSetCommand(self.session, self.config_writer)
+        set_command(args=['aws_session_token', 'foo'],
+                    parsed_globals=None)
+        self.config_writer.update_config.assert_called_with(
+            {'__section__': 'default',
+             'aws_session_token': 'foo'}, 'fake_credentials_file')
+
+    def test_access_key_written_to_shared_credentials_file_profile(self):
+        set_command = configure.ConfigureSetCommand(self.session, self.config_writer)
+        set_command(args=['profile.foo.aws_access_key_id', 'bar'],
+                    parsed_globals=None)
+        self.config_writer.update_config.assert_called_with(
+            {'__section__': 'foo',
+             'aws_access_key_id': 'bar'}, 'fake_credentials_file')
+
+class TestConfigValueMasking(unittest.TestCase):
+    def test_config_value_is_masked(self):
+        config_value = configure.ConfigValue(
+            'fake_access_key', 'config_file', 'aws_access_key_id')
+        self.assertEqual(config_value.value, 'fake_access_key')
+        config_value.mask_value()
+        self.assertEqual(config_value.value, '****************_key')
+
+    def test_dont_mask_unset_value(self):
+        no_config = configure.ConfigValue(configure.NOT_SET, None, None)
+        self.assertEqual(no_config.value, configure.NOT_SET)
+        no_config.mask_value()
+        self.assertEqual(no_config.value, configure.NOT_SET)