Repository: boto/boto

Issue:
TypeError populating keys from metadata server
In my server logs I'm seeing what looks like a rerun of #1907 with boto 2.38.0.

```
    conn = boto.connect_s3()
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/__init__.py", line 141, in connect_s3
    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/s3/connection.py", line 190, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/connection.py", line 555, in __init__
    profile_name)
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/provider.py", line 200, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/provider.py", line 376, in get_credentials
    self._populate_keys_from_metadata_server()
  File "/srv/encoded/eggs/boto-2.38.0-py3.4.egg/boto/provider.py", line 395, in _populate_keys_from_metadata_server
    self._access_key = security['AccessKeyId']
TypeError: string indices must be integers
```



Discussion:
Tried to repro using 2.39.0.  Also tried on python3.4.  Not seeing this issue.  IIRC #1907 was a regression that immediately broke the instance metadata credential retriever.   It doesn't look like this is the same case.

Some additional data that would help:
- How frequently are you seeing this issue
- Is this code running immediately when the instance starts up?

Seems to be happening only intermittently. On this instance which has been up for 10 days I see the same error logged at these times:

```
[Tue Feb 02 19:53:59.009887 2016]
[Wed Feb 03 19:05:34.216450 2016]
[Thu Feb 04 14:07:11.609744 2016]
[Thu Feb 04 14:07:11.609974 2016]
[Thu Feb 04 14:07:19.278424 2016]
[Thu Feb 04 18:03:41.861965 2016]
[Thu Feb 04 18:03:47.638778 2016]
[Thu Feb 04 18:03:52.541109 2016]
[Thu Feb 04 18:03:55.567427 2016]
[Thu Feb 04 18:04:24.094672 2016]
[Thu Feb 04 18:04:33.907620 2016]
[Thu Feb 04 18:04:41.790302 2016]
[Thu Feb 04 18:04:48.464800 2016]
[Thu Feb 04 18:05:02.140420 2016]
[Thu Feb 04 18:05:06.421956 2016]
[Thu Feb 04 18:05:07.461871 2016]
[Thu Feb 04 18:05:10.061933 2016]
[Thu Feb 04 18:06:19.965862 2016]
```

Give the value is a string, it looks to me that this branch of LazyLoadMetadata is getting hit: https://github.com/boto/boto/blob/2.39.0/boto/utils.py#L289

``` python
                    val = boto.utils.retry_url(
                        self._url + urllib.parse.quote(resource,
                                                       safe="/:"),
                        num_retries=self._num_retries,
                        timeout=self._timeout)
                    if val and val[0] == '{':
                        val = json.loads(val)
                        break
                    else:   # Hits this branch so val is not decoded as a json dict
                        p = val.find('\n')
                        if p > 0:
                            val = val.split('\n')  # Does not hit this branch as val is a string not a list
                        break
```

That certainly seems to be the case.  That would suggest that when we try to retrieve credentials from the instance metadata service, we're occasionally not actually getting back what we expect.

I'm thinking the best course of action would be to add retries to the `_populate_keys_from_metadata_server` if we don't get back a dictionary type we expect.

Looking into the code more, I'm really curious what that string actually is.  I wonder if it's an empty string.  There's a couple places in `retry_url` where it will return an empty string on error:
- https://github.com/boto/boto/blob/develop/boto/utils.py#L221
- https://github.com/boto/boto/blob/develop/boto/utils.py#L230

So what may be happening:
- retry_url is encountering an error.  After enough retries it will return an empty string.
- This is triggering the `else` clause you linked above.
- As a result, the `provider.py` module will try to access the return value as if it were a dictionary, resulting in the `TypeError`.

Are you setting metadata [retry/num_attempts](http://boto.readthedocs.org/en/latest/boto_config_tut.html)?  If not, that can help with this issue.

I think how this is fixed depends on two cases:
1. If this is failing because we're catching an exception and just returning an empty string, then we need to raise a much more clear error message, with steps on how to increase retry attempts.
2. If we're actually getting a 200 OK response from instance metadata that contains text that is not JSON, then we need to count that as a failed attempt and retry again.

I think we have to fix the first case before knowing whether or not we need to fix the second case.  I don't really see a good way to disambiguate the two cases in the code right now.

So for now, I'd suggest just fixing (1).  Seem reasonable? cc @kyleknap  @rayluo @JordonPhillips 


Resolving patch:
diff --git a/boto/exception.py b/boto/exception.py
--- a/boto/exception.py
+++ b/boto/exception.py
@@ -571,3 +571,15 @@ def __repr__(self):
             self.message,
             self.response
         )
+
+
+class InvalidInstanceMetadataError(Exception):
+    MSG = (
+        "You can set the 'metadata_service_num_attempts' "
+        "in your boto config file to increase the number "
+        "of times boto will attempt to retrieve "
+        "credentials from the instance metadata service."
+    )
+    def __init__(self, msg):
+        final_msg = msg + '\n' + self.MSG
+        super(InvalidInstanceMetadataError, self).__init__(final_msg)
diff --git a/boto/provider.py b/boto/provider.py
--- a/boto/provider.py
+++ b/boto/provider.py
@@ -34,6 +34,7 @@
 from boto import config
 from boto.compat import expanduser
 from boto.pyami.config import Config
+from boto.exception import InvalidInstanceMetadataError
 from boto.gs.acl import ACL
 from boto.gs.acl import CannedACLStrings as CannedGSACLStrings
 from boto.s3.acl import CannedACLStrings as CannedS3ACLStrings
@@ -390,16 +391,47 @@ def _populate_keys_from_metadata_server(self):
             timeout=timeout, num_retries=attempts,
             data='meta-data/iam/security-credentials/')
         if metadata:
+            creds = self._get_credentials_from_metadata(metadata)
+            self._access_key = creds[0]
+            self._secret_key = creds[1]
+            self._security_token = creds[2]
+            expires_at = creds[3]
             # I'm assuming there's only one role on the instance profile.
-            security = list(metadata.values())[0]
-            self._access_key = security['AccessKeyId']
-            self._secret_key = self._convert_key_to_str(security['SecretAccessKey'])
-            self._security_token = security['Token']
-            expires_at = security['Expiration']
             self._credential_expiry_time = datetime.strptime(
                 expires_at, "%Y-%m-%dT%H:%M:%SZ")
             boto.log.debug("Retrieved credentials will expire in %s at: %s",
-                           self._credential_expiry_time - datetime.now(), expires_at)
+                           self._credential_expiry_time - datetime.now(),
+                           expires_at)
+
+    def _get_credentials_from_metadata(self, metadata):
+        # Given metadata, return a tuple of (access, secret, token, expiration)
+        # On errors, an InvalidInstanceMetadataError will be raised.
+        # The "metadata" is a lazy loaded dictionary means that it's possible
+        # to still encounter errors as we traverse through the metadata dict.
+        # We try to be careful and raise helpful error messages when this
+        # happens.
+        creds = list(metadata.values())[0]
+        if not isinstance(creds, dict):
+            # We want to special case a specific error condition which is
+            # where get_instance_metadata() returns an empty string on
+            # error conditions.
+            if creds == '':
+                msg = 'an empty string'
+            else:
+                msg = 'type: %s' % creds
+            raise InvalidInstanceMetadataError("Expected a dict type of "
+                                               "credentials instead received "
+                                               "%s" % (msg))
+        try:
+            access_key = creds['AccessKeyId']
+            secret_key = self._convert_key_to_str(creds['SecretAccessKey'])
+            security_token = creds['Token']
+            expires_at = creds['Expiration']
+        except KeyError as e:
+            raise InvalidInstanceMetadataError(
+                "Credentials from instance metadata missing "
+                "required key: %s" % e)
+        return access_key, secret_key, security_token, expires_at
 
     def _convert_key_to_str(self, key):
         if isinstance(key, six.text_type):
diff --git a/boto/utils.py b/boto/utils.py
--- a/boto/utils.py
+++ b/boto/utils.py
@@ -220,10 +220,10 @@ def retry_url(url, retry_on_404=True, num_retries=10, timeout=None):
             if code == 404 and not retry_on_404:
                 return ''
         except Exception as e:
-            pass
-        boto.log.exception('Caught exception reading instance data')
+            boto.log.exception('Caught exception reading instance data')
         # If not on the last iteration of the loop then sleep.
         if i + 1 != num_retries:
+            boto.log.debug('Sleeping before retrying')
             time.sleep(min(2 ** i,
                            boto.config.get('Boto', 'max_retry_delay', 60)))
     boto.log.error('Unable to read instance data, giving up')
@@ -393,6 +393,8 @@ def get_instance_metadata(version='latest', url='http://169.254.169.254',
         metadata_url = _build_instance_metadata_url(url, version, data)
         return _get_instance_metadata(metadata_url, num_retries=num_retries, timeout=timeout)
     except urllib.error.URLError:
+        boto.log.exception("Exception caught when trying to retrieve "
+                           "instance metadata for: %s", data)
         return None
 
 


Test patch:
diff --git a/tests/unit/provider/test_provider.py b/tests/unit/provider/test_provider.py
--- a/tests/unit/provider/test_provider.py
+++ b/tests/unit/provider/test_provider.py
@@ -6,6 +6,7 @@
 
 from boto import provider
 from boto.compat import expanduser
+from boto.exception import InvalidInstanceMetadataError
 
 
 INSTANCE_CONFIG = {
@@ -343,6 +344,32 @@ def test_metadata_server_credentials(self):
             self.get_instance_metadata.call_args[1]['data'],
             'meta-data/iam/security-credentials/')
 
+    def test_metadata_server_returns_bad_type(self):
+        self.get_instance_metadata.return_value = {
+            'rolename': [],
+        }
+        with self.assertRaises(InvalidInstanceMetadataError):
+            p = provider.Provider('aws')
+
+    def test_metadata_server_returns_empty_string(self):
+        self.get_instance_metadata.return_value = {
+            'rolename': ''
+        }
+        with self.assertRaises(InvalidInstanceMetadataError):
+            p = provider.Provider('aws')
+
+    def test_metadata_server_returns_missing_keys(self):
+        self.get_instance_metadata.return_value = {
+            'allowall': {
+                u'AccessKeyId': u'iam_access_key',
+                # Missing SecretAccessKey.
+                u'Token': u'iam_token',
+                u'Expiration': u'2012-09-01T03:57:34Z',
+            }
+        }
+        with self.assertRaises(InvalidInstanceMetadataError):
+            p = provider.Provider('aws')
+
     def test_refresh_credentials(self):
         now = datetime.utcnow()
         first_expiration = (now + timedelta(seconds=10)).strftime(