Repository: boto/boto

Issue:
Can we add a retry solution with the corrupt IAM-Role credential provided by meta-data server?
In the real production environment, we just met several times that the meta-data server 169.254.169.254 had provided a corrupt IAM-Role credential, which caused the boto pop out this exception below.

So I think we need add a retry solution to handle this kind of situations, Like IAM-Role credential corrupt.

......
    self.conn = boto.connect_route53()
  File "/usr/local/lib/python2.7/site-packages/boto/**init**.py", line 364, in connect_route53
    **kwargs)
  File "/usr/local/lib/python2.7/site-packages/boto/route53/connection.py", line 68, in **init**
    validate_certs=validate_certs)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 541, in **init**
    security_token)
  File "/usr/local/lib/python2.7/site-packages/boto/provider.py", line 178, in **init**
    self.get_credentials(access_key, secret_key)
  File "/usr/local/lib/python2.7/site-packages/boto/provider.py", line 274, in get_credentials
    self._populate_keys_from_metadata_server()
  File "/usr/local/lib/python2.7/site-packages/boto/provider.py", line 292, in _populate_keys_from_metadata_server
    security = metadata.values()[0]
  File "/usr/local/lib/python2.7/site-packages/boto/utils.py", line 298, in values
    self._materialize()
  File "/usr/local/lib/python2.7/site-packages/boto/utils.py", line 261, in _materialize
    self[key]
  File "/usr/local/lib/python2.7/site-packages/boto/utils.py", line 279, in __getitem__
    val = json.loads(val)
  File "/usr/local/lib/python2.7/site-packages/simplejson/**init**.py", line 488, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 373, in decode
    raise JSONDecodeError("Extra data", s, end, len(s))
JSONDecodeError: Extra data: line 9 column 2 - line 9 column 22 (char 770 - 790)



Discussion:
I'm curious - what is the meta-data service actually returning? Please be sure to remove/obscure any real credentials before pasting it here!

I'm not opposed to adding retry logic here, but I'd like to understand the problem. Are you sure than immediately retrying the request succeeds? Have you been able to reproduce this issue with any consistency?

We already support this, but you have to explicitly configure this.

See: http://boto.readthedocs.org/en/latest/boto_config_tut.html#boto, you can specify the metadata_service_timeout as well as the metadata_service_num_attempts value.

I met this issue twice. But as you know, this happened in our production environment, so it is not so easy for me to add more debug info output there. Like to modify boto code and print the corrupted IAM-role credentials. Besides, it is not so easy to re-produce. I need to wait and wait for the next time. If any corrupted data found, I will paste here.

I have already configured the metadata_service options in boto.cfg. But this time, these two options do not cover.

For **timeout**,

``` python

    if timeout is not None:
        original = socket.getdefaulttimeout()
        socket.setdefaulttimeout(timeout)
    try:
        metadata_url = _build_instance_metadata_url(url, version, data)
        return _get_instance_metadata(metadata_url, num_retries=num_retries)
    except urllib2.URLError, e:
        return None
    finally:
        if timeout is not None:
            socket.setdefaulttimeout(original)
```

as code in boto/utils.py, it only covers the exception **urllib2.URLError**.

For **retries**, it only covers the "retry_url" function. But if any corrupted data found, the code will not handle that exception.

``` python
class LazyLoadMetadata(dict):
    def __init__(self, url, num_retries):
        self._url = url
        self._num_retries = num_retries
        self._leaves = {}
        self._dicts = []
        data = boto.utils.retry_url(self._url, num_retries=self._num_retries)
```

The exception happened here.

``` python
        if data:
            fields = data.split('\n')
            for field in fields:
                if field.endswith('/'):
                    key = field[0:-1]
                    self._dicts.append(key)
                else:
                    p = field.find('=')
                    if p > 0:
                        key = field[p + 1:]
                        resource = field[0:p] + '/openssh-key'
                    else:
                        key = resource = field
                    self._leaves[key] = resource
                self[key] = None
```

This issue happened in a short time. Later it will be recovered by itself. I guess that AWS metadata server has some issues there. So the return data is corrupted. 

Therefore, I think it is really necessary to add immediate retry here. and soon or later AWS will fix that issue.
But if no retry here, I have to handle this exception outside in my program. It is not difficult but maybe better to handle inside boto.

If you can write a test to mock the JSONDecodeError and then submit a pull request which properly handles the case with a retry, I'd be willing to merge it in. This looks like a worthwhile fix for a rare issue.


Resolving patch:
diff --git a/boto/utils.py b/boto/utils.py
--- a/boto/utils.py
+++ b/boto/utils.py
@@ -50,6 +50,7 @@
 import boto
 import boto.provider
 import tempfile
+import random
 import smtplib
 import datetime
 import re
@@ -75,6 +76,11 @@
 
 from boto.compat import json
 
+try:
+    from json import JSONDecodeError as ValueError
+except ImportError:
+    pass
+
 # List of Query String Arguments of Interest
 qsa_of_interest = ['acl', 'cors', 'defaultObjectAcl', 'location', 'logging',
                    'partNumber', 'policy', 'requestPayment', 'torrent',
@@ -89,7 +95,8 @@
                    # having it listed here should cause no problems because
                    # GET bucket?storageClass is not part of the S3 API.)
                    'storageClass',
-                   # websiteConfig is a QSA for buckets in Google Cloud Storage.
+                   # websiteConfig is a QSA for buckets in Google Cloud
+                   # Storage.
                    'websiteConfig',
                    # compose is a QSA for objects in Google Cloud Storage.
                    'compose']
@@ -117,8 +124,9 @@ def canonical_string(method, path, headers, expires=None,
     interesting_headers = {}
     for key in headers:
         lk = key.lower()
-        if headers[key] != None and (lk in ['content-md5', 'content-type', 'date'] or
-                                     lk.startswith(provider.header_prefix)):
+        if headers[key] is not None and \
+                (lk in ['content-md5', 'content-type', 'date'] or
+                 lk.startswith(provider.header_prefix)):
             interesting_headers[lk] = str(headers[key]).strip()
 
     # these keys get empty strings if they don't exist
@@ -156,7 +164,7 @@ def canonical_string(method, path, headers, expires=None,
         qsa = [a.split('=', 1) for a in qsa]
         qsa = [unquote_v(a) for a in qsa if a[0] in qsa_of_interest]
         if len(qsa) > 0:
-            qsa.sort(cmp=lambda x, y:cmp(x[0], y[0]))
+            qsa.sort(cmp=lambda x, y: cmp(x[0], y[0]))
             qsa = ['='.join(a) for a in qsa]
             buf += '?'
             buf += '&'.join(qsa)
@@ -272,15 +280,48 @@ def __getitem__(self, key):
 
         if key in self._leaves:
             resource = self._leaves[key]
-            val = boto.utils.retry_url(self._url + urllib.quote(resource,
-                                                                safe="/:"),
-                                       num_retries=self._num_retries)
-            if val and val[0] == '{':
-                val = json.loads(val)
+
+            for i in range(0, self._num_retries):
+                try:
+                    val = boto.utils.retry_url(
+                        self._url + urllib.quote(resource,
+                                                 safe="/:"),
+                        num_retries=self._num_retries)
+                    if val and val[0] == '{':
+                        val = json.loads(val)
+                        break
+                    else:
+                        p = val.find('\n')
+                        if p > 0:
+                            val = val.split('\n')
+                        break
+
+                except ValueError, e:
+                    boto.log.debug(
+                        "encountered '%s' exception: %s" % (
+                            e.__class__.__name__, e))
+                    boto.log.debug(
+                        'corrupted JSON data found: %s' % val)
+
+                except Exception, e:
+                    boto.log.debug("encountered unretryable" +
+                                   " '%s' exception, re-raising" % (
+                                       e.__class__.__name__))
+                    raise
+
+                boto.log.error("Caught exception reading meta data" +
+                               " for the '%s' try" % (i + 1))
+
+                if i + 1 != self._num_retries:
+                    next_sleep = random.random() * (2 ** i)
+                    time.sleep(next_sleep)
             else:
-                p = val.find('\n')
-                if p > 0:
-                    val = val.split('\n')
+                boto.log.error('Unable to read meta data, giving up')
+                boto.log.error(
+                    "encountered '%s' exception: %s" % (
+                        e.__class__.__name__, e))
+                raise
+
             self[key] = val
         elif key in self._dicts:
             self[key] = LazyLoadMetadata(self._url + key + '/',
@@ -358,7 +399,8 @@ def get_instance_identity(version='latest', url='http://169.254.169.254',
     Returns the instance identity as a nested Python dictionary.
     """
     iid = {}
-    base_url = _build_instance_metadata_url(url, version, 'dynamic/instance-identity')
+    base_url = _build_instance_metadata_url(url, version,
+                                            'dynamic/instance-identity')
     if timeout is not None:
         original = socket.getdefaulttimeout()
         socket.setdefaulttimeout(timeout)
@@ -396,6 +438,7 @@ def get_instance_userdata(version='latest', sep=None,
 ISO8601_MS = '%Y-%m-%dT%H:%M:%S.%fZ'
 RFC1123 = '%a, %d %b %Y %H:%M:%S %Z'
 
+
 def get_ts(ts=None):
     if not ts:
         ts = time.gmtime()
@@ -415,6 +458,7 @@ def parse_ts(ts):
             dt = datetime.datetime.strptime(ts, RFC1123)
             return dt
 
+
 def find_class(module_name, class_name=None):
     if class_name:
         module_name = "%s.%s" % (module_name, class_name)
@@ -444,13 +488,13 @@ def update_dme(username, password, dme_id, ip_address):
 
 def fetch_file(uri, file=None, username=None, password=None):
     """
-    Fetch a file based on the URI provided. If you do not pass in a file pointer
-    a tempfile.NamedTemporaryFile, or None if the file could not be
-    retrieved is returned.
+    Fetch a file based on the URI provided.
+    If you do not pass in a file pointer a tempfile.NamedTemporaryFile,
+    or None if the file could not be retrieved is returned.
     The URI can be either an HTTP url, or "s3://bucket_name/key_name"
     """
     boto.log.info('Fetching %s' % uri)
-    if file == None:
+    if file is None:
         file = tempfile.NamedTemporaryFile()
     try:
         if uri.startswith('s3://'):
@@ -495,7 +539,7 @@ def run(self, cwd=None):
                                         stderr=subprocess.PIPE,
                                         cwd=cwd)
         if(self.wait):
-            while self.process.poll() == None:
+            while self.process.poll() is None:
                 time.sleep(1)
                 t = self.process.communicate()
                 self.log_fp.write(t[0])
@@ -504,7 +548,8 @@ def run(self, cwd=None):
             self.exit_code = self.process.returncode
 
             if self.fail_fast and self.exit_code != 0:
-                raise Exception("Command " + self.command + " failed with status " + self.exit_code)
+                raise Exception("Command " + self.command +
+                                " failed with status " + self.exit_code)
 
             return self.exit_code
 
@@ -514,12 +559,14 @@ def setReadOnly(self, value):
     def getStatus(self):
         return self.exit_code
 
-    status = property(getStatus, setReadOnly, None, 'The exit code for the command')
+    status = property(getStatus, setReadOnly, None,
+                      'The exit code for the command')
 
     def getOutput(self):
         return self.log_fp.getvalue()
 
-    output = property(getOutput, setReadOnly, None, 'The STDIN and STDERR output of the command')
+    output = property(getOutput, setReadOnly, None,
+                      'The STDIN and STDERR output of the command')
 
 
 class AuthSMTPHandler(logging.handlers.SMTPHandler):
@@ -565,10 +612,10 @@ def emit(self, record):
             smtp.login(self.username, self.password)
             msg = self.format(record)
             msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
-                            self.fromaddr,
-                            ','.join(self.toaddrs),
-                            self.getSubject(record),
-                            email.utils.formatdate(), msg)
+                self.fromaddr,
+                ','.join(self.toaddrs),
+                self.getSubject(record),
+                email.utils.formatdate(), msg)
             smtp.sendmail(self.fromaddr, self.toaddrs, msg)
             smtp.quit()
         except (KeyboardInterrupt, SystemExit):
@@ -710,7 +757,7 @@ def __init__(self, str=None, hashfunc=None):
         """
         self.str = str
         if hashfunc:
-           self.hashfunc = hashfunc
+            self.hashfunc = hashfunc
 
     def set(self, value):
         self.str = self.hashfunc(value).hexdigest()
@@ -719,7 +766,7 @@ def __str__(self):
         return str(self.str)
 
     def __eq__(self, other):
-        if other == None:
+        if other is None:
             return False
         return str(self.hashfunc(other).hexdigest()) == str(self.str)
 
@@ -734,12 +781,14 @@ def notify(subject, body=None, html_body=None, to_string=None,
            attachments=None, append_instance_id=True):
     attachments = attachments or []
     if append_instance_id:
-        subject = "[%s] %s" % (boto.config.get_value("Instance", "instance-id"), subject)
+        subject = "[%s] %s" % (
+            boto.config.get_value("Instance", "instance-id"), subject)
     if not to_string:
         to_string = boto.config.get_value('Notification', 'smtp_to', None)
     if to_string:
         try:
-            from_string = boto.config.get_value('Notification', 'smtp_from', 'boto')
+            from_string = boto.config.get_value('Notification',
+                                                'smtp_from', 'boto')
             msg = email.mime.multipart.MIMEMultipart()
             msg['From'] = from_string
             msg['Reply-To'] = from_string
@@ -759,11 +808,13 @@ def notify(subject, body=None, html_body=None, to_string=None,
             for part in attachments:
                 msg.attach(part)
 
-            smtp_host = boto.config.get_value('Notification', 'smtp_host', 'localhost')
+            smtp_host = boto.config.get_value('Notification',
+                                              'smtp_host', 'localhost')
 
             # Alternate port support
             if boto.config.get_value("Notification", "smtp_port"):
-                server = smtplib.SMTP(smtp_host, int(boto.config.get_value("Notification", "smtp_port")))
+                server = smtplib.SMTP(smtp_host, int(
+                    boto.config.get_value("Notification", "smtp_port")))
             else:
                 server = smtplib.SMTP(smtp_host)
 
@@ -874,7 +925,7 @@ def guess_mime_type(content, deftype):
     :rtype: <type>:
     :return: <description>
     """
-    #Mappings recognized by cloudinit
+    # Mappings recognized by cloudinit
     starts_with_mappings = {
         '#include': 'text/x-include-url',
         '#!': 'text/x-shellscript',


Test patch:
diff --git a/tests/unit/utils/test_utils.py b/tests/unit/utils/test_utils.py
--- a/tests/unit/utils/test_utils.py
+++ b/tests/unit/utils/test_utils.py
@@ -25,11 +25,14 @@
 
 import mock
 
+import boto.utils
 from boto.utils import Password
 from boto.utils import pythonize_name
 from boto.utils import _build_instance_metadata_url
 from boto.utils import retry_url
+from boto.utils import LazyLoadMetadata
 
+from boto.compat import json
 
 class TestPassword(unittest.TestCase):
     """Test basic password functionality"""
@@ -185,6 +188,45 @@ def test_retry_url_uses_proxy(self):
         response = retry_url('http://10.10.10.10/foo', num_retries=1)
         self.assertEqual(response, 'no proxy response')
 
+class TestLazyLoadMetadata(unittest.TestCase):
+
+    def setUp(self):
+        self.retry_url_patch = mock.patch('boto.utils.retry_url')
+        boto.utils.retry_url = self.retry_url_patch.start()
+
+    def tearDown(self):
+        self.retry_url_patch.stop()
+
+    def set_normal_response(self, data):
+        # here "data" should be a list of return values in some order
+        fake_response = mock.Mock()
+        fake_response.side_effect = data
+        boto.utils.retry_url = fake_response
+
+    def test_meta_data_with_invalid_json_format_happened_once(self):
+        # here "key_data" will be stored in the "self._leaves"
+        # when the class "LazyLoadMetadata" initialized
+        key_data = "test"
+        invalid_data = '{"invalid_json_format" : true,}'
+        valid_data = '{ "%s" : {"valid_json_format": true}}' % key_data
+        url = "/".join(["http://169.254.169.254", key_data])
+        num_retries = 2
+
+        self.set_normal_response([key_data, invalid_data, valid_data])
+        response = LazyLoadMetadata(url, num_retries)
+        self.assertEqual(response.values()[0], json.loads(valid_data))
+
+    def test_meta_data_with_invalid_json_format_happened_twice(self):
+        key_data = "test"
+        invalid_data = '{"invalid_json_format" : true,}'
+        valid_data = '{ "%s" : {"valid_json_format": true}}' % key_data
+        url = "/".join(["http://169.254.169.254", key_data])
+        num_retries = 2
+
+        self.set_normal_response([key_data, invalid_data, invalid_data])
+        response = LazyLoadMetadata(url, num_retries)
+        with self.assertRaises(ValueError):
+            response.values()[0]
 
 if __name__ == '__main__':
     unittest.main()