Repository: boto/boto

Issue:
add retry mechanism in some scenarios with 4XX HTTP response 
I'd suggest boto could support retry in some scenarios with 4XX HTTP response.

The first one is that 400 bad request in Route53 API.
Route53 document describes: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html

" If Route 53 can't process a request before the next request arrives, it will reject subsequent requests for the same hosted zone and return an HTTP 400 error (Bad request)..."

As we know, 400 bad request is for client errors. However, in this case, it actually means that Route53 servers cannot handle this new request due to the last one still in processing -- mostly it could be considered as Server errors. Therefore, it is really necessary for boto to add retry mechanism on this scenario. Boto's retry mechanism will have increasing sleeping time in later retries. So retry will not make things worse.

Here is how i could get this error:
My application has many roles. Each role has many instances. Each instance has its own R53 DNS record. In instance bootstrapping, it will add its own record itself and delete it when shutting down. And all of these records are in the same hosted zone. So when auto-scaling group takes actions (like health check replacing or scaling due to the policy) , it is really possible that many record change requests happen in this hosted zone. Thus the 400 bad request will happen frequently.

Currently, boto only retries in 500 and 503 for HTTP responses. I suggest to adding this 400 bad request with retry mechanism.

If later I find any new scenarios, I will add them here.

Thanks
Henry



Discussion:
Broadly, boto should **NOT** be retrying 400s. Since this is limited to one particular API call & some known conditions for getting the 400, that case is reasonable. I'll try to work up a fix.

Thanks for your confirmation. I guess there would be other similar special cases. Is it possible for boto 2.x to support the solution like _retry.json in botocore? https://github.com/boto/botocore/blob/develop/botocore/data/aws/_retry.json

It would be nice, but things aren't that well separated & compartmentalized in boto. Boto 3 should support that `retry.json` transparently. I have some time today, so I'll be getting back here shortly.


Resolving patch:
diff --git a/boto/route53/connection.py b/boto/route53/connection.py
--- a/boto/route53/connection.py
+++ b/boto/route53/connection.py
@@ -18,22 +18,25 @@
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
 # OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
 # ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
-# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 # WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 #
 
-import xml.sax
-import uuid
+import exception
+import random
 import urllib
+import uuid
+import xml.sax
+
 import boto
 from boto.connection import AWSAuthConnection
 from boto import handler
+import boto.jsonresponse
 from boto.route53.record import ResourceRecordSets
 from boto.route53.zone import Zone
-import boto.jsonresponse
-import exception
+
 
 HZXML = """<?xml version="1.0" encoding="UTF-8"?>
 <CreateHostedZoneRequest xmlns="%(xmlns)s">
@@ -43,7 +46,7 @@
     <Comment>%(comment)s</Comment>
   </HostedZoneConfig>
 </CreateHostedZoneRequest>"""
-        
+
 #boto.set_stream_logger('dns')
 
 
@@ -60,12 +63,13 @@ class Route53Connection(AWSAuthConnection):
     def __init__(self, aws_access_key_id=None, aws_secret_access_key=None,
                  port=None, proxy=None, proxy_port=None,
                  host=DefaultHost, debug=0, security_token=None,
-                 validate_certs=True):
+                 validate_certs=True, https_connection_factory=None):
         AWSAuthConnection.__init__(self, host,
                                    aws_access_key_id, aws_secret_access_key,
                                    True, port, proxy, proxy_port, debug=debug,
                                    security_token=security_token,
-                                   validate_certs=validate_certs)
+                                   validate_certs=validate_certs,
+                                   https_connection_factory=https_connection_factory)
 
     def _required_auth_capability(self):
         return ['route53']
@@ -79,7 +83,8 @@ def make_request(self, action, path, headers=None, data='', params=None):
                 pairs.append(key + '=' + urllib.quote(str(val)))
             path += '?' + '&'.join(pairs)
         return AWSAuthConnection.make_request(self, action, path,
-                                              headers, data)
+                                              headers, data,
+                                              retry_handler=self._retry_handler)
 
     # Hosted Zones
 
@@ -118,7 +123,7 @@ def get_all_hosted_zones(self, start_marker=None, zone_list=None):
     def get_hosted_zone(self, hosted_zone_id):
         """
         Get detailed information about a particular Hosted Zone.
-        
+
         :type hosted_zone_id: str
         :param hosted_zone_id: The unique identifier for the Hosted Zone
 
@@ -158,7 +163,7 @@ def create_hosted_zone(self, domain_name, caller_ref=None, comment=''):
         """
         Create a new Hosted Zone.  Returns a Python data structure with
         information about the newly created Hosted Zone.
-        
+
         :type domain_name: str
         :param domain_name: The name of the domain. This should be a
             fully-specified domain, and should end with a final period
@@ -178,7 +183,7 @@ def create_hosted_zone(self, domain_name, caller_ref=None, comment=''):
             use that.
 
         :type comment: str
-        :param comment: Any comments you want to include about the hosted      
+        :param comment: Any comments you want to include about the hosted
             zone.
 
         """
@@ -204,7 +209,7 @@ def create_hosted_zone(self, domain_name, caller_ref=None, comment=''):
             raise exception.DNSServerError(response.status,
                                            response.reason,
                                            body)
-        
+
     def delete_hosted_zone(self, hosted_zone_id):
         uri = '/%s/hostedzone/%s' % (self.Version, hosted_zone_id)
         response = self.make_request('DELETE', uri)
@@ -226,7 +231,7 @@ def get_all_rrsets(self, hosted_zone_id, type=None,
         """
         Retrieve the Resource Record Sets defined for this Hosted Zone.
         Returns the raw XML data returned by the Route53 call.
-        
+
         :type hosted_zone_id: str
         :param hosted_zone_id: The unique identifier for the Hosted Zone
 
@@ -401,3 +406,24 @@ def _make_qualified(self, value):
             if value and not value[-1] == '.':
                 value = "%s." % value
             return value
+
+    def _retry_handler(self, response, i, next_sleep):
+        status = None
+        boto.log.debug("Saw HTTP status: %s" % response.status)
+
+        if response.status == 400:
+            code = response.getheader('Code')
+
+            if code and 'PriorRequestNotComplete' in code:
+                # This is a case where we need to ignore a 400 error, as
+                # Route53 returns this. See
+                # http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html
+                msg = "%s, retry attempt %s" % (
+                    'PriorRequestNotComplete',
+                    i
+                )
+                next_sleep = random.random() * (2 ** i)
+                i += 1
+                status = (msg, i, next_sleep)
+
+        return status


Test patch:
diff --git a/tests/unit/route53/__init__.py b/tests/unit/route53/__init__.py
new file mode 100644
diff --git a/tests/unit/route53/test_connection.py b/tests/unit/route53/test_connection.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/route53/test_connection.py
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+# Copyright (c) 2013 Amazon.com, Inc. or its affiliates.  All Rights Reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+import mock
+
+from boto.exception import BotoServerError
+from boto.route53.connection import Route53Connection
+from boto.route53.exception import DNSServerError
+
+from tests.unit import unittest
+from tests.unit import AWSMockServiceTestCase
+
+
+class TestRoute53Connection(AWSMockServiceTestCase):
+    connection_class = Route53Connection
+
+    def setUp(self):
+        super(TestRoute53Connection, self).setUp()
+        self.calls = {
+            'count': 0,
+        }
+
+    def default_body(self):
+        return """<Route53Result>
+    <Message>It failed.</Message>
+</Route53Result>
+"""
+
+    def test_typical_400(self):
+        self.set_http_response(status_code=400, header=[
+            ['Code', 'Throttling'],
+        ])
+
+        with self.assertRaises(DNSServerError) as err:
+            self.service_connection.get_all_hosted_zones()
+
+        self.assertTrue('It failed.' in str(err.exception))
+
+    @mock.patch('time.sleep')
+    def test_retryable_400(self, sleep_mock):
+        self.set_http_response(status_code=400, header=[
+            ['Code', 'PriorRequestNotComplete'],
+        ])
+
+        def incr_retry_handler(func):
+            def _wrapper(*args, **kwargs):
+                self.calls['count'] += 1
+                return func(*args, **kwargs)
+            return _wrapper
+
+        # Patch.
+        orig_retry = self.service_connection._retry_handler
+        self.service_connection._retry_handler = incr_retry_handler(
+            orig_retry
+        )
+        self.assertEqual(self.calls['count'], 0)
+
+        # Retries get exhausted.
+        with self.assertRaises(BotoServerError):
+            self.service_connection.get_all_hosted_zones()
+
+        self.assertEqual(self.calls['count'], 7)
+
+        # Unpatch.
+        self.service_connection._retry_handler = orig_retry