Repository: boto/botocore

Issue:
Can't put contents to s3 bucket outside of US standard region with file object
When you use a file like object as the body param to s3.PutObject for a bucket outside of US standard, you'll get a 307 followed by a 400 bad request and eventually a socket timeout.  To repro:

1) Create a bucket outside of us standard.
2) Send a PutObject request with a file like object for the body: `op.call(endpoint, body=open('/some/file', 'rb'), ...)`

You'll see a 307 then a 400.  I think this is because requests is configured to follow redirects, but there's no location header and hence the bad request.

I think the fact that this is a file like object exposes the bug because we send two packets, one for the request+headers then one for the body.  When the body is a string (like what we use in the integration tests), then httplib will do a `msg += message_body` and send it as a single chunk which succeeds.

This also might be related to the virtual host handler we have.  I noticed this in the log messages:

```
botocore.handlers: DEBUG: Checking for DNS compatible bucket for: https://s3-us-west-2.amazonaws.com/botocoretest1374195914-317
botocore.handlers: DEBUG: URI updated to: https://botocoretest1374195914-317.s3.amazonaws.com
```

Would be good to get an integration test written for this also.



Discussion:
I'm assuming that the bucket you were trying to write to was a newly created bucket?

In that case, you can receive a temporary redirect (307) when you try to access the bucket because the DNS records that have to be created to resolve the bucket using virtual-hosted addressing have not yet propagated throughout the network.

This 307 redirect should, however, contain a Location header.  Are you sure there was no location header present?

Did more digging into this.  You do get a location header with the region specific endpoint (bucket.s3region.amazonaws.com), but the problem is that the fileobj has been exhausted so `.read()` returns nothing.  When you send the request to the new location it waits for `content-length` bytes, but eventually s3 closes the socket on their end.

Does requests provide anything to rewind a stream?  Is this something we could do before requests retries the request?

I spent more time researching this and have a few observations:
- The ideal way to handle this would be through an `Expect: 100-continue` header.  From the research I've done it seems like `httplib` does not properly support this.  It will handle a 100 continue response but it already sends the body.  I also double checked that requests/urllib3 does not support 100 continue.
- The requests library has a hook when a request is received, this includes redirect responses.  We could add a hook that `.seek(0)`'s the body on redirects, otherwise it raises an error if it's not seekable.  The only downside here is that we're potentially sending part of the body twice.

The reason I'd like to get this implemented is for better memory utilization in this branch of the CLI: https://github.com/jamesls/aws-cli/compare/memory-utilization

This branch is ready to go, except that by switching from bytearrays to file objects, we run into this issue here.


Resolving patch:
diff --git a/botocore/awsrequest.py b/botocore/awsrequest.py
--- a/botocore/awsrequest.py
+++ b/botocore/awsrequest.py
@@ -20,9 +20,15 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 #
+import logging
 from requests import models
+from requests.sessions import REDIRECT_STATI
 
-from botocore.compat import HTTPHeaders
+from botocore.compat import HTTPHeaders, file_type
+from botocore.exceptions import UnseekableStreamError
+
+
+logger = logging.getLogger(__name__)
 
 
 class AWSRequest(models.RequestEncodingMixin, models.Request):
@@ -77,3 +83,14 @@ class AWSPreparedRequest(models.PreparedRequest):
     def __init__(self, original_request):
         self.original = original_request
         super(AWSPreparedRequest, self).__init__()
+        self.hooks.setdefault('response', []).append(self.reset_stream)
+
+    def reset_stream(self, response, **kwargs):
+        if response.status_code in REDIRECT_STATI and \
+                isinstance(self.body, file_type):
+            logger.debug("Redirect received, rewinding stream: %s", self.body)
+            try:
+                self.body.seek(0)
+            except Exception as e:
+                logger.debug("Unable to rewind stream: %s", e)
+                raise UnseekableStreamError(stream_object=self.body)
diff --git a/botocore/compat.py b/botocore/compat.py
--- a/botocore/compat.py
+++ b/botocore/compat.py
@@ -32,6 +32,8 @@ class HTTPHeaders(http_client.HTTPMessage):
     from urllib.parse import urlunsplit
     from urllib.parse import urljoin
     from urllib.parse import parse_qsl
+    from io import IOBase as _IOBase
+    file_type = _IOBase
 else:
     from urllib import quote
     from urllib import unquote
@@ -40,6 +42,7 @@ class HTTPHeaders(http_client.HTTPMessage):
     from urlparse import urljoin
     from urlparse import parse_qsl
     from email.message import Message
+    file_type = file
     class HTTPHeaders(Message):
 
         # The __iter__ method is not available in python2.x, so we have
diff --git a/botocore/exceptions.py b/botocore/exceptions.py
--- a/botocore/exceptions.py
+++ b/botocore/exceptions.py
@@ -209,3 +209,11 @@ class ChecksumError(BotoCoreError):
     fmt = ('Checksum {checksum_type} failed, expected checksum '
            '{expected_checksum} did not match calculated checksum '
            '{actual_checksum}.')
+
+
+class UnseekableStreamError(BotoCoreError):
+    """Need to seek a stream, but stream does not support seeking.
+
+    """
+    fmt = ('Need to rewind the stream {stream_object}, but stream '
+           'is not seekable.')


Test patch:
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -19,10 +19,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 #
+import os
 import time
 import random
 from tests import unittest
 from collections import defaultdict
+import tempfile
+import shutil
 import threading
 try:
     from itertools import izip_longest as zip_longest
@@ -39,6 +42,31 @@ def setUp(self):
         self.session = botocore.session.get_session()
         self.service = self.session.get_service('s3')
         self.endpoint = self.service.get_endpoint('us-east-1')
+        self.keys = []
+
+    def create_object(self, key_name, body='foo'):
+        self.keys.append(key_name)
+        operation = self.service.get_operation('PutObject')
+        response = operation.call(
+            self.endpoint, bucket=self.bucket_name, key=key_name,
+            body=body)[0]
+        self.assertEqual(response.status_code, 200)
+
+    def create_multipart_upload(self, key_name):
+        operation = self.service.get_operation('CreateMultipartUpload')
+        http_response, parsed = operation.call(self.endpoint,
+                                               bucket=self.bucket_name,
+                                               key=key_name)
+        upload_id = parsed['UploadId']
+        self.addCleanup(self.service.get_operation('AbortMultipartUpload').call,
+                        self.endpoint, upload_id=upload_id,
+                        bucket=self.bucket_name, key=key_name)
+
+    def create_object_catch_exceptions(self, key_name):
+        try:
+            self.create_object(key_name=key_name)
+        except Exception as e:
+            self.caught_exceptions.append(e)
 
 
 class TestS3Buckets(BaseS3Test):
@@ -61,7 +89,6 @@ def setUp(self):
             int(time.time()), random.randint(1, 1000))
         operation = self.service.get_operation('CreateBucket')
         operation.call(self.endpoint, bucket=self.bucket_name)
-        self.keys = []
 
     def tearDown(self):
         for key in self.keys:
@@ -71,33 +98,9 @@ def tearDown(self):
         operation = self.service.get_operation('DeleteBucket')
         operation.call(self.endpoint, bucket=self.bucket_name)
 
-    def create_object(self, key_name, body='foo'):
-        self.keys.append(key_name)
-        operation = self.service.get_operation('PutObject')
-        response = operation.call(
-            self.endpoint, bucket=self.bucket_name, key=key_name,
-            body=body)[0]
-        self.assertEqual(response.status_code, 200)
-
-    def create_multipart_upload(self, key_name):
-        operation = self.service.get_operation('CreateMultipartUpload')
-        http_response, parsed = operation.call(self.endpoint,
-                                               bucket=self.bucket_name,
-                                               key=key_name)
-        upload_id = parsed['UploadId']
-        self.addCleanup(self.service.get_operation('AbortMultipartUpload').call,
-                        self.endpoint, upload_id=upload_id,
-                        bucket=self.bucket_name, key=key_name)
-
     def increment_auth(self, request, auth, **kwargs):
         self.auth_paths.append(auth.auth_path)
 
-    def create_object_catch_exceptions(self, key_name):
-        try:
-            self.create_object(key_name=key_name)
-        except Exception as e:
-            self.caught_exceptions.append(e)
-
     def test_can_delete_urlencoded_object(self):
         key_name = 'a+b/foo'
         self.create_object(key_name=key_name)
@@ -269,5 +272,47 @@ def test_thread_safe_auth(self):
             "Expected 10 unique auth paths, instead received: %s" % (self.auth_paths))
 
 
+class TestS3Regions(BaseS3Test):
+    def setUp(self):
+        super(TestS3Regions, self).setUp()
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir)
+
+    def create_bucket_in_region(self, region):
+        bucket_name = 'botocoretest%s-%s' % (
+            int(time.time()), random.randint(1, 1000))
+        operation = self.service.get_operation('CreateBucket')
+        response, parsed = operation.call(
+            self.endpoint, bucket=bucket_name,
+            create_bucket_configuration={'LocationConstraint': 'us-west-2'})
+        self.assertEqual(response.status_code, 200)
+        self.addCleanup(self.service.get_operation('DeleteBucket').call,
+                        bucket=bucket_name, endpoint=self.endpoint)
+        return bucket_name
+
+    def test_reset_stream_on_redirects(self):
+        # Create a bucket in a non classic region.
+        bucket_name = self.create_bucket_in_region('us-west-2')
+        # Then try to put a file like object to this location.
+        filename = os.path.join(self.tempdir, 'foo')
+        with open(filename, 'wb') as f:
+            f.write(b'foo' * 1024)
+        put_object = self.service.get_operation('PutObject')
+        with open(filename, 'rb') as f:
+            response, parsed = put_object.call(
+                self.endpoint, bucket=bucket_name, key='foo', body=f)
+        self.assertEqual(
+            response.status_code, 200,
+            "Non 200 status code (%s) received: %s" % (
+                response.status_code, parsed))
+
+        operation = self.service.get_operation('GetObject')
+        data = operation.call(self.endpoint, bucket=bucket_name,
+                              key='foo')[1]
+        self.assertEqual(data['Body'].read(), b'foo' * 1024)
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/tests/unit/test_awsrequest.py b/tests/unit/test_awsrequest.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_awsrequest.py
@@ -0,0 +1,100 @@
+#!/usr/bin/env
+# Copyright 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+from tests import unittest
+import os
+import tempfile
+import shutil
+
+from mock import Mock
+
+from botocore.exceptions import UnseekableStreamError
+from botocore.awsrequest import AWSRequest
+from botocore.compat import file_type
+
+
+class Unseekable(file_type):
+    def __init__(self, stream):
+        self._stream = stream
+
+    def read(self):
+        return self._stream.read()
+
+    def seek(self, offset, whence):
+        # This is a case where seek() exists as part of the object's interface,
+        # but it doesn't actually work (for example socket.makefile(), which
+        # will raise an io.* error on python3).
+        raise ValueError("Underlying stream does not support seeking.")
+
+
+class TestAWSRequest(unittest.TestCase):
+
+    def setUp(self):
+        self.tempdir = tempfile.mkdtemp()
+        self.request = AWSRequest(url='http://example.com')
+        self.prepared_request = self.request.prepare()
+        self.filename = os.path.join(self.tempdir, 'foo')
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir)
+
+    def test_should_reset_stream(self):
+        with open(self.filename, 'wb') as f:
+            f.write(b'foobarbaz')
+        body = open(self.filename, 'rb')
+        self.prepared_request.body = body
+
+        # Now pretend we try to send the request.
+        # This means that we read the body:
+        body.read()
+        # And create a response object that indicates
+        # a redirect.
+        fake_response = Mock()
+        fake_response.status_code = 307
+
+        # Then requests calls our reset_stream hook.
+        self.prepared_request.reset_stream(fake_response)
+
+        # The stream should now be reset.
+        self.assertEqual(body.tell(), 0)
+
+    def test_cannot_reset_stream_raises_error(self):
+        with open(self.filename, 'wb') as f:
+            f.write(b'foobarbaz')
+        body = open(self.filename, 'rb')
+        self.prepared_request.body = Unseekable(body)
+
+        # Now pretend we try to send the request.
+        # This means that we read the body:
+        body.read()
+        # And create a response object that indicates
+        # a redirect.
+        fake_response = Mock()
+        fake_response.status_code = 307
+
+        # Then requests calls our reset_stream hook.
+        with self.assertRaises(UnseekableStreamError):
+            self.prepared_request.reset_stream(fake_response)
+
+
+if __name__ == "__main__":
+    unittest.main()