Repository: boto/botocore (python)

Issue:
SignatureDoesNotMatch on first request
i'm having a weird issue where the first request via boto results in a 403 Forbidden, while subsequent requests succeed. this is the result of the first request:

```
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ACCESSKEY</AWSAccessKeyId><StringToSign>GET


1458556343
/prod-jimdo-userfiles/16261693/image/7195799060</StringToSign><SignatureProvided>4tFMCWtbbZMTJCgAUeTe Jx7K2I=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 31 34 35 38 35 35 36 33 34 33 0a 2f 70 72 6f 64 2d 6a 69 6d 64 6f 2d 75 73 65 72 66 69 6c 65 73 2f 31 36 32 36 31 36 39 33 2f 69 6d 61 67 65 2f 37 31 39 35 37 39 39 30 36 30</StringToSignBytes><RequestId>B310FAD6A4EF1700</RequestId><HostId>6QCy5Tv5bm+yO8Fv78ZZd6a4IQPVLpRggP0+eOv9P0xqT04obzT3sD2WcRv1mEBvvGU/nvolSYg=</HostId></Error>
```

we are using boto as a backend-adapter in thumbor and those are the application logs for the same request-uri. as you can see the first request fails while the second one succeeds:

```
2016-03-21 10:32:22 botocore.credentials:DEBUG Skipping environment variable credential check because profile name was explicitly set.
2016-03-21 10:32:22 botocore.credentials:DEBUG Looking for credentials via: env
2016-03-21 10:32:22 botocore.credentials:INFO Found credentials in environment variables.
2016-03-21 10:32:22 botocore.client:DEBUG Registering retry handlers for service: s3
2016-03-21 10:32:22 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10ed2c488>
2016-03-21 10:32:22 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10ed26c08>
2016-03-21 10:32:23 botocore.endpoint:DEBUG Setting s3 timeout as (60, 60)
2016-03-21 10:32:23 thumbor:DEBUG Cleaning key: 16261693/image/7195799060
2016-03-21 10:32:23 thumbor:DEBUG Cleansed key: 16261693/image/7195799060
2016-03-21 10:32:23 botocore.utils:DEBUG Checking for DNS compatible bucket for: https://s3-eu-west-1.amazonaws.com/prod-jimdo-userfiles/16261693/image/7195799060
2016-03-21 10:32:23 botocore.utils:DEBUG URI updated to: https://prod-jimdo-userfiles.s3.amazonaws.com/16261693/image/7195799060
2016-03-21 10:32:23 botocore.auth:DEBUG Calculating signature using hmacv1 auth.
2016-03-21 10:32:23 botocore.auth:DEBUG HTTP request method: GET
2016-03-21 10:32:23 botocore.auth:DEBUG StringToSign:
GET


1458556343
/prod-jimdo-userfiles/16261693/image/7195799060
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG GET https://prod-jimdo-userfiles.s3.amazonaws.com/16261693/image/7195799060?AWSAccessKeyId=ACCESSKEY&Expires=1458556343&Signature=4tFMCWtbbZMTJCgAUeTe+Jx7K2I=
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Trying 54.231.134.50...
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Connected to prod-jimdo-userfiles.s3.amazonaws.com (54.231.134.50) port 443 (#0)
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Server certificate: *.s3.amazonaws.com
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Server certificate: DigiCert SHA2 High Assurance Server CA
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Server certificate: DigiCert High Assurance EV Root CA
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG > GET /16261693/image/7195799060?AWSAccessKeyId=ACCESSKEY&Expires=1458556343&Signature=4tFMCWtbbZMTJCgAUeTe+Jx7K2I= HTTP/1.1
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG > Host: prod-jimdo-userfiles.s3.amazonaws.com
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG > User-Agent: Thumbor/6.0.0
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG > Accept: */*
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG > Accept-Encoding: gzip,deflate
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG >
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < HTTP/1.1 403 Forbidden
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < x-amz-request-id: 758EA6A945952077
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < x-amz-id-2: DyHphy98auesGDfoH9apyipSqcF+pO8tgfDYVM1FTZwlnKjiYhEKADNRiY+xwqTO0ZYCoq+bUC4=
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < Content-Type: application/xml
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < Transfer-Encoding: chunked
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < Date: Mon, 21 Mar 2016 09:32:23 GMT
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG < Server: AmazonS3
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG <
2016-03-21 10:32:23 tornado.curl_httpclient:DEBUG Connection #0 to host prod-jimdo-userfiles.s3.amazonaws.com left intact
2016-03-21 10:32:23 thumbor:WARNING ERROR retrieving image https://prod-jimdo-userfiles.s3.amazonaws.com/16261693/image/7195799060?AWSAccessKeyId=ACCESSKEY&Expires=1458556343&Signature=4tFMCWtbbZMTJCgAUeTe+Jx7K2I=: HTTP 403: Forbidden
2016-03-21 10:32:23 tornado.access:WARNING 404 HEAD /unsafe/300x400/filters:format(jpg)/prod-jimdo-userfiles/16261693/image/7195799060 (127.0.0.1) 830.67ms
2016-03-21 10:32:34 botocore.client:DEBUG Registering retry handlers for service: s3
2016-03-21 10:32:34 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10ed2c488>
2016-03-21 10:32:34 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10ed26c08>
2016-03-21 10:32:34 botocore.endpoint:DEBUG Setting s3 timeout as (60, 60)
2016-03-21 10:32:34 thumbor:DEBUG Cleaning key: 16261693/image/7195799060
2016-03-21 10:32:34 thumbor:DEBUG Cleansed key: 16261693/image/7195799060
2016-03-21 10:32:34 botocore.utils:DEBUG Checking for DNS compatible bucket for: https://s3-eu-west-1.amazonaws.com/prod-jimdo-userfiles/16261693/image/7195799060
2016-03-21 10:32:34 botocore.utils:DEBUG URI updated to: https://prod-jimdo-userfiles.s3.amazonaws.com/16261693/image/7195799060
2016-03-21 10:32:34 botocore.auth:DEBUG Calculating signature using hmacv1 auth.
2016-03-21 10:32:34 botocore.auth:DEBUG HTTP request method: GET
2016-03-21 10:32:34 botocore.auth:DEBUG StringToSign:
GET


1458556354
/prod-jimdo-userfiles/16261693/image/7195799060
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG GET https://prod-jimdo-userfiles.s3.amazonaws.com/16261693/image/7195799060?AWSAccessKeyId=ACCESSKEY&Expires=1458556354&Signature=M7BgdiStlesfgTX7KZmTCZ9z9HE=
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Connection 0 seems to be dead!
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Closing connection 0
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Hostname prod-jimdo-userfiles.s3.amazonaws.com was found in DNS cache
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Trying 54.231.134.50...
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Connected to prod-jimdo-userfiles.s3.amazonaws.com (54.231.134.50) port 443 (#1)
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG SSL re-using session ID
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Server certificate: *.s3.amazonaws.com
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Server certificate: DigiCert SHA2 High Assurance Server CA
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG Server certificate: DigiCert High Assurance EV Root CA
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG > GET /16261693/image/7195799060?AWSAccessKeyId=ACCESSKEY&Expires=1458556354&Signature=M7BgdiStlesfgTX7KZmTCZ9z9HE= HTTP/1.1
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG > Host: prod-jimdo-userfiles.s3.amazonaws.com
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG > User-Agent: Thumbor/6.0.0
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG > Accept: */*
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG > Accept-Encoding: gzip,deflate
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG >
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < HTTP/1.1 200 OK
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < x-amz-id-2: +BAMlvUDPKbbLzSIjuoaw4nXOWPYDjgmHHEyeCV6VvcRkT2pJIniy4RXZQtZCsta4/b7cQ8DIGw=
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < x-amz-request-id: 3905F44007793DCE
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Date: Mon, 21 Mar 2016 09:32:35 GMT
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < x-amz-version-id: WE1YzAUEXpbyOToJWD_jPH2uyOqV7De5
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Last-Modified: Mon, 22 Feb 2016 11:37:57 GMT
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < ETag: "ce4179f66ac27601473499f9eea90417"
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Accept-Ranges: bytes
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Content-Type: binary/octet-stream
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Content-Length: 335593
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG < Server: AmazonS3
2016-03-21 10:32:34 tornado.curl_httpclient:DEBUG <
2016-03-21 10:32:35 tornado.curl_httpclient:DEBUG Connection #1 to host prod-jimdo-userfiles.s3.amazonaws.com left intact
2016-03-21 10:32:35 thumbor:DEBUG Format specified: jpg
2016-03-21 10:32:35 thumbor:DEBUG Image format specified as .jpg.
2016-03-21 10:32:35 thumbor:DEBUG Content Type of image/jpeg detected.
2016-03-21 10:32:35 tornado.access:INFO 200 HEAD /unsafe/300x400/filters:format(jpg)/prod-jimdo-userfiles/16261693/image/7195799060 (127.0.0.1) 697.12ms
```

Pull request:
Do not sign user-agent
The user agent may be changed somewhere along the line, so we should not
be signing it.

 #846

cc @kyleknap @jamesls 

Patch:
diff --git a/botocore/auth.py b/botocore/auth.py
index 3621016fd..c9a660935 100644
--- a/botocore/auth.py
+++ b/botocore/auth.py
@@ -44,7 +44,8 @@ PAYLOAD_BUFFER = 1024 * 1024
 ISO8601 = '%Y-%m-%dT%H:%M:%SZ'
 SIGV4_TIMESTAMP = '%Y%m%dT%H%M%SZ'
 SIGNED_HEADERS_BLACKLIST = [
-    'expect'
+    'expect',
+    'user-agent'
 ]
 
 


Test patch:
diff --git a/tests/unit/auth/test_signers.py b/tests/unit/auth/test_signers.py
index 6ce0e5259..54c7cd574 100644
--- a/tests/unit/auth/test_signers.py
+++ b/tests/unit/auth/test_signers.py
@@ -319,16 +319,22 @@ class TestS3SigV4Auth(BaseTestWithFixedDate):
         cqs = self.auth.canonical_query_string(request)
         self.assertEqual('marker=%C3%A4%C3%B6%C3%BC-01.txt&prefix=', cqs)
 
-    def test_blacklist_expect_headers(self):
+    def _test_blacklist_header(self, header, value):
         request = AWSRequest()
         request.url = 'https://s3.amazonaws.com/bucket/foo'
         request.method = 'PUT'
-        request.headers['expect'] = '100-Continue'
+        request.headers[header] = value
         credentials = botocore.credentials.Credentials('access_key',
                                                        'secret_key')
         auth = botocore.auth.S3SigV4Auth(credentials, 's3', 'us-east-1')
         auth.add_auth(request)
-        self.assertNotIn('expect', request.headers['Authorization'])
+        self.assertNotIn(header, request.headers['Authorization'])
+
+    def test_blacklist_expect_headers(self):
+        self._test_blacklist_header('expect', '100-continue')
+
+    def test_blacklist_headers(self):
+        self._test_blacklist_header('user-agent', 'botocore/1.4.11')
 
 
 class TestSigV4(unittest.TestCase):