POST to S3 Bucket Returning 403

I am writing a program that will POST a file to a website that is making me go through a few steps to get the file properly uploaded.  In the first step, I perform a simple POST which alerts the site that a file will be uploaded and returns a URL to an S3 bucket, along with some more response content.  The next step is to POST the file to the URL returned in step one, and this is where things break.  An abbreviated version of my code is as follows:

```
url = 'https://myurl.s3.amazonaws.com/'
payload = {all of the content returned from step one, minus the URL}
file = {'file': open('C:\...\file_name.imscc', 'rb')}
r = requests.post(url, data=payload, files=file)
```

At this point the code waits for about 15-20 seconds and then gives back a 403 error.  My initial thought was that the file was too large (about 18.58MB) so I tried to use the Multipart Encoder from the Requests Toolbelt.  When I ran this code:

```
m = MultipartEncoder(fileds={the same values as above, 'file': (files[0], open('C:\Users\jonathan.kulakofsky\Documents\Code\Downloads\\' + files[0], 'rb'))})
```

I got a Traceback Error that ended with `'Connection aborted.', error(10054, 'An existing connection was forcibly closed by the remote host')`.

As far as I can tell, I am passing the proper authentication key back to Amazon with the rest of the fields in the payload.  Does anyone have any ideas why I'm getting these errors?


I _suspect_ that Amazon is returning a 4XX status code and then getting mad when we keep sending the data.

Do you need to multipart encode this data, or do you just want to post only the bytes of the file?


I only need to post the file.  I only tried the multipart encoder because the requests documentation has a line about `requests.post` failing for larger files, but doesn't state what qualifies as "large."


Are you sure? You're definitely still posting the data, which suggests you do need the multipart encoding.

Is there any documentation for this API anywhere?


I am not sure, I am a fairly new developer.  Here is the documentation https://canvas.instructure.com/doc/api/file.file_uploads.html#method.file_uploads.post


The most interesting requirement here is that the file _must_ be the last element in the multipart form data. Otherwise, the streaming multipart encoder from the toolbelt is exactly the right thing to be using.

Try changing your `fields` argument from a dictionary to a list of two-tuples, ensuring that the file parameter comes last, and don't bother providing any other arguments to the file. That would mean, for example:

``` python
m = MultipartEncoder(fields=[
    ('key1', 'value1'),
    ('key2', 'value2'),
    ('file', open(<path>, 'rb'))
])
```

See if that works better. Don't forget to set the headers [as in the example](https://toolbelt.readthedocs.org/en/latest/uploading-data.html#streaming-multipart-data-encoder).


It's certainly taking longer to process now, but it's still giving me a 403.  Which has to be an update from the previous error 10054 I was getting when using the multipart encoder.


It's giving a 403? You're sure?


The next line in the program is `print r.status_code` and it says 403.


That's interesting @JKulakofsky. A forbidden means that you're not allowed to access that. Are you sure your constructing your request correctly? Is there a reason you can't show us exactly the code you're using?


Well, splitting a list along ':' is great until you hit the item in the list that is 'https://......'.  I was splitting up the success action redirect URL which was throwing off the rest of the post.  I fixed that issue and now I'm getting a 500 error.  So... progress?


@sigmavirus24 The main reason I was avoiding posting the whole code was because it's long and dirty, but here you go:

```
import requests
import os
import re
from requests_toolbelt.multipart.encoder import MultipartEncoder

etlmig = '*******'
auth = {'Authorization': 'Bearer ' + etlmig}

files = os.listdir('C:\Users\jkulakofsky\Documents\Code\Downloads')

parameters = {'name': files[0]}
r1 = requests.post('https://......', headers=auth, data=parameters)
print r1.status_code # This returns a 200
content1 = r1.content

contentlist1 = content1.split('{')
url1 = contentlist1[1]
url2 = url1.split(',')
url3 = url2[0]
url4 = 'ht.+/'
url5 = re.compile(url4)
url6 = re.findall(url5,url3)
url = url6[0]
contentlist2 = contentlist1[2].split('}')
contentlist3 = contentlist2[0].split(',')
aws = contentlist3[0].split(':')
aws1 = aws[0]
aws1 = aws1[1:-1]
aws2 = aws[1]
aws2 = aws2[1:-1]
fil = contentlist3[1].split(':')
fil1 = fil[0]
fil1 = fil1[1:-1]
fil2 = ''
key = contentlist3[2].split(':')
key1 = key[0]
key1 = key1[1:-1]
key2 = key[1]
key2 = key2[1:-1]
acl = contentlist3[3].split(':')
acl1 = acl[0]
acl1 = acl1[1:-1]
acl2 = acl[1]
acl2 = acl2[1:-1]
pol = contentlist3[4].split(':')
pol1 = pol[0]
pol1 = pol1[1:-1]
pol2 = pol[1]
pol2 = pol2[1:-1]
sig = contentlist3[5].split(':')
sig1 = sig[0]
sig1 = sig1[1:-1]
sig2 = sig[1]
sig2 = sig2[1:-1]
suc = contentlist3[6].split('"')
suc1 = suc[1]
suc2 = suc[3]

m = MultipartEncoder(fields=[
    (key1, key2),
    (acl1, acl2),
    (fil1, fil2),
    (aws1, aws2),
    (pol1, pol2),
    (sig1, sig2),
    (suc1, suc2),
    ('file', open('C:\Users\jkulakofsky\Documents\Code\Downloads\\' + files[0], 'rb'))
])
r2 = requests.post(url, data=m, headers={'Content-Type': m.content_type})
print r2.status_code # This returns a 500
```


It looks like (from the documentation) that you are receiving JSON back. Can you try using `r1.json()` and accessing that like a dictionary instead of trying to parse out the values by hand?


(Also at this point, I'm very certain this problem isn't in requests and this support request is better suited for [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).


OK, I'll move it over there.  Thanks for your help, moving the multipart encoder from a dictionary to a list at least solved one of the issues.  When I eventually get a solution working I'll post it here and close the thread.


I think when you start using `r1.json()` you'll be able to wrap this up quickly. StackOverflow will help you write this code though, which is not what we're here for. We're here to fix bugs in requests.


@JKulakofsky instead of using split across a url string, why not use the module urlparse (i think the relevant method is urlparse.urlsplit)?


@ueg1990 you're misunderstanding the problem.