POST with json data and oauth auth_header signature_type not working

The following code does not generate an Authorization header.

``` python
import requests
from requests.auth import OAuth1
import json


client_key = u'mykey'
client_secret = u'mysecret'

headeroauth = OAuth1(
    client_key,
    client_secret,
    None,
    None,
    signature_type='auth_header'
)

payload = {
    "type": "command_line",
    "params":  {
        "command": "sleep 0"
    }
}

url = u'http://localhost:8003/test'
data = json.dumps(payload, sort_keys=True, indent=4)

r = requests.post(url, auth=headeroauth, data=data )
```

If I replace 

``` python
data=data
```

by

``` python
data={'data':'payload'}
```

I get and Authorization header but that's not what I need...

Am I missing anything?  This seem to be broken.


Nowhere in requests does `data` mean `json data` (at least from reading the code). If it's in the documentation then that needs to be corrected, but there is no section I know of where what you expect is promised.

Granted, I would support a separate parameter `json` that would guarantee encoding of the data as json data, but that is likely outside of the scope of requests.


Correct.

Sent from my iPhone

On Oct 25, 2012, at 1:07 PM, Ian Cordasco notifications@github.com wrote:

> Nowhere in requests does data mean json data (at least from reading the code). If it's in the documentation then that needs to be corrected, but there is no section I know of where what you expect is promised.
> 
> Granted, I would support a separate parameter json that would guarantee encoding of the data as json data, but that is likely outside of the scope of requests.
> 
> —
> Reply to this email directly or view it on GitHub.


from the doc http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests

```
import json
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}

 r = requests.post(url, data=json.dumps(payload))
```

and http://docs.python-requests.org/en/latest/user/quickstart/#custom-headers it sounds like I should be able to pass json data, right?

even if I specify the content type it still doesn't produce an Authorization header.

```
headers = {'content-type': 'application/json'}
r = requests.post(url, auth=headeroauth, data={'data':'payload'}, headers=headers)
```

Sorry I'm not sure I understand clearly what you mean.  Are you saying that I can't post json data?
or that my request code is incorrect?

This is the actual code: https://gist.github.com/3955898

Thanks.


@alex-ethier you have to also look at the response provided from `'http://httpbin.org/post'` It interprets that as form data. From the same code block you referenced:

``` python
>>> print r.text
{
  // ...snip... //
  "form": {
    "key2": "value2",
    "key1": "value1"
  },
  // ...snip... //
}
```

Try it yourself with HTTPbin. You'll be able to see how the server interprets it yourself.

And I agree that the docs may seem to be misleading in this sense, but they do say:

> Typically, you want to send some form-encoded data — much like an HTML form. To do this, simply pass a dictionary to the data argument. Your dictionary of data will automatically be form-encoded when the request is made

Also right below the code block you reference, there's this:

> There are many times that you want to send data that is not form-encoded. If you pass in a string instead of a dict, that data will be posted directly.

And the example directly beneath it is for the GitHub API (which I have [experience](/sigmavirus24/github3.py) with) which accepts JSON requests.

---

And please delete your gist entirely. You have the client_{key,secret} there and that's a terrible idea.


@sigmavirus24 Thank you for your prompt reply.

I'm realizing that I wasn't clear enough about what I'm trying to accomplish.  Sorry about that.

We are a PaaS provider. I'm writing a client for the public api of our service.  ( I left the api key and secret in the gist since they are from a dev server running on my laptop. ) 

One of the call I have to make is a POST to http://localhost:8003/jobs.  This call requires a job parameters JSON data in the request body such as:

``` python
{
    "opType": "command_line",
    "params": {
        "command": "sleep 0",
        "callback": "localhost:8010",
        "name": "myJob"
    }
}
```

and also requires a signed oauth header.  Our api implement 2-legged oauth with header signature type.

I tried bunch of different variations:

---

if I pass a simple dictionary structure as data such as:

``` python
payload = {'some': 'data'}
...
r = requests.post(url, auth=headeroauth, data=payload)
```

requests generate the following request:

``` python
{'Content-Length': '9', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/0.14.1 CPython/2.7.1 Darwin/11.4.2', 'Host': 'localhost:8003', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'OAuth oauth_nonce="114662130218029821421351270615", oauth_timestamp="1351270615", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="9fac88f8645bdfbd6f8a68f60c5130d082244a8a8787674e82e85f68ff3716b8", oauth_signature="UNOYsMproYbF9t3mOrqkANRo%2Bqk%3D"'}
some=data
```

It includes an Authorization header, set the Content-Type to application/x-www-form-urlencoded and some data in the body as I would expect.

---

if I replace the simple dictionary structure above with something more complex, such as:

``` python
payload = {
    "opType": "command_line",
    "params": {
        "command": "sleep 0",
        "callback": "localhost:8010",
        "name": "myJob"
    }
}
...
r = requests.post(url, auth=headeroauth, data=payload)
```

requests generates a traceback:

```
  File "/Users/alex/.local/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/utils.py", line 52, in escape
    raise ValueError('Only unicode objects are escapable.')
ValueError: Only unicode objects are escapable.
```

---

if i pass JSON encoded string as data such as: 

``` python
payload = {
    "opType": "command_line",
    "params": {
        "command": "sleep 0",
        "callback": "localhost:8010",
        "name": "myJob"
    }
}
payload = json.dumps(payload)
...
r = requests.post(url, auth=headeroauth, data=payload)
```

requests generate the following request:

``` python
{'Host': 'localhost:8003', 'User-Agent': 'python-requests/0.14.1 CPython/2.7.1 Darwin/11.4.2', 'Content-Length': '107', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate, compress'}
{"opType": "command_line", "params": {"callback": "localhost:8010", "command": "sleep 0", "name": "myJob"}}
```

It contains the JSON data as string but no Authorization header and not content-type

---

if i pass JSON encoded string as data and specify the content type such as: 

``` python
payload = {
    "opType": "command_line",
    "params": {
        "command": "sleep 0",
        "callback": "localhost:8010",
        "name": "myJob"
    }
}
payload = json.dumps(payload)
headers = {'content-type': 'application/json'}
...
r = requests.post(url, auth=headeroauth, data=payload, headers=headers)
```

requests generate the following request:

``` python
{'Content-Length': '127', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/0.14.1 CPython/2.7.1 Darwin/11.4.2', 'Host': 'localhost:8003', 'Content-Type': 'application/json'}
"{\"opType\": \"command_line\", \"params\": {\"callback\": \"localhost:8010\", \"command\": \"sleep 0\", \"name\": \"myJob\"}}"
```

It contains JSON data, Content-Type but no Authorization header

---

So basically the client I'm writing has to be able to send json data and must contain an Authorization header.
Is this possible using the requests module? 

I have successfully built a ruby client that does what I need.  I like working with requests and really want to figure this out so I can do the same with requests.

Thank you for your time


Yeah, either the docs are wrong (which I just followed) or you and I are doing the same thing wrong. I made a mini flask app to print out the information I thought might be relevant using the following:

``` python
from flask import Flask, request, jsonify

app = Flask(__name__)


@app.route('/', methods=['POST'])
def index():
    resp = {
            'Headers': dict(request.headers),
            'Authorization': request.authorization,
            'Content-Type': request.content_type,
            'Data': request.data,
            'Form': request.form,
            'Files': request.files,
            }
    return jsonify(**resp)


if __name__ == '__main__':
    app.run()
```

And tested it from the interactive interpreter:

``` python
from requests import session
from requests.auth import OAuth1
from json import dumps


s = session()
s.auth = OAuth1('key', 'secret', None, None, 'auth_header')
local = 'http://127.0.0.1:5000/'
simple = {'some': 'data'}
complex = {'outer': 'layer', 'inner': simple}

r = s.post(local, data=simple)
# <Response [200]>
r.json
# Returns the response from the flask app with the relevant info being:
#  u'Headers': {u'Content-Length': u'48', u'Accept-Encoding': u'identity, deflate, compress, gzip', u'Accept': u'*/*', u'User-Agent': u'python-requests/0.14.1 CPython/2.6.2 Linux/2.6.30.3', u'Host': u'127.0.0.1:5000', u'Content-Type': u''}
r = s.post(local, data=complex)
# Same headers
r = s.post(local, data=dumps(simple))
r = s.post(local, data=dumps(complex))
```

None of them had any kind of Authorization headers. Also, the `authorization` part of the request in Flask was consistently empty.


Thank you @sigmavirus24.

@kennethreitz, Any thoughts on this? 

Thanks!
A


I'm working with Alex on this and made a fix that seems to be working for our purposes.  As far as I can tell it's not possible to get requests to generate an OAuth header if content-type is "application/json" without something like this fix.

in

``` python
class OAuth1(AuthBase):
```

I added a definition for CONTENT_TYPE_JSON, and changed from this:

``` python
        _oauth_signed = True
        if r.files and contenttype == CONTENT_TYPE_MULTI_PART:
            # Omit body data in the signing and since it will always
            # be empty (cant add paras to body if multipart) and we wish
            # to preserve body.
            r.url, r.headers, _ = self.client.sign(
                unicode(r.full_url), unicode(r.method), None, r.headers)
        elif decoded_body is not None and contenttype in (CONTENT_TYPE_FORM_URLENCODED, ''):
            # Normal signing
            if not contenttype:
                r.headers['Content-Type'] = CONTENT_TYPE_FORM_URLENCODED
            r.url, r.headers, r.data = self.client.sign(
                unicode(r.full_url), unicode(r.method), r.data, r.headers)
        else:
            _oauth_signed = False
```

to this:

``` python
        _oauth_signed = True
        if r.files and contenttype == CONTENT_TYPE_MULTI_PART:
            # Omit body data in the signing and since it will always
            # be empty (cant add paras to body if multipart) and we wish
            # to preserve body.
            r.url, r.headers, _ = self.client.sign(
                unicode(r.full_url), unicode(r.method), None, r.headers)
        elif decoded_body is not None and contenttype in (CONTENT_TYPE_FORM_URLENCODED, ''):
            # Normal signing
            if not contenttype:
                r.headers['Content-Type'] = CONTENT_TYPE_FORM_URLENCODED
            r.url, r.headers, r.data = self.client.sign(
                unicode(r.full_url), unicode(r.method), r.data, r.headers)
        elif contenttype == CONTENT_TYPE_JSON:
            r.url, r.headers, r.data = self.client.sign(
                unicode(r.full_url), unicode(r.method), r.data, r.headers)            
        else:
            _oauth_signed = False
```

Prettier: https://github.com/tower3/requests/commit/d6817551d0122cef6c86d5f5ca17ef9d475ce62e

It feels like a hack to special-case JSON, so I haven't made a pull request.


If you're making changes to oauthlib you have to contribute those to idan/oauthlib


You have to forgive me if I get stuff wrong.  I took 20 years off from writing software and just started back up a few months ago.

However, the change I made is in auth.py, which I'm pretty sure is part of requests.


Ah, no worries. I was reading that from the OctoDroid app. I wasn't sure where it was from, I was just making sure you were aware. And you got nothing wrong. Sorry if I came off confrontational or angry. That certainly wasn't my intent. Thanks for digging into the issue!


I seem to remember this having come up before, though I can't find a good example of this issue.

@idan, you're way more familiar with OAuth than me. Is the principle supposed to be that OAuth doesn't include the body for signing purposes, or is it just specifically not supposed to sign the body for `multipart/form-data`?


Per the [spec (3.4.1.3.1)](http://tools.ietf.org/html/rfc5849#section-3.4.1.3.1), OAuth1 only signs parameters from the request body if it is single-part, encoded according to `x-www-form-urlencoded`, and has the according `Content-Type` header set. If any of these three requirements are not met, the request body is ignored for the purposes of generating a signature, as per the spec.


If this isn't clear, it means that file uploads are never signed according to the OAuth1 spec.


Awesome. So, my understanding of this situation is that JSON data, while encoded correctly, does not have the correct Content-Type header, so we should not be including it in the signing. Correct?


The _only_ kind of data from the POST body which should be included in the signature base string (and thus signed) is form-urlencoded data, and even then only if the correct Content-Type header is set. JSON data is obviously not form-urlencoded, and specifying the Content-Type will not magically make it so. :)

So yes, JSON data should be (correctly) ignored for the purposes of signing. Put another way, if you include two requests that are otherwise identical but one of which contains JSON body data, the signature should not vary.


Thankyou @idan! I think this means we've got a bug in the OAuth code here. I'll fix it up. Do you mind if I notify you when I've done it so you can take a quick look?


Go for it. Open a ticket on idan/oauthlib :)  

On Saturday, November 24, 2012 at 10:30 PM, Cory Benfield wrote:

> Thankyou @idan (https://github.com/idan)! I think this means we've got a bug in the OAuth code here. I'll fix it up. Do you mind if I notify you when I've done it so you can take a quick look?
> 
> —
> Reply to this email directly or view it on GitHub (https://github.com/kennethreitz/requests/issues/910#issuecomment-10683343).  


Oh, no, OAuthlib is fine, it's the logic in Requests that's wrong. =)


Any updates on this? I think there were some changes to requests/requests-oauth recently that were important for 0.3. @frankslaughter and @alex-ethier any updates on this? 


I think I've just stumbled upon this with PUT requests to the ProjectPlace API.  That API expects the the auth to exclude the body. I was tempted to have a local copy and re-order the prepare function in models.py so that prepare_auth is done before prepare_body.  Unfortuantely, ignorance of how packaging works ("ValueError: Attempted relative import in non-package), is currently preventing me from trying this…

``` python
def prepare(self):
        """Constructs a :class:`PreparedRequest <PreparedRequest>` for transmission and returns it."""
        p = PreparedRequest()

        p.prepare_method(self.method)
        p.prepare_url(self.url, self.params)
        p.prepare_headers(self.headers)
        p.prepare_cookies(self.cookies)
        # Note that prepare_auth must be last to enable authentication schemes
        # such as OAuth to work on a fully prepared request.
        p.prepare_auth(self.auth)

        p.prepare_body(self.data, self.files)

        return p
```


@idan @Lukasa any opinions?


Having conquered my package ignorance - I was thinking too much - I've tried this change to the prepare function, and I'm able to successfully PUT to Project Place.  I know that this may not be appropriate for all uses, I don't really understand the protocol.

Note that the donkey work was done by Jon Nylander of ProjectPlace, who kindly sent me a sample script that prepares and signs the headers separately, before using requests to add the body and send (https://gist.github.com/pellepim/74fe738317e1789ccc38).  I had the same problem with python-oauth2 and refactored to use requests in the hope that it would solve my problem.   It was only by seeing Jon's working version that I could could figure out why requests wasn't working for me.


@daycoder: Goddamn I hate OAuth. Ok, so, `requests-oauthlib` _should_ correctly handle whether or not to include the body data in signing. What kind of data are you sending?


@Lukasa You don't know how happy you've made me by saying "Goddam I hate OAuth" :D

http://api.projectplace.com/index.php/DocumentUpload is the one that was giving me problems. The body content is just the contents of a file.

```
response = requests.get(…, data=open('A Test Report.xlsx','rb'))
```

(This works with my local copy `prepare` bodge).


Can you print the content of `response.request.headers` on one of these failed lookups please?

(I wrote a blog post entitled "OAuth Needs To Die In A Fire", so I have form when it comes to hating OAuth. =D )


This is what I'd posted to Jon at ProjectPlace earlier this week. Is that what you're after. Happy to reproduce and capture more info if it helps.

```
url: u'https://api.projectplace.com/1/document/864880393/upload.json'

headers: {u'Accept': u'*/*',
u'Accept-Encoding': u'gzip, deflate, compress',
'Authorization': 'OAuth oauth_nonce="47969000444228248131361304547",
oauth_timestamp="1361304547",
oauth_version="1.0",
oauth_signature_method="HMAC-SHA1",
oauth_consumer_key="dbeb09bfe9a88bfcc1fbf92ffffffff",
oauth_token="44a2beb972ca1598632c3081ffffffff", 
oauth_signature="tyzsjgreGxrml9VhV0GIaQu8ffffffff"',
u'Content-Length': u'28',
u'Content-Type': u'application/x-www-form-urlencoded',
u'User-Agent': u'python-requests/1.1.0 CPython/2.7.2 Darwin/12.2.0'}

body:u'{"body": "contents of file"}'
```


So, my concern is the `Content-Type` header. If you have `x-www-form-urlencoded` data, you are _supposed_ to include the data in the OAuth signature. If you use the 'data' header with a file, it _shouldn't_ set the data type to `x-www-form-urlencoded`, but it appears to be.

Humour a hunch of mine: can you run `oauthlib.common.extract_params()` on the file object and tell me what it returns?


oauthlib.common.extract_params(file('test.txt','rb'))
None

also, just to confirm I'm getting something back…

oauthlib.common.extract_params(u'File contents')
[(u'File contents', u'')]

oauthlib.common.extract_params({u'a':u'dictionary'})
[(u'a', u'dictionary')]


Right, so I bet my left arm that the issue is [L47 of requests-oauthlib.core](https://github.com/requests/requests-oauthlib/blob/master/requests_oauthlib/core.py#L47). I don't think we can use `extract_params()` as a good test of form-urlencoding.