bugs on HTTP-request nested json.

I think, there's a bug on serializing a nested json.
for example, the following payload
{'username': 'bambang', 'device_info': {'os_version': '9.3.1', 'model': 'samsung', 'os_name': 'android'}}

would be serialized as:
"username=bambang&device_info=os_version&device_info=model&device_info=os_name"

however, sending the same json via ajax-javascript, would be serialized as:
username=bambang&device_info%5Bos_version%5D=9.3.1&device_info%5Bmodel%5D=samsung&device_info%5Bos_name%5D=android


I think you're a little confused about what JSON is. You cannot 'serialize' JSON. JSON is a serialization format, it's something you serialize _to_. What you are serializing there is a dictionary.

The second element of your confusion is that you aren't even serializing _to_ JSON. You're serializing to a format known as `urlencoded` or `form-encoded`. This form of data serialization does not have any inherent specified form for nesting, all that it has is convention.

There are a couple of problems here.

Firstly, we're not handling the nesting well. That _might_ be a bug, but as I stated earlier, form-encoding doesn't really _do_ nested in any consistent way.

Secondly, if we passed it straight through to the standard library you still wouldn't get the result you wanted:

``` python
>>> from urllib import urlencode
>>> urlencode({'username': 'bambang', 'device_info': {'os_version': '9.3.1', 'model': 'samsung', 'os_name': 'android'}})
'username=bambang&device_info=%7B%27os_version%27%3A+%279.3.1%27%2C+%27model%27%3A+%27samsung%27%2C+%27os_name%27%3A+%27android%27%7D'
```

What this fundamentally boils down to is that it's not clearly an area that requests can help you with. One approach that might be 'right' is someone else's definition of 'wrong'. What you should do is specify the form you want yourself. For example, you should transform the above dictionary to:

``` python
{'username': 'bambang', 'device_info[os_version]': '9.3.1', 'device_info[model]': 'samsung', 'device_info[os_name]': 'android'}
```

That would get you the function you want.

@sigmavirus24, @kennethreitz, do you think we should be trying to handle the nesting ourselves? I'd be interested to see if there's actually a consistent convention here, even though form-data was never intended to do this.


form-data was never intended to work this way, but sadly it does. This is not something requests should handle because there's no _defined_ or _standard_ way to do it. This can be handled by the toolbelt, and I'll be happy to handle the controversy there. That said, I can't count how many times this has come up on this project. That combined with the order of magnitude more times it has come up on StackOverflow is incredible. It is _not_ hard for users to search for this and find exactly what they're looking for in 15 minutes or (much much) less (if they don't confuse terminology).


@Lukasa, @sigmavirus24. The 'json serialization' ramadokayano is describing is exactly what jQuery $.ajax methods produce. It's been a while since I used prototype.js but IIRC they did this too. angularjs does nested form-data encoding with []. I'm not sure what ember or meteor do.

There aren't native browser methods for this other than mozilla's https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/querystring (which is unstable and not available in the latest firefox) so I've no idea what that does.

PHP has been doing the nested object serialization and de serialization in exactly the way the same way as jQuery since like forever.

```
<?php

$something = json_decode(
    "{'username': 'bambang', 'device_info': {'os_version': '9.3.1', 'model': 'samsung', 'os_name': 'android'}}"
    , true
);

// username=bambang&device_info%5Bos_version%5D=9.3.1&device_info%5Bmodel%5D=samsung&device_info%5Bos_name%5D=android
echo http_build_query($something);
```

The node eco system isn't 100% consistent but all main modules and libs do the encoding with [].

```
var params = {'username': 'bambang', 'device_info': {'os_version': '9.3.1', 'model': 'samsung', 'os_name': 'android'}};

// working though the node modules with this functionality in order of decending popularity
// see, https://nodejsmodules.org/tags/querystring

// QS - 55.8%
var Qs = require('qs');
// username=bambang&device_info%5Bos_version%5D=9.3.1&device_info%5Bmodel%5D=samsung&device_info%5Bos_name%5D=android
Qs.stringify(params);

// form2json - 6.8%
// Is getting very old at this point and isn't under active development. It does object decoding only.
// Uses object properties separated by '.'

// querystring - 5.8%
querystring = require('querystring');
// username=bambang&device_info=
// which I'm led to believe is intentional - https://github.com/joyent/node/issues/1665
querystring.stringify(params)

// perry - 1.3%
var Perry = require('perry');
// username=bambang&device_info[os_version]=9.3.1&device_info[model]=samsung&device_info[os_name]=android
Perry.stringify(params);

// connect-queryparser -- 0.1 but was used in a lot of frameworks. Now depreciated.
// only did object decoding. When passed the output of  QS, Perry or jQuery is would return params
```

I'm not sure why nodejs' native querystring lacks support for any composite objects but all the main frameworks (restify, express, connect-middleware, ...) do nested objects with []. I suppose you trade the not being able to put unescaped literal `[` in your object property names (who does this anyway?) without escaping vs allowing nested objects with keys go through a querystring-ing and unquerystring-ing progress unscathed which seems like a excellent trade off to make to me.

For good or bad, jQuery has now utterly dominated in the browser. I think it's fair to say with all the major players doing nested object serialisation in browser with [] it's basically a defacto standard at this point.


> I'm not sure why nodejs' native querystring lacks support for any composite objects

I know why! It's not the _actual_ standard so there's no reason to support it, just like the Python standard library doesn't support it for the _exact same reason_.

> I think it's fair to say with all the major players doing nested object serialisation in browser with [] it's basically a defacto standard at this point.

I think you meant it's the defacto standard for how jQuery and JS frameworks work in the browser. I can't find an example of a browser that would construct something like this without form input ids being pre-formatted to this structure.

If you think Python as a whole should support this, I suggest you either open a new issue at bugs.python.org (preferably after searching for already open/closed bugs that match the request). Until Python supports it, the toolbelt will support it but I'm pretty confident this doesn't belong in requests-core.


> It's not the actual standard

I agree. There isn't a official standard to follow.

> there no reason to support it

Of course python isn't doing anything contrary to a official standard because, well, there isn't one but I don't think you can go so far as to say there's _no_ reason to support it. People writing PHP, ruby, node, .NET and stuff in browsers have seemingly all come to see this way of encoding as a defacto standard. Which ever way you cut it python is clearly the odd-one-out in this not-small group and it doesn't look like this is going to change. This seems like a bad place for python to be. You said yourself "I can't count how many times this has come up on this project"

> I can't find an example of a browser that would construct something like this

That's right. To do something like this in a browsers you need javascript and the native javascript api doesn't cover this _at all_ so there's no standard to follow other than the frameworks which are now near universal. I imagine the OP is comparing python's url query string builder to one the output of pretty much any of these.

I think there are good reasons to not do any of this is requests e.g. this belongs in python core, it breaks backward compatibility,  ... but at this point I think there-is-no-standard argument doesn't hold up.

> preferably after searching for already open/closed bugs that match the request

Touché


While I understand both sides of this debate... I do recall Urllib handles this fine as nested Json structure per contract binding was common place in prior jobs.  


@DavidHwu 
1. We're not talking about JSON. This isn't JSON. I'll repeat this until I'm blue in the face.
2. If urllib behaved as @squareproton suggests already, requests would behave the same way since we're almost certainly using the same libraries.