Repository: web2py/web2py

Issue:
parameter of ajax put request is empty in Python 3
I tested the latest version 2.15.2 and I've got a problem when accessing ajax post parameters with web2py under Python 3.

The client sends a PUT request using jquery like this:
`$.ajax('<url>', {
    data: JSON.stringify({
        outputFormat: 'pdf'
    }),
    type: "PUT", contentType: "application/json",
    success: function(data) {
        alert('success');
    },
    error: function(jqXHR, textStatus, errorThrown) {
        alert('error');
    }
});`

in the controller I access the parameter
`output_format = request.vars.outputFormat`

when using Python 2.7 the variable output_format is set to 'pdf' as expected. When using Python 3.5 the variable is None.


Discussion:
I can't reproduce this. For me it works perfectly fine. So either this was already fixed or you have a typo in your actual code for request.vars.outputFormat  (e.g. Maybe you have request.vars.outputForma)

Can you print(request.vars) in the controller function this ajax is calling and tell me what it shows?
request.vars is an empty Storage object. There can't be a typo because I'm running the same code with Python 2.7 and Python 3.5. It works fine when using Python 2.7
@alhman What I did was.
1 - Go to admin, create new basic application
2 - in its default.py I added this function
```python
    def jsonvars():
        print(request.vars)
        output_format = request.vars.outputFormat
        print(output_format)
        return ''
```
3 - to the bottom of views/default/index.html I added

```html
<script>
$.ajax("{{=URL('default', 'jsonvars')}}",
    {
      data: JSON.stringify({ outputFormat: 'pdf' }), 
      type: "PUT", 
      contentType: "application/json", 
      success: function(data) { alert('success'); }, 
      error: function(jqXHR, textStatus, errorThrown) { alert('error ' + textStatus + ' : ' + errorThrown); } 
    }
);
</script>
```
4 - visit the new application default/index

I got an alert saying "success" and in the termina I ran web2py with python 3 was
```
<Storage {'outputFormat': 'pdf'}>
pdf
```

Which was exactly what I expected.

Can we close this?
@mdipierro not sure, I really can't reproduce it. Maybe @alhman can provide an app where we see it happening.
I created a new minimalistic test application and the behavior is still the same. It works fine with Python 2 but not with Python 3.5. I even tested it with different browsers (Opera and FF) and jquery versions (3.1.1 and 3.2.1). If you can't reproduce it with this app then I'll test it on another machine (Linux instead of Windows).

[test.zip](https://github.com/web2py/web2py/files/1204843/test.zip)

@alhman just tested that in microsoft windows python 3.6.1 and it printed the result just fine.  
  
My only guess right now is that your web2py code is not right somehow, did you download the latest source version? Or did you update your web2py via admin? If the latter please download the web2py source version and try again.
I used the official version 2.15.2.  Now I tried it with last tagged version from github (R-2.16.0b1) and with master branch. Always the same result. I'll make another test tomorrow on different machine to see what happens there.
I tested it on a completely different machine (with Ubuntu, Python 3.5 and web2py 2.15.3) and I get exactly the same behavior. Works fine with Python 2.7 but not with 3.5
do you know where exactly in the code the parameters should be set? (file/method)
I could debug the code and see what's going on and why the parameters are not set with Py3
@alhman yes, they're set in Request.parse_post_vars
https://github.com/web2py/web2py/blob/master/gluon/globals.py#L211
json_vars = json_parser.load(body)
raises an exception in Py3. the exception is thrown in loads function in python json module:
```python
if not isinstance(s, str):
    raise TypeError('the JSON object must be str, not {!r}'.format(
                            s.__class__.__name__))
```

the parameter 's' is a byte sequence and not a string:
b'{"outputFormat":"pdf"}'
@alhman I suspected as much, I just don't understand why it's not happening to me.
@alhman since I can't reproduce it, can you try something for me? in gluon/globals.py in function copystream_progress, can you replace all `return dest` with `return to_native(dest)`  
I tested on another machine (win10 & Python 3.6) and there it works. I found out that the difference is the Python version. It works with 3.6 but not with 3.5.

in 3.5 as shown above:
```python
if not isinstance(s, str):
    raise TypeError('the JSON object must be str, not {!r}'.format(
                            s.__class__.__name__))
```

and in 3.6:
```python
    if isinstance(s, str):
        if s.startswith('\ufeff'):
            raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                  s, 0)
    else:
        if not isinstance(s, (bytes, bytearray)):
            raise TypeError('the JSON object must be str, bytes or bytearray, '
                            'not {!r}'.format(s.__class__.__name__))
        s = s.decode(detect_encoding(s), 'surrogatepass')
```

so Python 3.6 also accepts bytes and bytearray for loading json while 3.5 only accepts str.
Do we really care about python 3.5?
I'd hope so. python 3.6 was released end of 2016, so I doubt that all Python 3 users are already using 3.6

I couldn't install python 3.6 on my Win7 computer so I had to use 3.5
@alhman would you run your code on both py27 and py35 and check if it works with the following change?
Replace line https://github.com/web2py/web2py/blob/5e6c3dba812f2c82d288fe93c5bfe32b6cc624cd/gluon/globals.py#L223

with
```
     json_vars = json_parser.load(to_native(body))
```
this fails in both versions because body is a file descriptor.
I replaced it with
`json_vars = json_parser.loads(to_native(body.read()))`
which works in 2.7 and 3.5
Body is a `<class 'tempfile._TemporaryFileWrapper'>` and currently contains bytes. As a result, in py3 you get the error reported.
Can't install 3.6?  Just update,  or use Anaconda.
what should I update? I'd have to do a complete new setup of my system and install Win10 (ugh) instead of the old (but still working) Win7. and what is Anaconda?

As I mentioned above the issue could be fixed with a one-liner so I don't understand why it is not resolved already. If you're not interested in Python 3.5 support then you should mark this issue as "wont fix" and further mention on the homepage and in the docs that you are only supporting Python >= 3.6

Resolving patch:
diff --git a/gluon/globals.py b/gluon/globals.py
--- a/gluon/globals.py
+++ b/gluon/globals.py
@@ -220,7 +220,12 @@ def parse_post_vars(self):
 
         if is_json:
             try:
-                json_vars = json_parser.load(body)
+                # In Python 3 versions prior to 3.6 load doesn't accept bytes and
+                # bytearray, so we read the body convert to native and use loads
+                # instead of load.
+                # This line can be simplified to json_vars = json_parser.load(body)
+                # if and when we drop support for python versions under 3.6
+                json_vars = json_parser.loads(to_native(body.read())) 
             except:
                 # incoherent request bodies can still be parsed "ad-hoc"
                 json_vars = {}


Test patch: