Repository: Pylons/pyramid

Issue:
JSON is not text and has no charset (upcoming WebOb changes)
Due to changes in WebOb (https://github.com/Pylons/webob/pull/261) to fix content type/charset handling, various tests within Pyramid are failing because the JSON renderer returns `str` objects and then the `RenderHelper` attempts to set `Response.text` which fails if there is no `charset`.

All none-text types (so anything that does not match mime type `text/*`, `application/xml` or `*/*+xml`) no longer automatically have a charset set, this means they are treated as bytes objects only.

https://github.com/Pylons/pyramid/blob/master/pyramid/renderers.py#L275
https://github.com/Pylons/pyramid/blob/master/pyramid/renderers.py#L382

Should both have a `.encode('UTF-8')` added, to correctly encode it's rendering into a bytes object.

Extra protection could also be added here:

https://github.com/Pylons/pyramid/blob/master/pyramid/renderers.py#L468-476

It should verify that a `charset` is set, before attempting to set the `response.text`.

With the addition of the call to `encode()` in the JSON renderers, the test suite would have to be appropriately updated to use `b''` wherever there are string literals that are being compared, currently there are 12 tests that fail with the `encode()` fix in place.



Discussion:
If we add the two fixes proposed we can release a bugfix to 1.7 and 1.6 in preparation for people upgrading webob prior to pyramid.

As far as the "verify that a `charset` is set" I'm not clear what the end game is there. If it is not set do we just encode to utf-8?

@mmerickel I am going to implement the change in WebOb whereby there is a `default_encoding`. If no `charset` is set or there is no valid `charset` for the `Content-Type` then accessing `resp.text` will use `default_encoding` instead.

`default_encoding` is going to be set to `UTF-8` by default.

---

My comment regarding checking for a `charset`, if there is none, then the data is technically binary so if a renderer returns a text type, it should either raise, or one could just encode it to "utf-8" and call it good (but I wonder if that would allow for subtle errors).

Either way with the fix I am proposing in WebOb before I make the changes, that becomes entirely moot since `resp.text` will just work (albeit with an implicit encode using `default_encoding`, which in this case will be 'UTF-8').

@mmerickel 

I've added a `default_encoding` on `Response`, but it does have backwards incompatibility changes as a side-effect:

https://github.com/Pylons/webob/pull/261/commits/22bd28178d126946f86a4db6ba9e1917c7e5ad52#diff-bb3c4deeba77c2bbfd893f2e90fecc77R28

Would definitely like your input on whether this is a good idea, or if this should get fixed in Pyramid instead for 1.6/1.7.

Only reason I am attempting to fix it is due to Pyramid using `.text`, and I am not sure it is worth the backwards incompatible change. Would like your feedback.


Resolving patch:
diff --git a/pyramid/renderers.py b/pyramid/renderers.py
--- a/pyramid/renderers.py
+++ b/pyramid/renderers.py
@@ -1,6 +1,7 @@
 import json
 import os
 import re
+import warnings
 
 from zope.interface import (
     implementer,
@@ -272,7 +273,7 @@ def _render(value, system):
                 if ct == response.default_content_type:
                     response.content_type = 'application/json'
             default = self._make_default(request)
-            return self.serializer(value, default=default, **self.kw)
+            return self.serializer(value, default=default, **self.kw).encode('UTF-8')
 
         return _render
 
@@ -379,7 +380,7 @@ def _render(value, system):
                         raise HTTPBadRequest('Invalid JSONP callback function name.')
 
                     ct = 'application/javascript'
-                    body = '/**/{0}({1});'.format(callback, val)
+                    body = '/**/{0}({1});'.format(callback, val).encode('UTF-8')
                 response = request.response
                 if response.content_type == response.default_content_type:
                     response.content_type = ct
@@ -467,7 +468,17 @@ def _make_response(self, result, request):
 
         if result is not None:
             if isinstance(result, text_type):
-                response.text = result
+                if response.charset is None:
+                    warnings.warn(
+                        "Renderer returned a result of type {0}, "
+                        "however the response Content-Type <{1}> does not "
+                        "have a charset. Implicitly encoding the result as "
+                        "UTF-8.".format(type(result), response.content_type),
+                        RuntimeWarning
+                    )
+                    response.body = result.encode('UTF-8')
+                else:
+                    response.text = result
             elif isinstance(result, bytes):
                 response.body = result
             elif hasattr(result, '__iter__'):


Test patch:
diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py
--- a/pyramid/tests/test_config/test_views.py
+++ b/pyramid/tests/test_config/test_views.py
@@ -2168,7 +2168,7 @@ def test_add_notfound_view_with_renderer(self):
                                      ctx_iface=implementedBy(HTTPNotFound),
                                      request_iface=IRequest)
         result = view(None, request)
-        self._assertBody(result, '{}')
+        self._assertBody(result, b'{}')
 
     def test_add_forbidden_view_with_renderer(self):
         from zope.interface import implementedBy
@@ -2185,7 +2185,7 @@ def test_add_forbidden_view_with_renderer(self):
                                      ctx_iface=implementedBy(HTTPForbidden),
                                      request_iface=IRequest)
         result = view(None, request)
-        self._assertBody(result, '{}')
+        self._assertBody(result, b'{}')
 
     def test_set_view_mapper(self):
         from pyramid.interfaces import IViewMapperFactory
diff --git a/pyramid/tests/test_renderers.py b/pyramid/tests/test_renderers.py
--- a/pyramid/tests/test_renderers.py
+++ b/pyramid/tests/test_renderers.py
@@ -18,7 +18,7 @@ def _makeOne(self, **kw):
     def test_it(self):
         renderer = self._makeOne()(None)
         result = renderer({'a':1}, {})
-        self.assertEqual(result, '{"a": 1}')
+        self.assertEqual(result, b'{"a": 1}')
 
     def test_with_request_content_type_notset(self):
         request = testing.DummyRequest()
@@ -43,7 +43,7 @@ def adapter(obj, req):
         renderer = self._makeOne()
         renderer.add_adapter(datetime, adapter)
         result = renderer(None)({'a':now}, {'request':request})
-        self.assertEqual(result, '{"a": "%s"}' % now.isoformat())
+        self.assertEqual(result, '{{"a": "{0}"}}'.format(now.isoformat()).encode('UTF-8'))
 
     def test_with_custom_adapter2(self):
         request = testing.DummyRequest()
@@ -54,7 +54,7 @@ def adapter(obj, req):
         now = datetime.utcnow()
         renderer = self._makeOne(adapters=((datetime, adapter),))
         result = renderer(None)({'a':now}, {'request':request})
-        self.assertEqual(result, '{"a": "%s"}' % now.isoformat())
+        self.assertEqual(result, '{{"a": "{0}"}}'.format(now.isoformat()).encode('UTF-8'))
 
     def test_with_custom_serializer(self):
         class Serializer(object):
@@ -66,7 +66,7 @@ def __call__(self, obj, **kw):
         renderer = self._makeOne(serializer=serializer, baz=5)
         obj = {'a':'b'}
         result = renderer(None)(obj, {})
-        self.assertEqual(result, 'foo')
+        self.assertEqual(result, b'foo')
         self.assertEqual(serializer.obj, obj)
         self.assertEqual(serializer.kw['baz'], 5)
         self.assertTrue('default' in serializer.kw)
@@ -84,7 +84,7 @@ def __json__(self, req):
         objects = [MyObject(1), MyObject(2)]
         renderer = self._makeOne()(None)
         result = renderer(objects, {'request':request})
-        self.assertEqual(result, '[{"x": 1}, {"x": 2}]')
+        self.assertEqual(result, b'[{"x": 1}, {"x": 2}]')
 
     def test_with_object_adapter_no___json__(self):
         class MyObject(object):
@@ -290,6 +290,19 @@ def test__make_response_result_is_str(self):
         response = helper._make_response(la.encode('utf-8'), request)
         self.assertEqual(response.body, la.encode('utf-8'))
 
+    def test__make_response_result_is_str_no_charset(self):
+        from pyramid.response import Response
+        request = testing.DummyRequest()
+        request.response = Response(content_type='application/json', charset=None)
+
+        self.assertIsNone(request.response.charset)
+
+        helper = self._makeOne('loo.foo')
+        la = text_('/La Pe\xc3\xb1a', 'utf-8')
+        response = helper._make_response(la, request)
+        self.assertIsNone(response.charset)
+        self.assertEqual(response.body, la.encode('utf-8'))
+
     def test__make_response_result_is_iterable(self):
         from pyramid.response import Response
         request = testing.DummyRequest()
@@ -492,7 +505,7 @@ def test_response_preserved(self):
         request.response = response
         # use a json renderer, which will mutate the response
         result = self._callFUT('json', dict(a=1), request=request)
-        self.assertEqual(result, '{"a": 1}')
+        self.assertEqual(result, b'{"a": 1}')
         self.assertEqual(request.response, response)
 
     def test_no_response_to_preserve(self):
@@ -507,7 +520,7 @@ def response(self):
         request = DummyRequestWithClassResponse()
         # use a json renderer, which will mutate the response
         result = self._callFUT('json', dict(a=1), request=request)
-        self.assertEqual(result, '{"a": 1}')
+        self.assertEqual(result, b'{"a": 1}')
         self.assertFalse('response' in request.__dict__)
 
 class Test_render_to_response(unittest.TestCase):
@@ -627,7 +640,7 @@ def test_render_to_jsonp(self):
         request = testing.DummyRequest()
         request.GET['callback'] = 'callback'
         result = renderer({'a':'1'}, {'request':request})
-        self.assertEqual(result, '/**/callback({"a": "1"});')
+        self.assertEqual(result, b'/**/callback({"a": "1"});')
         self.assertEqual(request.response.content_type,
                          'application/javascript')
 
@@ -637,7 +650,7 @@ def test_render_to_jsonp_with_dot(self):
         request = testing.DummyRequest()
         request.GET['callback'] = 'angular.callbacks._0'
         result = renderer({'a':'1'}, {'request':request})
-        self.assertEqual(result, '/**/angular.callbacks._0({"a": "1"});')
+        self.assertEqual(result, b'/**/angular.callbacks._0({"a": "1"});')
         self.assertEqual(request.response.content_type,
                          'application/javascript')