Repository: stephenmcd/mezzanine

Issue:
UnicodeDecodeError ('ascii' codec can't decode) when memcached enabled
I can reproduce this on Ubuntu 12 and CentOS 6.   

memcached installed and enabled.  

Install new virtualenv and activate
~/venv/workingvirtualenv
source ~/venv/workingvirtulaenv/bin/activate

pip install mezzanine
pip install python-memcached
mezzanine-project testproject

add the live_settings.py cache settings to local_settings.py

Now create the DB
python manage.py createdb

Select all defaults and answer yes to questions including creating demo data..
Can browse demo pages including /gallery with no errors and caching appears to be working.

Now create and activate a new virtualenv and install everything exactly the same only this time install mezzanine from github.
git clone https://github.com/stephenmcd/mezzanine.git
cd mezzanine
python setup.py install

Now when you try browse the pages they will work until you go to /gallery/ and you will get the following error.

The error is:
"UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 31: ordinal not in range(128)"

If I install the DB with caching not configured in local_settings.py I can browse to /gallery with no error.  Then when I try enable caching and browse to /gallery I get the error again.  I also tried recreating the DB in the new virtualenv with the newer version of mezzanine.  The problem still occurs.

I can reproduce all this when I use MySQL instead of SQLite as well.  Also occurs on blogposts.  My live site is using this github version of Mezzanine because my theme uses bootstrap3.  Just want to get caching working now.



Discussion:
I believe the locale is not being set properly. The clue is that the error includes "'ascii' codec", but the images in the the gallery require UTF-8. Take a look at https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/fabfile.py#L341, in which the Fabric script installs the base system including memcached, for one idea of how to fix the locale.

I don't believe this is related to the locale setting on the http server.  I can recreate this just using the built in development http server.  It also occurs on Apache webserver with the proper locale environment variables set.  Django assumes utf-8 by default but just to be sure I also explicity set  DEFAULT_CHARSET = 'utf-8' and as expected it didn't make any difference.

As I said everything works fine when I install mezzanine from pip with everything else exactly the same.  The problem only occurs when I install mezzanine from github master branch and only when memcached is enabled so it must be a bug that was recently introduced.

Also not a problem with .bashrc

$locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

Just to clarify again.  It must be a recently introduced bug with mezzanine because it works when I pip install mezzanine but not when I install from master branch on github with everything else remaining exactly the same.

I can reproduce this on mutiple servers running Ubuntu and CentOS so I am pretty sure I have eliminated every other possibility.

Looking into this now; I suspect it has to do with the string handling for
Python 3 support in the current master.

On Sun, Dec 8, 2013 at 6:35 AM, shadowym notifications@github.com wrote:

> I am aware of that setting. I don't believe this is related. As per my
> explanation, I can reproduce this from command line when creating the DB.
> So no http server is involved at all.
> 
> As I said everything works fine when I install mezzanine from pip. The
> problem only occurs when I install mezzanine from github master branch and
> only when memcached is enabled so it must be a bug that was recently
> introduced.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/stephenmcd/mezzanine/issues/843#issuecomment-30067072
> .

Awesome thanks Alex! I had the same hunch.

Gotta head out for a while, but `nevercache_token()` is returning a unicode string and trying to split `response.content` with it. Then I think Python tries to decode response.content (using the 'ascii' codec) in order to split it, and runs into some non-ASCII characters in the titles of the gallery images (in this case Ávila).

Should be able to fix it by making nevercache_token always return a native string so there's no encode/decode - but I think there's something else weird going on too, because the byte sequence `A\xc3\x8c\xc2\x81vila` appears in response.content...which seems wrong, for a single combining diacritic.

I'll dig some more tonight.

This is at https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/core/middleware.py#L152


Resolving patch:
diff --git a/mezzanine/core/middleware.py b/mezzanine/core/middleware.py
--- a/mezzanine/core/middleware.py
+++ b/mezzanine/core/middleware.py
@@ -149,7 +149,12 @@ def process_response(self, request, response):
         # content. Split on the delimiter the ``nevercache`` tag
         # wrapped its contents in, and render only the content
         # enclosed by it, to avoid possible template code injection.
-        parts = response.content.split(nevercache_token())
+        token = nevercache_token()
+        try:
+            token = token.encode('utf-8')
+        except AttributeError:
+            pass
+        parts = response.content.split(token)
         content_type = response.get("content-type", "")
         if content_type.startswith("text") and len(parts) > 1:
             # Restore csrf token from cookie - check the response
@@ -169,7 +174,7 @@ def process_response(self, request, response):
                 if i % 2:
                     part = Template(part).render(context).encode("utf-8")
                 parts[i] = part
-            response.content = "".join(parts)
+            response.content = b"".join(parts)
             response["Content-Length"] = len(response.content)
             if hasattr(request, '_messages'):
                 # Required to clear out user messages.


Test patch: