Repository: web2py/web2py

Issue:
Invalid HTTP header values on some locales
I'm currently testing web2py using Python3, uWSGI + nginx and a Python module for uWSGI server.

I've discovered an interesting bug, in which my server returns blank pages every Thursday. Looking at uWSGI error logs revealed this message:

>   Traceback (most recent call last):
  File "/srv/www/web2py/gluon/main.py", line 565, in wsgibase
    return http_response.to(responder, env=env)
  File "/srv/www/web2py/gluon/http.py", line 123, in to
    responder(status, rheaders)
TypeError: http header must be encodable in latin1

Raising value of `rheaders` I found that it contains this key-value pair

> ('Expires', 'Čt, 30 srp 2018 11:18:00 GMT')

As you can see, there's a `Č` character, that is not latin1 encodable. Finding a line that generates this key revealed this code:

https://github.com/web2py/web2py/blob/63972386c23961360d99c5ce49240b0aed5c820f/gluon/main.py#L202-L203

`%a` - Weekday as locale’s abbreviated name.
`%b` - Month as locale’s abbreviated name.

Now, my server runs on "cs_CZ.UTF-8" locale, so `%a` generates `Čt` every thursday. That value was fine while running web2py on Python2, but with Python3, the uWSGI module converts every value from unicode to latin1, as seen in https://github.com/unbit/uwsgi/blob/d960f8fdc36b33c60b7291ca8e32dbb5df22cf15/plugins/python/wsgi_headers.c#L140

Converting to latin1 is correct, as from RFC 7230:

> Historically, HTTP has allowed field content with text in the
   ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
   through use of [RFC2047] encoding.  In practice, most HTTP header
   field values use only a subset of the US-ASCII charset [USASCII].
   Newly defined header fields SHOULD limit their field values to
   US-ASCII octets.  A recipient SHOULD treat other octets in field
   content (obs-text) as opaque data.

Possible fix would be to temporary use "en_US.UTF-8" encoding independent on locale.

EDIT: Seems like file streaming is affected by this bug as well, as seen in the next lines of code:

https://github.com/web2py/web2py/blob/63972386c23961360d99c5ce49240b0aed5c820f/gluon/streamer.py#L77-L79

EDIT2: Digging deeper into RFCs, I've found that the `Expires` header is defined in [RFC 7234, section 5.3](https://tools.ietf.org/html/rfc7234#section-5.3). In here, format is defined as `Expires = HTTP-date`. `HTTP-date`'s format is defined in [RFC 7231, section 7.1.1.1](https://tools.ietf.org/html/rfc7231#section-7.1.1.1). The format has predefined values for dates and uses english language, not local-specific words.


Discussion:

Resolving patch:
diff --git a/gluon/main.py b/gluon/main.py
--- a/gluon/main.py
+++ b/gluon/main.py
@@ -31,7 +31,7 @@
 
 from gluon.fileutils import abspath, write_file
 from gluon.settings import global_settings
-from gluon.utils import web2py_uuid
+from gluon.utils import web2py_uuid, unlocalised_http_header_date
 from gluon.admin import add_path_first, create_missing_folders, create_missing_app_folders
 from gluon.globals import current
 
@@ -199,8 +199,7 @@ def serve_controller(request, response, session):
         ('Content-Type', contenttype('.' + request.extension)),
         ('Cache-Control',
          'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'),
-        ('Expires', time.strftime('%a, %d %b %Y %H:%M:%S GMT',
-                                  time.gmtime())),
+        ('Expires', unlocalised_http_header_date(time.gmtime())),
         ('Pragma', 'no-cache')]
     for key, value in default_headers:
         response.headers.setdefault(key, value)
diff --git a/gluon/streamer.py b/gluon/streamer.py
--- a/gluon/streamer.py
+++ b/gluon/streamer.py
@@ -16,6 +16,7 @@
 import re
 import errno
 from gluon.http import HTTP
+from gluon.utils import unlocalised_http_header_date
 from gluon.contenttype import contenttype
 from gluon._compat import PY2
 
@@ -74,7 +75,7 @@ def stream_file_or_304_or_206(
     stat_file = os.stat(static_file)
     fsize = stat_file[stat.ST_SIZE]
     modified = stat_file[stat.ST_MTIME]
-    mtime = time.strftime('%a, %d %b %Y %H:%M:%S GMT', time.gmtime(modified))
+    mtime = unlocalised_http_header_date(time.gmtime(modified))
     headers.setdefault('Content-Type', contenttype(static_file))
     headers.setdefault('Last-Modified', mtime)
     headers.setdefault('Pragma', 'cache')
diff --git a/gluon/utils.py b/gluon/utils.py
--- a/gluon/utils.py
+++ b/gluon/utils.py
@@ -462,3 +462,51 @@ def local_html_escape(data, quote=False):
             data = data.replace(b'"', b"&quot;")
             data = data.replace(b'\'', b"&#x27;")
         return data
+
+
+def unlocalised_http_header_date(data):
+    """
+    Converts input datetime to format defined by RFC 7231, section 7.1.1.1
+
+    Previously, %a and %b formats were used for weekday and month names, but
+    those are not locale-safe. uWSGI requires latin1-encodable headers and
+    for example in cs_CS locale, fourth day in week is not encodable in latin1,
+    as it's "Čt".
+
+    Example output: Sun, 06 Nov 1994 08:49:37 GMT
+    """
+
+    short_weekday = {
+        "0": "Sun",
+        "1": "Mon",
+        "2": "Tue",
+        "3": "Wed",
+        "4": "Thu",
+        "5": "Fri",
+        "6": "Sat",
+    }.get(time.strftime("%w", data))
+
+    day_of_month = time.strftime("%d", data)
+
+    short_month = {
+        "01": "Jan",
+        "02": "Feb",
+        "03": "Mar",
+        "04": "Apr",
+        "05": "May",
+        "06": "Jun",
+        "07": "Jul",
+        "08": "Aug",
+        "09": "Sep",
+        "10": "Oct",
+        "11": "Nov",
+        "12": "Dec",
+    }.get(time.strftime("%m", data))
+
+    year_and_time = time.strftime("%Y %H:%M:%S GMT")
+
+    return "{}, {} {} {}".format(
+        short_weekday,
+        day_of_month,
+        short_month,
+        year_and_time)


Test patch: