Repository: web2py/web2py

Issue:
python3: an error ticket shows UnicodeDecodeError if starting web2py with --nogui
python 3.5, web2py version 2.16.1-stable+timestamp.2017.11.14.05.54.25

An error ticket shows always UnicodeDecodeError:
```
...
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/veiko/projektid/termin/web2py/gluon/restricted.py", line 141, in __init__
    self.traceback = traceback.format_exc()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 947: ordinal not in range(128)
```
And files in app 'errors' folder contain weird characters.

I start web2py with
`python3 web2py.py --nogui`
If I add inside default controller:
`import locale; print(locale.getpreferredencoding(False))`
it prints ANSI_X3.4-1968

If I start web2py with 
`python3 web2py.py`
then same line prints UTF-8 , and there are no decoding errors.


Discussion:
I can't reproduce the issue running welcome app on python3.6. Can you check your issue using master branch and welcome app?
Using Python3.5, web2py version 2.16.1-stable+timestamp.2017.11.13.23.50.07, Debian 9/Stretch

modified welcome app index function, add utf-8 character and not defined variable:
```
# ---- example index page ----
def index():
    response.flash = T("Hello Wörld")
    s
    return dict(message=T('Welcome to web2py!'))
```

Running web2py without --nogui, got error ticket:
```
Traceback (most recent call last):
  File "/home/veiko/projektid/test/web2py/gluon/restricted.py", line 219, in restricted
    exec(ccode, environment)
  File "/home/veiko/projektid/test/web2py/applications/welcome/controllers/default.py", line 61, in <module>
  File "/home/veiko/projektid/test/web2py/gluon/globals.py", line 419, in <lambda>
    self._caller = lambda f: f()
  File "/home/veiko/projektid/test/web2py/applications/welcome/controllers/default.py", line 10, in index
    s
NameError: name 's' is not defined
```

running with --nogui:
```
Traceback (most recent call last):
  File "/home/veiko/projektid/test/web2py/gluon/restricted.py", line 219, in restricted
    exec(ccode, environment)
NameError: name 's' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/veiko/projektid/test/web2py/gluon/restricted.py", line 141, in __init__
    self.traceback = traceback.format_exc()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 362: ordinal not in range(128)
```
@veiko99 could you check the error with python 3.6.4?
Seems like this line in main.py causing the problem
`locale.setlocale(locale.LC_CTYPE, "C") # IMPORTANT, web2py requires locale "C"`

Found related Python issue https://bugs.python.org/issue19846

In my system the code:
```
print("ö")
s
```
From command line:
`LC_CTYPE=C python3 test.py`
throws:
```
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    print("\xf6")
UnicodeEncodeError: 'ascii' codec can't encode character '\xf6' in position 0: ordinal not in range(128)
```
Not sure how to solve this so that I can run web2py with wsgi or --nogui using python3.5.
Maybe it is sensible do not set  LC_CTYPE=C when Python3 is used because str.lower() is not locale-aware in Python3.

```
$ python2 -c "import locale; locale.setlocale(locale.LC_CTYPE, 'tr_TR.utf8');print('REQUEST_URI'.lower()=='request_uri')"                         
False
$ python3 -c "import locale; locale.setlocale(locale.LC_CTYPE, 'tr_TR.utf8');print('REQUEST_URI'.lower()=='request_uri')"
True
```
Web2py issue 1583 introduced this LC_CTYPE=C change:
https://storage.googleapis.com/google-code-archive/v2/code.google.com/web2py/issues/issue-1583.json
( or in html https://pste.eu/p/X2Gm.html )
https://groups.google.com/forum/#!searchin/web2py-developers/locale|sort:date/web2py-developers/MwpLAy8yQDw/ehu67OQe9UwJ

Resolving patch:
diff --git a/gluon/widget.py b/gluon/widget.py
--- a/gluon/widget.py
+++ b/gluon/widget.py
@@ -1288,17 +1288,18 @@ def getline(filename, lineno, *args, **kwargs):
         line = py2exe_getline(filename, lineno, *args, **kwargs)
         if not line:
             try:
-                f = open(filename, "r")
+                f = open(filename, "rb")
                 try:
                     for i, line in enumerate(f):
+                        line = line.decode('utf-8')
                         if lineno == i + 1:
                             break
                     else:
-                        line = None
+                        line = ''
                 finally:
                     f.close()
             except (IOError, OSError):
-                line = None
+                line = ''
         return line
     linecache.getline = getline
 


Test patch: