Repository: internetarchive/openlibrary

Issue:
Books with non-ascii titles erroring when clicked from search
### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to https://openlibrary.org/search?q=h%C3%A9las&mode=everything
2. Click on any book with a non-ascii character in the title

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 500 internal error on e.g. https://openlibrary.org/works/OL11565520W?edition=
* Expected: See the book

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF82
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@cclauss 



Discussion:
It is possible to raise similar errors by doing http://localhost:8080/books/add with Unicode characters in the `Title` like:
`Life in Lüsai, Graubünden`
```
0.08 (1): 200 POST /openlibrary.org/save_many {'query': '[{"title": "Life in L\\u00fcsai, Graub\\u00fcnden", "authors": [{"author": {"key": ""}}], "key": "/works/OL1W", "type": {"key": "/type/work"}}, {"works": [{"key": "/works/OL1W"}], "title": "Life in L\\u00fcsai, Graub\\u00fcnden", "publishers": ["\\u00d1an\\u00f8"], "publish_date": "2021", "key": "/books/OL5M", "type": {"key": "/type/edition"}}]', 'comment': 'Added new book.', 'action': 'add-book', 'data': '{}'}
[2020-11-06 12:06:26 +0000] [217] [ERROR] Error handling request /books/add
Traceback (most recent call last):
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 182, in handle_request
    resp.write(item)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 333, in write
    self.send_headers()
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 329, in send_headers
    util.write(self.sock, util.to_bytestring(header_str, "ascii"))
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/util.py", line 507, in to_bytestring
    return value.encode(encoding)
UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 250: ordinal not in range(128)
```
When `gunicorn` is updated to the current [v20.0.4](https://pypi.org/project/gunicorn) the exception changes to:
```
0.09 (1): 200 POST /openlibrary.org/save_many {'query': '[{"title": "\\u00fc", "authors": [{"author": {"key": ""}}], "key": "/works/OL1W", "type": {"key": "/type/work"}}, {"works": [{"key": "/works/OL1W"}], "title": "\\u00fc", "publishers": ["Nano"], "publish_date": "2021", "key": "/books/OL5M", "type": {"key": "/type/edition"}}]', 'comment': 'Added new book.', 'action': 'add-book', 'data': '{}'}
[2020-11-06 13:38:34 +0000] [215] [ERROR] Error handling request /books/OL5M/%FC/edit?mode=add-work
Traceback (most recent call last):
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/web/httpserver.py", line 271, in __call__
    return self.app(environ, start_response)
  File "/openlibrary/openlibrary/core/middleware.py", line 42, in __call__
    data = self.app(environ, new_start_response)
  File "scripts/openlibrary-server", line 118, in wrapper
    return app(environ, start_response)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/web/application.py", line 313, in wsgi
    self.load(env)
  File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/web/application.py", line 457, in load
    ctx.path = env.get("PATH_INFO").encode("latin1").decode("utf8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 12: invalid start byte
```
Related to question https://github.com/webpy/webpy/issues/687
Tried `environ['PATH_INFO'] = environ['PATH_INFO'].encode('utf-8').decode('latin1')` after line 117 of `openlibrary/scripts/openlibrary-server`

Looked at `normpath` in nfogami/infogami/utils/app.py

https://stackoverflow.com/questions/57754478/django-utf-8-urls
Rolledback the deploy to resolve this issue over the weekend; we can find a fix next week before going back live.

Resolving patch:
diff --git a/openlibrary/core/processors/readableurls.py b/openlibrary/core/processors/readableurls.py
--- a/openlibrary/core/processors/readableurls.py
+++ b/openlibrary/core/processors/readableurls.py
@@ -1,5 +1,6 @@
 """Various web.py application processors used in OL.
 """
+import logging
 import os
 import web
 
@@ -8,6 +9,7 @@
 
 from six.moves import urllib
 
+logger = logging.getLogger("openlibrary.readableurls")
 
 try:
     from booklending_utils.openlibrary import is_exclusion
@@ -123,8 +125,9 @@ def get_readable_path(site, path, patterns, encoding=None):
     The patterns is a list of (path_regex, type, property_name, default_value)
     tuples.
     """
+
     def match(path):
-        for pat, type, property, default_title in patterns:
+        for pat, _type, _property, default_title in patterns:
             m = web.re_compile('^' + pat).match(path)
             if m:
                 prefix = m.group()
@@ -137,11 +140,12 @@ def match(path):
                 if suffix:
                     suffix = "/" + suffix
 
-                return type, property, default_title, prefix, middle, suffix
+                return _type, _property, default_title, prefix, middle, suffix
         return None, None, None, None, None, None
 
-    type, property, default_title, prefix, middle, suffix = match(path)
-    if type is None:
+    _type, _property, default_title, prefix, middle, suffix = match(path)
+
+    if _type is None:
         path = web.safeunicode(path)
         return (path, path)
 
@@ -161,9 +165,14 @@ def match(path):
     if thing:
         prefix = thing.key
 
-    if thing and thing.type.key == type:
-        title = thing.get(property) or default_title
-        middle = '/' + h.urlsafe(title.strip())
+    if thing and thing.type.key == _type:
+        title = thing.get(_property) or default_title
+        try:
+            # Explicitly only run for python3 to solve #4033
+            from urllib.parse import quote_plus
+            middle = '/' + quote_plus(h.urlsafe(title.strip()))
+        except ImportError:
+            middle = '/' + h.urlsafe(title.strip())
     else:
         middle = ""
 


Test patch: