Repository: internetarchive/openlibrary

Issue:
Adding non-Latin character book failing 500:  "UnicodeEncodeError: 'ascii' codec can't encode characters"
### Evidence / Screenshot (if possible)

![Screenshot_2020-12-19_13-30-48](https://user-images.githubusercontent.com/265872/102686055-7841aa00-41fe-11eb-8b4d-a50c70a6f9e3.png)

### Relevant url?
https://openlibrary.org/books/add

### Steps to Reproduce
1. Log-in with account
2. More -> Add Book
3. Enter title: `Справа Василя Стуса. Збірка документів з архіву колишнього КДБ УРСР`
4. Enter unknown author: `Вахтанг Кіпіані` (Click "create new record for")
5. Enter publisher: `Віват`
6. Enter published date: `2019`
7. Enter USBN13: `9789669429278`
8. Press "Add"

* Actual: Error 500
* Expected: Book added to system

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** Firefox latest
- **Operating system?** Debian Linux 10
- **Environment (prod/dev/local)?** prod

### Proposal & Constraints
I'm using cyrillic symbols for book and author name. Might be the issue?


Discussion:
Thanks for the report, @dector.  Can you update the Steps To Reproduce with the text you're using for the book and author name?
@SouthGoingZax Sure! My second suggestion is that it's happening because of new author.

Title: `Справа Василя Стуса. Збірка документів з архіву колишнього КДБ УРСР`
Author: `Вахтанг Кіпіані` (Click "create new record for")
Publisher: `Віват`
Publish year: `2019`
ISBN13: `9789669429278`
Thanks.

I can reproduce it.  The error from my local dev environment is:

```
solr_1       | INFO: [] webapp=/solr path=/select params={wt=json&q.op=AND&q=title:"??????+??????+?????.+??????+??????????+?+??????+??????????+???+????"+AND+author_key:"OL5A"+AND+publisher:"?????"+AND+publish_date:"2019"+AND+isbn:"9789669429278"&start=0} hits=1 status=0 QTime=7 
web_1        | [2020-12-21 20:20:56 +0000] [218] [ERROR] Error handling request /books/add?debug=true
web_1        | Traceback (most recent call last):
web_1        |   File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 135, in handle
web_1        |     self.handle_request(listener, req, client, addr)
web_1        |   File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 182, in handle_request
web_1        |     resp.write(item)
web_1        |   File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 333, in write
web_1        |     self.send_headers()
web_1        |   File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 329, in send_headers
web_1        |     util.write(self.sock, util.to_bytestring(header_str, "ascii"))
web_1        |   File "/home/openlibrary/.pyenv/versions/3.8.6/lib/python3.8/site-packages/gunicorn/util.py", line 507, in to_bytestring
web_1        |     return value.encode(encoding)
web_1        | UnicodeEncodeError: 'ascii' codec can't encode characters in position 201-206: ordinal not in range(128)
```

It looks like gunicorn is trying to encode response headers as ascii, which triggers the error.

Looking at the headers of a successful book addition attempt, this request results in an HTTP 303 and redirects to the edit page for the added book so the user can continue adding additional metadata.  The edit page in the Location header of the 303 response contains the title of the added book, which could be the problem when using non-ascii characters:

![image](https://user-images.githubusercontent.com/72705998/102830185-9a256300-439d-11eb-86e1-5c68dc339ac2.png)

The version of gunicorn being used is 2.5 years old.  It looks like that encoding was changed in a later version ([commit](https://github.com/benoitc/gunicorn/commit/879651bb6f873631725cc31cbda783349fdf9dc3)).  Maybe upgrading gunicorn could help.  If the above theory is correct, a work-around could also be to not redirect the user to the edit page automatically if non-ascii characters are in the book title, but instead return a page with a link to the edit page for the user to click on. @cdrini, what are your thoughts on how to proceed? 

Also, it looks like the [book addition succeeded](https://openlibrary.org/works/OL24096573W).
@SouthGoingZax Awesome, thanks! Can I remove duplicates somehow?
You can request that duplicates be merged using the [contact form](https://openlibrary.org/contact).  It's an ability limited to people designated as admins and librarians.
Got it, thanks a lot.
@SouthGoingZax Is this still an issue? A number of our utf8 issues were fixed in the last few months during the py2/3 migration. If the issue is still there, upgrading gunicorn to see if that fixes it seems reasonable :+1:
@cclauss This issue seems vaguely familiar; was there another issue very similar to this? I thought we might've already tried updating gunicorn.
Like #4225 Internal Server Error attempting to merge authors with Unicode names which we were to discuss on today's call.

We are running current gunicorn https://github.com/internetarchive/openlibrary/blob/8203f8c262b01250e52cf386c9fdc5fe2e3b7e1e/requirements_common.txt#L11 vs. https://pypi.org/project/gunicorn
The error messages look like the issue I fixed on borrow redirects in #4861 -- that PR does not fix this, but somewhere a redirect is occurring with a non-URL-encoded URL with Unicode characters in the redirect header `location:` field  -- gunicorn does not like that.   
@dhruvmanila  would you by any chance be able to help out with this one? It's causing a lot of our non-English users grip, and I won't be able to look at it before next month :( If you have the time, would love to jump on a call to explain this.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/addbook.py b/openlibrary/plugins/upstream/addbook.py
--- a/openlibrary/plugins/upstream/addbook.py
+++ b/openlibrary/plugins/upstream/addbook.py
@@ -27,6 +27,7 @@
 
 import six
 from six.moves import urllib
+from web.webapi import SeeOther
 
 
 logger = logging.getLogger("openlibrary.book")
@@ -130,6 +131,31 @@ def create_authors_from_form_data(self, authors, author_names, _test=False):
         return created
 
 
+def encode_url_path(url: str) -> str:
+    """Encodes the path part of the url to avoid issues with non-latin characters as
+    non-latin characters was breaking `web.seeother`.
+
+    >>> encode_url_path('/books/OL10M/Вас_ил/edit?mode=add-work')
+    '/books/OL10M/%D0%92%D0%B0%D1%81_%D0%B8%D0%BB/edit?mode=add-work'
+    >>> encode_url_path('')
+    ''
+    >>> encode_url_path('/')
+    '/'
+    >>> encode_url_path('/books/OL11M/进入该海域?mode=add-work')
+    '/books/OL11M/%E8%BF%9B%E5%85%A5%E8%AF%A5%E6%B5%B7%E5%9F%9F?mode=add-work'
+    """
+    result = urllib.parse.urlparse(url)
+    correct_path = "/".join(urllib.parse.quote(part) for part in result.path.split("/"))
+    result = result._replace(path=correct_path)
+    return result.geturl()
+
+
+def safe_seeother(url: str) -> SeeOther:
+    """Safe version of `web.seeother` which encodes the url path appropriately using
+     `encode_url_path`."""
+    return web.seeother(encode_url_path(url))
+
+
 class addbook(delegate.page):
     path = "/books/add"
 
@@ -137,7 +163,7 @@ def GET(self):
         """Main user interface for adding a book to Open Library."""
 
         if not self.has_permission():
-            return web.seeother("/account/login?redirect={}".format(self.path))
+            return safe_seeother("/account/login?redirect={}".format(self.path))
 
         i = web.input(work=None, author=None)
         work = i.work and web.ctx.site.get(i.work)
@@ -347,7 +373,7 @@ def work_match(self, saveutil, work, i):
         comment = utils.get_message("comment_add_book")
         saveutil.commit(comment=comment, action="add-book")
 
-        raise web.seeother(edition.url("/edit?mode=add-book"))
+        raise safe_seeother(edition.url("/edit?mode=add-book"))
 
     def work_edition_match(self, edition):
         """
@@ -355,7 +381,7 @@ def work_edition_match(self, edition):
         Redirect user to the found item's edit page to add any missing details.
         :param Edition edition:
         """
-        raise web.seeother(edition.url("/edit?mode=found"))
+        raise safe_seeother(edition.url("/edit?mode=found"))
 
     def no_match(self, saveutil, i):
         """
@@ -383,7 +409,7 @@ def no_match(self, saveutil, i):
         comment = utils.get_message("comment_add_book")
         saveutil.commit(action="add-book", comment=comment)
 
-        raise web.seeother(edition.url("/edit?mode=add-work"))
+        raise safe_seeother(edition.url("/edit?mode=add-work"))
 
     def _make_edition(self, work, i):
         """
@@ -772,7 +798,7 @@ def POST(self, key):
             else:
                 add_flash_message("info", utils.get_message("flash_book_updated"))
 
-            raise web.seeother(urllib.parse.quote(edition.url()))
+            raise safe_seeother(edition.url())
         except ClientException as e:
             add_flash_message('error', e.args[-1] or e.json)
             return self.GET(key)
@@ -822,7 +848,7 @@ def POST(self, key):
             helper = SaveBookHelper(work, None)
             helper.save(web.input())
             add_flash_message("info", utils.get_message("flash_work_updated"))
-            raise web.seeother(work.url())
+            raise safe_seeother(work.url())
         except (ClientException, ValidationException) as e:
             add_flash_message('error', str(e))
             return self.GET(key)
@@ -853,11 +879,11 @@ def POST(self, key):
             elif "_save" in i:
                 author.update(formdata)
                 author._save(comment=i._comment)
-                raise web.seeother(key)
+                raise safe_seeother(key)
             elif "_delete" in i:
                 author = web.ctx.site.new(key, {"key": key, "type": {"key": "/type/delete"}})
                 author._save(comment=i._comment)
-                raise web.seeother(key)
+                raise safe_seeother(key)
         except (ClientException, ValidationException) as e:
             add_flash_message('error', str(e))
             author.update(formdata)


Test patch: