Repository: internetarchive/openlibrary

Issue:
Borrow redirects contain utf8 in the header location: field, causing gunicorn error
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
From
https://sentry.archive.org/sentry/ol-web/issues/10505/
mentioned in #4092

example:
```
'HTTP/1.0 303 See Other\r\nServer: gunicorn/20.0.4\r\nDate: Mon, 15 Mar 2021 05:22:00 GMT\r\nConnection: close\r\nSet-Cookie: session=; expires=Fri, 07 Jul 1989 03:35:20 GMT; Path=/\r\nContent-Type: text/html\r\nLocation: https://openlibrary.org/account/login?redirect=/books/OL21782175M/Democratie_française./borrow?action=borrow\r\nX-OL-Stats: "MC 2 0.001 TT 0 0.126"\r\n\r\n'
```

The sentry logs show this occurring in borrow redirects, maybe `type/redirects` do the correct URL encoding.


Looks like gunicorn is has a hard coded latin-1 as a character set for HTTP headers:

https://github.com/benoitc/gunicorn/blob/b83448bfb42184f43adf155f86bcd2a570a1139d/gunicorn/http/wsgi.py#L322

The headers in question contain the utf-8 book title names as redirect URLs. <-- according to specs (https://tools.ietf.org/html/rfc7230#page-28 ) headers _should_ (rather than MUST?) be USASCII....

The method that does the conversion has a sensible utf-8 default https://github.com/benoitc/gunicorn/blob/b83448bfb42184f43adf155f86bcd2a570a1139d/gunicorn/util.py#L558-L565


* Confirm redirect types encode utf8 in `Location:` header fields
* Check how borrow redirects set the `Location:` field


### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/core/lending.py b/openlibrary/core/lending.py
--- a/openlibrary/core/lending.py
+++ b/openlibrary/core/lending.py
@@ -230,8 +230,12 @@ def get_groundtruth_availability(ocaid, s3_keys=None):
     including 1-hour borrows"""
     params = '?action=availability&identifier=' + ocaid
     url = S3_LOAN_URL % config_bookreader_host
-    r = requests.post(url + params, data=s3_keys)
-    data = r.json().get('lending_status')
+   try:
+        response = requests.post(url + params, data=s3_keys)
+        response.raise_for_status()
+    except requests.HTTPError:
+        pass  # TODO: Handle unexpected responses from the availability server.
+    data = response.json().get('lending_status')
     # For debugging
     data['__src__'] = 'core.models.lending.get_groundtruth_availability'
     return data
@@ -308,7 +312,7 @@ def get_availability(key, ids):
     :param list of str ids:
     :rtype: dict
     """
-
+    ids = [id_ for id_ in ids if id_]  # remove infogami.infobase.client.Nothing
     if not ids:
         return {}
 
diff --git a/openlibrary/plugins/upstream/borrow.py b/openlibrary/plugins/upstream/borrow.py
--- a/openlibrary/plugins/upstream/borrow.py
+++ b/openlibrary/plugins/upstream/borrow.py
@@ -129,7 +129,8 @@ def POST(self, key):
         if availability and availability['status'] == 'open':
             raise web.seeother(archive_url)
 
-        error_redirect = (archive_url)
+        error_redirect = archive_url
+        edition_redirect = urllib.parse.quote(edition.url())
         user = accounts.get_current_user()
 
         if user:
@@ -139,7 +140,7 @@ def POST(self, key):
         if not user or not ia_itemname or not s3_keys:
             web.setcookie(config.login_cookie_name, "", expires=-1)
             redirect_url = "/account/login?redirect=%s/borrow?action=%s" % (
-                edition.url(), action)
+                edition_redirect, action)
             if i._autoReadAloud is not None:
                 redirect_url += '&_autoReadAloud=' + i._autoReadAloud
             raise web.seeother(redirect_url)
@@ -147,15 +148,15 @@ def POST(self, key):
         if action == 'return':
             lending.s3_loan_api(edition.ocaid, s3_keys, action='return_loan')
             stats.increment('ol.loans.return')
-            raise web.seeother(edition.url())
+            raise web.seeother(edition_redirect)
         elif action == 'join-waitinglist':
             lending.s3_loan_api(edition.ocaid, s3_keys, action='join_waitlist')
             stats.increment('ol.loans.joinWaitlist')
-            raise web.redirect(edition.url())
+            raise web.redirect(edition_redirect)
         elif action == 'leave-waitinglist':
             lending.s3_loan_api(edition.ocaid, s3_keys, action='leave_waitlist')
             stats.increment('ol.loans.leaveWaitlist')
-            raise web.redirect(edition.url())
+            raise web.redirect(edition_redirect)
 
         # Intercept a 'borrow' action if the user has already
         # borrowed the book and convert to a 'read' action.


Test patch: