internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: internetarchive/openlibrary

Issue:
Lists-related pages throwing "AttributeError: 'module' object has no attribute 'Seed' (falling back to default template)"
Caused by https://github.com/internetarchive/openlibrary/releases/tag/deploy-2020-11-04 .

`/opt/openlibrary/deploys/openlibrary/824a904/openlibrary/templates/recentchanges/lists/comment.html: error in processing template: AttributeError: 'module' object has no attribute 'Seed' (falling back to default template)`

Lots of occurrences in sentry: https://sentry.archive.org/sentry/ol-web/issues/10323/?environment=production

### Evidence / Screenshot (if possible)
![image](https://user-images.githubusercontent.com/6251786/98184251-ea249500-1ed7-11eb-9362-c54580ad530d.png)

### Relevant url?
e.g:
- https://openlibrary.org/people/ScarTissue/lists
- https://openlibrary.org/people/ScarTissue/lists/OL139466L/Test_Sep_2019

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF82
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@cclauss @mekarpeles 



Discussion:
Is this a dupe of #4015?
Will it be fixed but #4018?
This looks like a different error, and on py2. Not sure if that will fix it?
Maybe this?

![image](https://user-images.githubusercontent.com/6251786/98184987-b0ed2480-1ed9-11eb-8d34-109c2695c7f9.png)

https://github.com/internetarchive/openlibrary/compare/deploy-2020-10-19...deploy-2020-11-04#diff-9a03e714bd068dae76b2a99fdde09aff001fd7c547b11b0ea016079d3a567c8eL22
Tested on dev; bringing that back fixed it!

Resolving patch:
diff --git a/openlibrary/core/models.py b/openlibrary/core/models.py
--- a/openlibrary/core/models.py
+++ b/openlibrary/core/models.py
@@ -16,7 +16,8 @@
 from openlibrary.utils.isbn import to_isbn_13, isbn_13_to_isbn_10, canonical
 from openlibrary.core.vendors import create_edition_from_amazon_metadata
 
-from openlibrary.core.lists.model import ListMixin
+# Seed might look unused, but removing it causes an error :/
+from openlibrary.core.lists.model import ListMixin, Seed
 from . import cache, waitinglist
 
 from six.moves import urllib


Test patch:
Repository: internetarchive/openlibrary

Issue:
Search Facets broken
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

### Relevant url?
e.g. https://openlibrary.org/search?q=title%3A+%22test%22&mode=everything

### Steps to Reproduce
1. Go to https://openlibrary.org/search?q=title%3A+%22test%22&mode=everything
2. Click any of the facets

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: A weird map object is added to the url; e.g. https://openlibrary.org/search?q=title%3A+%22test%22&mode=everything&author_key=%3Cmap+object+at+0x7f0754411d00%3E
* Expected: Facet works

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF82
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@SouthGoingZax @cclauss 



Discussion:
Opened #4035 which appears when attempting to use Firefox Inspector on this page.

The problem is `author_key` as being created in [openlibrary/templates/work_search.html](https://github.com/internetarchive/openlibrary/blob/24c2235628239393dad62da1501ecda57c114e42/openlibrary/templates/work_search.html#L267)

Py2: author_key=OL1873431A
* https://openlibrary.org/search?q=title%3A+%22test%22&mode=everything&author_key=OL1873431A

Py3: author_key=%3Cmap+object+at+0x7f0754593b20%3E
* https://openlibrary.org/search?q=title%3A+%22test%22&mode=everything&author_key=%3Cmap+object+at+0x7f0754593b20%3E

Resolving patch:
diff --git a/openlibrary/plugins/openlibrary/code.py b/openlibrary/plugins/openlibrary/code.py
--- a/openlibrary/plugins/openlibrary/code.py
+++ b/openlibrary/plugins/openlibrary/code.py
@@ -703,7 +703,10 @@ def changequery(query=None, **kw):
         else:
             query[k] = v
 
-    query = dict((k, (map(web.safestr, v) if isinstance(v, list) else web.safestr(v))) for k, v in query.items())
+    query = dict(
+        (k, (list(map(web.safestr, v)) if isinstance(v, list) else web.safestr(v)))
+        for k, v in query.items()
+    )
     out = web.ctx.get('readable_path', web.ctx.path)
     if query:
         out += '?' + urllib.parse.urlencode(query, doseq=True)


Test patch:
Repository: internetarchive/openlibrary

Issue:
i18n files not built automatically on Python 3 web nodes
### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to https://openlibrary.org/books/OL25428864M/Americanah?lang=fr

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: Page is in English
* Expected: Page has buttons in French (like https://dev.openlibrary.org/books/OL25428864M/Americanah?lang=fr)

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF82
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@cclauss 



Discussion:
Dates are being translated, so the issue may lie in our i18n compilation process:
![i18n_issue_fr](https://user-images.githubusercontent.com/28732543/98397258-130a6e80-202d-11eb-9422-80393a8df6fe.png)


Is there an `.mo` file in `openlibrary/i18n/fr`?  If not, try running `make i18n` and check for errors.


:+1: That looks like it; `make i18n` looks like it wasn't run at all :/ Dropping in priority
I ran it manually, and it's working now 🎉 but we have to figure out why this wasn't run automatically during the build process
https://github.com/internetarchive/openlibrary/pull/3906#pullrequestreview-518213407
https://github.com/internetarchive/openlibrary/pull/2253

Resolving patch:
diff --git a/openlibrary/core/helpers.py b/openlibrary/core/helpers.py
--- a/openlibrary/core/helpers.py
+++ b/openlibrary/core/helpers.py
@@ -31,6 +31,9 @@
 from infogami.infobase.utils import parse_datetime
 from infogami.utils.view import safeint
 
+# TODO: i18n should be moved to core or infogami
+from openlibrary.i18n import gettext as _  # noqa: F401
+
 __all__ = [
     "sanitize",
     "json_encode",


Test patch:
Repository: internetarchive/openlibrary

Issue:
Add "noindex" meta tag to people pages less than 30 days old
### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
Thousands of spam pages have been added to OL over time.  In many instances, people and list pages are where the spam shows up.  Spam can harm user trust in OL, affect OL's ranking in search engines, and potentially point to malicious sites and thus harm our users.

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
In this week's community call, the decision was made to add the ["noindex" meta tag](https://en.wikipedia.org/wiki/Noindex) to people (and I think list) pages belonging to accounts less than 30 days old.  This will result in those pages not being indexed by search engines that honor that meta tag.

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles @cdrini 




Discussion:

Resolving patch:
diff --git a/openlibrary/core/helpers.py b/openlibrary/core/helpers.py
--- a/openlibrary/core/helpers.py
+++ b/openlibrary/core/helpers.py
@@ -35,7 +35,7 @@
     "sanitize",
     "json_encode",
     "safesort",
-    "datestr", "format_date",
+    "days_since", "datestr", "format_date",
     "sprintf", "cond", "commify", "truncate", "datetimestr_utc",
     "urlsafe", "texsafe",
     "percentage", "affiliate_id", "bookreader_host",
@@ -111,6 +111,12 @@ def safekey(x):
         return (k.__class__.__name__, k)
     return sorted(iterable, key=safekey, reverse=reverse)
 
+
+def days_since(then, now=None):
+    delta = then - (now or datetime.now())
+    return abs(delta.days)
+
+
 def datestr(then, now=None, lang=None, relative=True):
     """Internationalized version of web.datestr."""
     lang = lang or web.ctx.get('lang') or "en"


Test patch:
Repository: internetarchive/openlibrary

Issue:
Exceptions don't have e.message in Py3/isbn/ endpoint (when importing) throws error
* [ ] fixed in #4084
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

An error on this path should cause an error (it tries to import and hits a bug, which we'd love to know more about! instead...) it throws a different error, which is e.message not a thing in Python3 (it is in Py2). The solution is to use e.args[0], or e.args[1]

* [x] Whoever addresses this PR is responsible for opening a new issue for the /isbn endpoint as a test case for e.messages being fixed (so we know what the actual bug is for /isbn in Python).  _See #4083_

### Evidence / Screenshot (if possible)

![2020-11-11-141348_1600x900_scrot](https://user-images.githubusercontent.com/978325/98870778-aaeeca80-2428-11eb-9047-c24f8b0c6f8c.png)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

https://openlibrary.org/isbn/0998216429?debug=true

### Details

- **Logged in (Y/N)?**  Y

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

either core/vendor or plugins/openlibrary/code.py (isbn resolver ~L380) -- see screenshot

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/openlibrary/code.py b/openlibrary/plugins/openlibrary/code.py
--- a/openlibrary/plugins/openlibrary/code.py
+++ b/openlibrary/plugins/openlibrary/code.py
@@ -173,7 +173,7 @@ def sampleload(filename='sampledump.txt.gz'):
     else:
         f = open(filename)
 
-    queries = [simplejson.loads(line) for  line in f]
+    queries = [simplejson.loads(line) for line in f]
     print(web.ctx.site.save_many(queries))
 
 
@@ -378,7 +378,7 @@ def GET(self, isbn):
                 return web.found(ed.key + ext)
         except Exception as e:
             logger.error(e)
-            return e.message
+            return repr(e)
 
         web.ctx.status = '404 Not Found'
         return render.notfound(web.ctx.path, create=False)
@@ -526,7 +526,7 @@ def get_data(self, key):
             if e.json:
                 msg = self.dump(simplejson.loads(e.json))
             else:
-                msg = e.message
+                msg = str(e)
             raise web.HTTPError(e.status, data=msg)
 
         return simplejson.loads(d)


Test patch:
Repository: internetarchive/openlibrary

Issue:
Add cover is raising TypeError for posting a str, not bytes
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

Related to  #2852, #4080

Fixed in  #4102

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/catalog/add_book/__init__.py#L261 is posting str, not bytes.

### Evidence / Screenshot (if possible)
From https://sentry.archive.org/sentry/ol-web/issues/9042
![Screenshot 2020-11-12 at 16 23 16](https://user-images.githubusercontent.com/3709715/98959662-cb3f8780-2503-11eb-985d-b18a2f2875b5.png)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
The same error is thrown here: https://openlibrary.org/search?q=9781948434065&debug=true
The fix to this should be near-identical to https://github.com/internetarchive/openlibrary/pull/3973/files

Resolving patch:
diff --git a/openlibrary/catalog/add_book/__init__.py b/openlibrary/catalog/add_book/__init__.py
--- a/openlibrary/catalog/add_book/__init__.py
+++ b/openlibrary/catalog/add_book/__init__.py
@@ -25,7 +25,6 @@
 import json
 import re
 
-from six.moves import urllib
 import unicodedata as ucd
 import web
 
@@ -33,6 +32,8 @@
 from copy import copy
 from time import sleep
 
+import requests
+
 from infogami import config
 
 from openlibrary import accounts
@@ -258,16 +259,17 @@ def add_cover(cover_url, ekey, account=None):
     reply = None
     for attempt in range(10):
         try:
-            res = urllib.request.urlopen(upload_url, urllib.parse.urlencode(params))
-        except IOError:
+            payload = requests.compat.urlencode(params).encode('utf-8')
+            response = requests.post(upload_url, data=payload)
+        except requests.HTTPError:
             sleep(2)
             continue
-        body = res.read()
-        if res.getcode() == 500:
+        body = response.text
+        if response.status_code == 500:
             raise CoverNotSaved(body)
         if body not in ['', 'None']:
-            reply = json.loads(body)
-            if res.getcode() == 200 and 'id' in reply:
+            reply = response.json()
+            if response.status_code == 200 and 'id' in reply:
                 break
         sleep(2)
     if not reply or reply.get('message') == 'Invalid URL':


Test patch:
Repository: internetarchive/openlibrary

Issue:
Some pages with unicode characters rendering but displaying error
Appeared after https://github.com/internetarchive/openlibrary/releases/tag/deploy-2020-11-17

### Evidence / Screenshot (if possible)
![image](https://user-images.githubusercontent.com/6251786/99467710-fa436800-290c-11eb-98b0-cd6cc19cb73f.png)

### Relevant url?
e.g. https://openlibrary.org/works/OL11565520W/H%C3%A9las?edition=
e.g. https://openlibrary.org/authors/OL6644297A/Christopher_McDougall

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF82
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@mekarpeles @cclauss 



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/utils.py b/openlibrary/plugins/upstream/utils.py
--- a/openlibrary/plugins/upstream/utils.py
+++ b/openlibrary/plugins/upstream/utils.py
@@ -11,6 +11,7 @@
 import logging
 
 import six
+from six import PY3
 from six.moves import urllib
 from six.moves.collections_abc import MutableMapping
 
@@ -432,7 +433,9 @@ def __init__(self, tag="meta", **attrs):
         self.attrs = attrs
 
     def __str__(self):
-        attrs = ' '.join('%s="%s"' % (k, websafe(v)) for k, v in self.attrs.items())
+        attrs = ' '.join(
+            '%s="%s"' % (k, websafe(v) if PY3 else websafe(v).encode('utf8'))
+            for k, v in self.attrs.items())
         return '<%s %s />' % (self.tag, attrs)
 
     def __repr__(self):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Admin spamwords functionality should support regular expressions
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
The admin [spamwords functionality supports string matching](https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/spamcheck.py#L45) but not full regular expressions.  There are OL spam detection efforts external to this repository that utilize regular expressions, and it would be nice for all the OL spam detection options to have consistent functionality in this regard.

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
The admin spamwords functionality should support regular expressions.

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
Update the spamwords library to use Python's ```re.search()``` function.  The amount of text being matched against each time isn't that large, and I feel like updates aren't frequent enough to cause performance problems.

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini @mekarpeles 


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/spamcheck.py b/openlibrary/plugins/upstream/spamcheck.py
--- a/openlibrary/plugins/upstream/spamcheck.py
+++ b/openlibrary/plugins/upstream/spamcheck.py
@@ -1,3 +1,4 @@
+import re
 import web
 
 def get_spam_words():
@@ -42,7 +43,7 @@ def is_spam(i=None):
     if i is None:
         i = web.input()
     text = str(dict(i)).lower()
-    return any(w.lower() in text for w in spamwords)
+    return any(re.search(w.lower(), text) for w in spamwords)
 
 def is_spam_email(email):
     domain = email.split("@")[-1].lower()


Test patch:
Repository: internetarchive/openlibrary

Issue:
Coverstore: Properly deal with covers = [None]
<!-- What issue does this PR close? -->
Closes https://sentry.archive.org/sentry/ol-covers/issues/11156

<!-- What does this PR achieve? [feature|hotfix|fix|refactor] -->

Sometimes `covers` is stored as `[None]` or `[-1]` to indicate that there are no covers.  The current logic does `covers[0] <= 0` which will raise an uncaught TypeError when covers[0] is None.  This modification converts None to 0 to avoid the uncaught exception.

### Technical
<!-- What should be noted about the implementation? -->

### Testing
<!-- Steps for reviewer to reproduce/verify what this PR does/fixes. -->

### Screenshot
<!-- If this PR touches UI, please post evidence (screenshots) of it behaving correctly. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/coverstore/code.py b/openlibrary/coverstore/code.py
--- a/openlibrary/coverstore/code.py
+++ b/openlibrary/coverstore/code.py
@@ -39,7 +39,7 @@ def get_cover_id(olkeys):
         covers = doc.get('photos' if is_author else 'covers', [])
         # Sometimes covers is stored as [None] or [-1] to indicate no covers.
         # If so, consider there are no covers.
-        if covers and (covers[0] or 0) >= 0:
+        if covers and (covers[0] or -1) >= 0:
             return covers[0]
 
 def _query(category, key, value):


Test patch:
Repository: internetarchive/openlibrary

Issue:
solrupdater getting randomly stuck
Solrupdater has been getting stuck randomly at various dates. It seems related to the infobase py3 deploy.

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Do nothing

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: It seems to occasionally get stuck in the log
* Expected: Solrupdater runs through the infobase logs, making updates

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@seabelis 



Discussion:
Uh oh;  `curl 'http://ol-home:7000/openlibrary.org/log/2021-12-13:0?limit=100'` from ol-home (not it's a day in the future) should return an empty object (that's how solrupdater knows there are no new edits). It's returning:

```
{
    data: [...edits...]
    offset: "2020-12-01:339934"
}
```
^^ That offset is around the time we moved infobase to its new server with python 3 :grimacing:

Resolving patch:
diff --git a/scripts/new-solr-updater.py b/scripts/new-solr-updater.py
--- a/scripts/new-solr-updater.py
+++ b/scripts/new-solr-updater.py
@@ -94,6 +94,18 @@ def read_records(self, max_fetches=10):
             # no more data is available
             if not data:
                 logger.debug("no more records found")
+                # There's an infobase bug where we'll sometimes get 0 items, but the
+                # binary offset will have incremented...?
+                if 'offset' in d:
+                    # There's _another_ infobase bug where if you query a future date,
+                    # it'll return back 2020-12-01. To avoid solrupdater getting stuck
+                    # in a loop, only update the offset if it's newer than the current
+                    old_day, old_boffset = self.offset.split(':')
+                    old_boffset = int(old_boffset)
+                    new_day, new_boffset = d['offset'].split(':')
+                    new_boffset = int(new_boffset)
+                    if new_day >= old_day and new_boffset >= old_boffset:
+                        self.offset = d['offset']
                 return
 
             for record in data:


Test patch: