internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: internetarchive/openlibrary

Issue:
Unable to import csv generated by Goodreads
When trying to import my library with the generated .csv file from Goodreads, I receive an internal error.

```
Hmm...

Sorry. There seems to be a problem with what you were just looking at.

We've noted the error 2020-12-04/191144042614 and will look into it as soon as possible. Head for home?
```

https://openlibrary.org/account/import/goodreads

- **Logged in (Y/N)?** yes
- **Browser type/version?** Mozilla Firefox 83.0
- **Operating system?** GNU/Linux
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->



Discussion:
@tabshaikh one for you to investigate when you have time :)
@tabshaikh I'd like to help with this. Point me where to start
Note: The timeframe suggests this might be a python 3 issue, so we should also check that.
It does look like a regression from the migration to Py3.  If you use the Chrome developer tools and add "?debug=true" to the form submit action, you get:

![image](https://user-images.githubusercontent.com/72705998/102017157-06f38a00-3d1a-11eb-93ab-71355df9ad9c.png)

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -864,8 +864,8 @@ class fetch_goodreads(delegate.page):
     def POST(self):
         import csv
         i = web.input(csv={})
-        csv_file = csv.reader((i.csv.value).splitlines(),
-                              delimiter=',', quotechar='"')
+        csv_payload = i.csv.value if isinstance(i.csv.value, str) else i.csv.value.decode()
+        csv_file = csv.reader(csv_payload.splitlines(), delimiter=',', quotechar='"')
         header = next(csv_file)
         books = {}
         books_wo_isbns = {} 


Test patch:
Repository: internetarchive/openlibrary

Issue:
A GET to /account/import/goodreads should redirect to /account/import
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
https://news.ycombinator.com/item?id=25409173 points out that a GET to https://openlibrary.org/account/import/goodreads returns "method not allowed"

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
It would be a better user experience to redirect such requests to /account/import

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles @cdrini 




Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -860,6 +860,9 @@ def GET(self):
 class fetch_goodreads(delegate.page):
     path = "/account/import/goodreads"
 
+    def GET(self):
+        raise web.seeother("/account/import")
+
     @require_login
     def POST(self):
         import csv
@@ -868,7 +871,7 @@ def POST(self):
         csv_file = csv.reader(csv_payload.splitlines(), delimiter=',', quotechar='"')
         header = next(csv_file)
         books = {}
-        books_wo_isbns = {} 
+        books_wo_isbns = {}
         for book in list(csv_file):
             _book = dict(zip(header, book))
             _book['ISBN'] = _book['ISBN'].replace('"','').replace('=','')
@@ -879,7 +882,7 @@ def POST(self):
                 books[_book['ISBN13']] = _book
                 books[_book['ISBN13']]['ISBN'] = _book['ISBN13']
             else:
-                books_wo_isbns[_book['Book Id']] = _book     
+                books_wo_isbns[_book['Book Id']] = _book
         return render['account/import'](books, books_wo_isbns)
 
 class export_books(delegate.page):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Remove send_email_change_email dead code
After #2595, this function will be dead code. It has some weird stuff in its body, so proceed with care, but it and any single-use functions/templates it references should be deleted

https://github.com/internetarchive/openlibrary/blob/0bb56098f087de09ce85592f0bf14b3be5ef6162/openlibrary/plugins/upstream/account.py#L801

## Stakeholders
@jdlrobson 


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -928,17 +928,6 @@ def GET(self):
 #         return render.notfound(path, create=False)
 
 
-def send_email_change_email(username, email):
-    key = "account/%s/email" % username
-
-    doc = create_link_doc(key, username, email)
-    web.ctx.site.store[key] = doc
-
-    link = web.ctx.home + "/account/email/verify/" + doc['code']
-    msg = render_template("email/email/verify", username=username, email=email, link=link)
-    sendmail(email, msg)
-
-
 def send_forgot_password_email(username, email):
     key = "account/%s/password" % username
 


Test patch:
Repository: internetarchive/openlibrary

Issue:
Extend anti-spam functionality to list pages
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
List pages are a common source of spam.

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
We should extend the existing anti-spam functionality to list pages.

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles @cdrini 




Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/openlibrary/lists.py b/openlibrary/plugins/openlibrary/lists.py
--- a/openlibrary/plugins/openlibrary/lists.py
+++ b/openlibrary/plugins/openlibrary/lists.py
@@ -12,6 +12,7 @@
 from openlibrary.core import formats, cache
 import openlibrary.core.helpers as h
 from openlibrary.utils import dateutil
+from openlibrary.plugins.upstream import spamcheck
 from openlibrary.plugins.worksearch import subjects
 
 
@@ -154,6 +155,9 @@ def POST(self, user_key):
             seeds=seeds
         )
 
+        if spamcheck.is_spam(lst):
+            raise self.forbidden()
+
         try:
             result = site.save(lst.dict(),
                 comment="Created new list.",


Test patch:
Repository: internetarchive/openlibrary

Issue:
Account Creation: 1st login OL redirects you to IA (referer)
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -307,7 +307,7 @@ def render_error(self, error_key, i):
     def GET(self):
         referer = web.ctx.env.get('HTTP_REFERER', '/')
         # Don't set referer on user activation
-        if '//archive.org/account/verify.php' in referer:
+        if 'archive.org' in referer:
             referer = None
         i = web.input(redirect=referer)
         f = forms.Login()


Test patch:
Repository: internetarchive/openlibrary

Issue:
Send S3 keys when POSTing observations to TheBestBookOn.com
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
In order to improve authentication at The Best Book On, S3 keys should be sent when patrons make observations about books.

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
[Related TBBO issue](https://github.com/Open-Book-Genome-Project/TheBestBookOn.com/issues/45)
https://github.com/Open-Book-Genome-Project/TheBestBookOn.com/issues/45

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles 




Discussion:

Resolving patch:
diff --git a/openlibrary/core/observations.py b/openlibrary/core/observations.py
new file mode 100644
--- /dev/null
+++ b/openlibrary/core/observations.py
@@ -0,0 +1,19 @@
+"""Module for handling patron observation functionality"""
+
+import requests
+
+from infogami import config
+from openlibrary import accounts
+
+# URL for TheBestBookOn
+TBBO_URL = config.get('tbbo_url')
+
+def post_observation(data, s3_keys):
+    headers = {
+        'x-s3-access': s3_keys['access'],
+        'x-s3-secret': s3_keys['secret']
+    }
+
+    response = requests.post(TBBO_URL + '/api/observations', data=data, headers=headers)
+
+    return response.text
diff --git a/openlibrary/plugins/openlibrary/api.py b/openlibrary/plugins/openlibrary/api.py
--- a/openlibrary/plugins/openlibrary/api.py
+++ b/openlibrary/plugins/openlibrary/api.py
@@ -19,6 +19,7 @@
 from openlibrary.plugins.worksearch.subjects import get_subject
 from openlibrary.accounts.model import OpenLibraryAccount
 from openlibrary.core import ia, db, models, lending, helpers as h
+from openlibrary.core.observations import post_observation
 from openlibrary.core.sponsorships import qualifies_for_sponsorship
 from openlibrary.core.vendors import (
     get_amazon_metadata, create_edition_from_amazon_metadata,
@@ -373,3 +374,19 @@ def GET(self):
                     metadata['ocaid'] = ed.ocaid
 
         return json.dumps(metadata)
+
+
+class observations(delegate.page):
+    path = "/observations"
+    encoding = "json"
+
+    def POST(self):
+        user = accounts.get_current_user()
+
+        if user:
+            account = OpenLibraryAccount.get_by_email(user.email)
+            s3_keys = web.ctx.site.store.get(account._key).get('s3_keys')
+
+            if s3_keys:
+                response = post_observation(web.data(), s3_keys)
+                return delegate.RawText(response)


Test patch:
Repository: internetarchive/openlibrary

Issue:
Is openlibrary/catalog/amazon/crawl_top_books.py used?
### Question
<!-- What question needs to be answered to close this issue? This should be one sentence. -->
[This file](../blob/master/openlibrary/catalog/amazon/crawl.py) makes a lot of assumptions about bytes vs. str.  Does it need to be ported to Python 3?

### Additional context
<!-- Add any other context or details here. -->


### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
@cclauss that file is written as a stand-alone command line script. I don't believe it is used as part of any current OL process. 

Moving it an archived `scripts` folder would be appropriate. I don't think converting it to Python3 is necessary. It's not a core part of OL, so I'd say move or remove without conversion.

Resolving patch:
diff --git a/openlibrary/catalog/amazon/crawl_top_books.py b/scripts/legacy/crawl_top_books.py
similarity index 100%
rename from openlibrary/catalog/amazon/crawl_top_books.py
rename to scripts/legacy/crawl_top_books.py


Test patch:
Repository: internetarchive/openlibrary

Issue:
/openlibrary/openlibrary/templates/lists/widget.html: error in processing template: TypeError: Object of type Nothing is not JSON serializable (falling back to default template)
Patron is reporting the following error:
`/openlibrary/openlibrary/templates/lists/widget.html: error in processing template: TypeError: Object of type Nothing is not JSON serializable (falling back to default template)`

### Evidence / Screenshot (if possible)

### Relevant url?
https://openlibrary.org/works/OL24171550W/Kelebihan_Amalan_Bulan_Rejab_Sya%E2%80%99ban_Ramadhan

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ... https://openlibrary.org/works/OL24171550W/Kelebihan_Amalan_Bulan_Rejab_Sya%E2%80%99ban_Ramadhan
2. Do ... view error.

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
Unable to reproduce this issue on dev: https://dev.openlibrary.org/works/OL24171550W/Kelebihan_Amalan_Bulan_Rejab_Sya%E2%80%99ban_Ramadhan
https://dev.openlibrary.org/status shows that it is running legacy Python 2.7.6
https://openlibrary.org/status shows that it is running Python 3.8.6

Resolving patch:
diff --git a/openlibrary/core/helpers.py b/openlibrary/core/helpers.py
--- a/openlibrary/core/helpers.py
+++ b/openlibrary/core/helpers.py
@@ -101,7 +101,7 @@ def get_nofollow(name, event):
 def json_encode(d, **kw):
     """Same as json.dumps.
     """
-    return json.dumps(d, **kw)
+    return json.dumps(d or {}, **kw)
 
 
 def safesort(iterable, key=None, reverse=False):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Adding to lists broken
Adding an item to a list no longer works as of 12-02-2021.

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...an edition, etc.
2. Do ...add item to list.

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: List link loads list page.
* Expected: Item should be added to list.

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** Chrome Version 88.0.4324.150 (Official Build) (x86_64)
- **Operating system?** Mac Big Sur
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cclauss 


Discussion:
Seeing this error in the console:

![image](https://user-images.githubusercontent.com/6251786/107823265-7e12c200-6d4d-11eb-904a-1f516f7e6227.png)


Resolving patch:
diff --git a/openlibrary/core/helpers.py b/openlibrary/core/helpers.py
--- a/openlibrary/core/helpers.py
+++ b/openlibrary/core/helpers.py
@@ -101,7 +101,7 @@ def get_nofollow(name, event):
 def json_encode(d, **kw):
     """Same as json.dumps.
     """
-    return json.dumps(d or {}, **kw)
+    return json.dumps(d, **kw)
 
 
 def safesort(iterable, key=None, reverse=False):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Bookshelves appear empty on Chrome
Bookshelves appear empty until scrolling happens.

### Evidence / Screenshot (if possible)

### Relevant url?
https://openlibrary.org/explore

### Steps to Reproduce

1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/languages.py b/openlibrary/plugins/worksearch/languages.py
--- a/openlibrary/plugins/worksearch/languages.py
+++ b/openlibrary/plugins/worksearch/languages.py
@@ -1,9 +1,11 @@
 """Language pages
 """
 
+from infogami.plugins.api.code import jsonapi
 from infogami.utils import delegate, stats
 from infogami.utils.view import render_template, safeint
 import web
+import json
 import logging
 
 from . import subjects
@@ -40,19 +42,38 @@ def process_key(self, key):
         return key.replace("_", " ")
 
 
+def get_top_languages(limit):
+    from . import search
+    result = search.get_solr().select('*:*', rows=0, facets=['language'], facet_limit=limit)
+    return [
+        web.storage(
+            name=get_language_name(row.value),
+            key='/languages/' + row.value,
+            count=row.count
+        )
+        for row in result['facets']['language']
+    ]
+
+
 class index(delegate.page):
     path = "/languages"
 
     def GET(self):
-        from . import search
-        result = search.get_solr().select('*:*', rows=0, facets=['language'], facet_limit=500)
-        languages = [web.storage(name=get_language_name(row.value), key='/languages/' + row.value, count=row.count)
-                    for row in result['facets']['language']]
-        return render_template("languages/index", languages)
+        return render_template("languages/index", get_top_languages(500))
 
     def is_enabled(self):
         return True
 
+
+class index_json(delegate.page):
+    path = "/languages"
+    encoding = "json"
+
+    @jsonapi
+    def GET(self):
+        return json.dumps(get_top_languages(15))
+
+
 class language_search(delegate.page):
     path = '/search/languages'
 


Test patch:
Repository: internetarchive/openlibrary

Issue:
Cache The Best Book On aspects
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
Every time the book notes modal is opened on a book page, a call to The Best Book On is made to fetch a list of aspects that populate the form.  These aspects will not often change, so it makes sense to cache them instead of fetching the same data each time a patron opens the modal.


Discussion:

Resolving patch:
diff --git a/openlibrary/core/observations.py b/openlibrary/core/observations.py
--- a/openlibrary/core/observations.py
+++ b/openlibrary/core/observations.py
@@ -4,6 +4,7 @@
 
 from infogami import config
 from openlibrary import accounts
+from . import cache
 
 # URL for TheBestBookOn
 TBBO_URL = config.get('tbbo_url')
@@ -17,3 +18,9 @@ def post_observation(data, s3_keys):
     response = requests.post(TBBO_URL + '/api/observations', data=data, headers=headers)
 
     return response.text
+
+@cache.memoize(engine="memcache", key="tbbo_aspects", expires=config.get('tbbo_aspect_cache_duration'))
+def get_aspects():
+    response = requests.get(TBBO_URL + '/api/aspects')
+
+    return response.text
diff --git a/openlibrary/plugins/openlibrary/api.py b/openlibrary/plugins/openlibrary/api.py
--- a/openlibrary/plugins/openlibrary/api.py
+++ b/openlibrary/plugins/openlibrary/api.py
@@ -19,8 +19,8 @@
 from openlibrary.plugins.worksearch.subjects import get_subject
 from openlibrary.accounts.model import OpenLibraryAccount
 from openlibrary.core import ia, db, models, lending, helpers as h
+from openlibrary.core.observations import post_observation, get_aspects
 from openlibrary.core.models import Booknotes
-from openlibrary.core.observations import post_observation
 from openlibrary.core.sponsorships import qualifies_for_sponsorship
 from openlibrary.core.vendors import (
     get_amazon_metadata, create_edition_from_amazon_metadata,
@@ -438,3 +438,11 @@ def POST(self):
             if s3_keys:
                 response = post_observation(web.data(), s3_keys)
                 return delegate.RawText(response)
+
+
+class aspects(delegate.page):
+    path = "/aspects"
+    encoding = "json"
+
+    def GET(self):
+        return delegate.RawText(get_aspects())
diff --git a/openlibrary/plugins/upstream/utils.py b/openlibrary/plugins/upstream/utils.py
--- a/openlibrary/plugins/upstream/utils.py
+++ b/openlibrary/plugins/upstream/utils.py
@@ -287,11 +287,6 @@ def get_coverstore_url():
     return config.get('coverstore_url', 'https://covers.openlibrary.org').rstrip('/')
 
 
-@public
-def get_the_best_book_on_url():
-    return config.get('tbbo_url')
-
-
 def _get_changes_v1_raw(query, revision=None):
     """Returns the raw versions response.
 


Test patch: