internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: internetarchive/openlibrary

Issue:
covers.openlibrary.org should hit archive.org/download
Subtask of #3399
Closes #3399 

We can pretty decently simplify our covers.openlibrary.org code, as well remove 1 network request to IA for each cover image, by using the archive.org/download endpoint instead of the server-specific url.

This code: 

https://github.com/internetarchive/openlibrary/blob/ba0b052a64dc3e20d4014743778741bb6ecea874/openlibrary/coverstore/code.py#L191-L196

Makes a request to IA to determine the `server`; we can essentially replace this with just `%(protocol)s://archive.org/download/%(item)s/%(item)s.zip/%(filename)s` (e.g. https://archive.org/download/olcovers259/olcovers259.zip/2598178.jpg ).

This will let us delete the `locate_item` code, which will remove a network request and speed things up (\o/). This will also close #3399 , since the /download endpoint is CORS-enabled.

### Summary of Requirements
- [ ] `zipview_url` is updated to the new url format (above)
- [ ] `zipview_url` no longer calls `locate_item`
- [ ] `locate_item` (and friends) deleted if not used anywhere else

### Stakeholder
@jamesachamp 


Discussion:
Would you be interested in trying this one, @jamesachamp ? Should have a decent impact on performance!
I'll take it up.

Resolving patch:
diff --git a/openlibrary/coverstore/code.py b/openlibrary/coverstore/code.py
--- a/openlibrary/coverstore/code.py
+++ b/openlibrary/coverstore/code.py
@@ -5,7 +5,6 @@
 import json
 import logging
 import os
-import time
 
 import memcache
 import requests
@@ -164,36 +163,10 @@ def trim_microsecond(date):
     return datetime.datetime(*date.timetuple()[:6])
 
 
-@web.memoize
-def get_memcache():
-    servers = config.get("memcache_servers")
-    return memcache.Client(servers)
-
-def _locate_item(item):
-    """Locates the archive.org item in the cluster and returns the server and directory.
-    """
-    print(time.asctime(), "_locate_item", item, file=web.debug)
-    d = requests.get("https://archive.org/metadata/" + item).json()
-    return d['server'], d['dir']
-
-def locate_item(item):
-    mc = get_memcache()
-    if not mc:
-        return _locate_item(item)
-    else:
-        x = mc.get(item)
-        if not x:
-            x = _locate_item(item)
-            print(time.asctime(), "mc.set", item, x, file=web.debug)
-            mc.set(item, x, time=600) # cache it for 10 minutes
-        return x
-
 def zipview_url(item, zipfile, filename):
-    server, dir = locate_item(item)
-
     # http or https
     protocol = web.ctx.protocol
-    return "%(protocol)s://%(server)s/zipview.php?zip=%(dir)s/%(zipfile)s&file=%(filename)s" % locals()
+    return "%(protocol)s://archive.org/download/%(item)s/%(zipfile)s/%(filename)s" % locals()
 
 # Number of images stored in one archive.org item
 IMAGES_PER_ITEM = 10000


Test patch:
Repository: internetarchive/openlibrary

Issue:
504 Timeouts occurring for Patrons w/ several large Lists
e.g. https://openlibrary.org/people/Hermgirl/lists.json

See: https://twitter.com/TheKateFreeman/status/1313314985129377793


Discussion:
A gist with the output of _profile=true: https://gist.github.com/SouthGoingZax/fba883cb94f499dc28cfdc4d0bf2e79c


Resolving patch:
diff --git a/openlibrary/core/lists/model.py b/openlibrary/core/lists/model.py
--- a/openlibrary/core/lists/model.py
+++ b/openlibrary/core/lists/model.py
@@ -244,9 +244,6 @@ def process_all():
 
         return sorted(process_all(), reverse=True, key=lambda s: s["count"])
 
-    def get_top_subjects(self, limit=20):
-        return self._get_all_subjects()[:limit]
-
     def get_subjects(self, limit=20):
         def get_subject_type(s):
             if s.url.startswith("/subjects/place:"):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Updating book notes fails due to KeyError
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
All attempts to update an existing book note cause a KeyError and fail.

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to a book page and open the book notes modal
2. Submit a book note
3. Navigate back to the page from the `/notes.json` page
4. Reopen the modal and submit an updated note

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: Error screen is displayed and trace stack in logs references a KeyError for `work_id`
* Expected: JSON is displayed with a success messafe

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** N/A
- **Operating system?** N/A
- **Environment (prod/dev/local)?** prod

Must be in beta-tester usergroup to use this feature.
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
Add `vars` parameter to `oldb.update()` call in `Booknotes.add()`.

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->
`/openlibrary/core/booknotes.py`

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles 



Discussion:

Resolving patch:
diff --git a/openlibrary/core/booknotes.py b/openlibrary/core/booknotes.py
--- a/openlibrary/core/booknotes.py
+++ b/openlibrary/core/booknotes.py
@@ -110,7 +110,8 @@ def add(cls, username, work_id, notes, edition_id=None):
             'booknotes',
             where="work_id=$work_id AND username=$username",
             notes=notes,
-            edition_id=edition_id
+            edition_id=edition_id,
+            vars=data
         )
 
 


Test patch:
Repository: internetarchive/openlibrary

Issue:
Create a json endpoint for the /account/loans page
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->

using core.lending ia_lending_api in plugins.upstream.accounts.py to display loans as json

```
class account_loans_json(delegate.page):

    encoding = "json"
    path = "/account/loans"

    @require_login
    def GET(self):
        user = accounts.get_current_user()
        user.update_loan_status()
        loans = borrow.get_loans(user)
        return delegate.RawText(json.dumps({
            "loans": loans
        }))
```

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->





Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -1,4 +1,3 @@
-import json
 
 import web
 import logging
@@ -903,6 +902,22 @@ def GET(self):
         loans = borrow.get_loans(user)
         return render['account/borrow'](user, loans)
 
+class account_loans_json(delegate.page):
+
+    encoding = "json"
+    path = "/account/loans"
+
+    @require_login
+    def GET(self):
+        user = accounts.get_current_user()
+        user.update_loan_status()
+        loans = borrow.get_loans(user)
+        web.header('Content-Type', 'application/json')
+        return delegate.RawText(json.dumps({
+            "loans": loans
+        }))
+
+
 # Disabling be cause it prevents account_my_books_redirect from working
 # for some reason. The purpose of this class is to not show the "Create" link for
 # /account pages since that doesn't make any sense.


Test patch:
Repository: internetarchive/openlibrary

Issue:
Import API failing with "startswith first arg must be bytes or a tuple of bytes, not str"
Saw a spike in sentry ( https://sentry.archive.org/sentry/ol-web/issues/16149 ) when ImportBot BWB import was running. Don't know what specific datum caused the issue, but the endpoint was a POST https://openlibrary.org/api/import . The error was `startswith first arg must be bytes or a tuple of bytes, not str` in https://github.com/internetarchive/openlibrary/blob/c8b602677e95ab70d05524f19c6870d4ad800f69/openlibrary/plugins/importapi/code.py#L87

### Relevant url?
- https://openlibrary.org/api/import

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
Unclear

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** Python Requests
- **Operating system?** NA
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
- https://github.com/internetarchive/openlibrary/blob/c8b602677e95ab70d05524f19c6870d4ad800f69/openlibrary/plugins/importapi/code.py#L87

### Stakeholders
@mekarpeles 



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/importapi/code.py b/openlibrary/plugins/importapi/code.py
--- a/openlibrary/plugins/importapi/code.py
+++ b/openlibrary/plugins/importapi/code.py
@@ -84,7 +84,7 @@ def parse_data(data: bytes) -> Tuple[Optional[Dict], Optional[str]]:
             format = 'marcxml'
         else:
             raise DataError('unrecognized-XML-format')
-    elif data.startswith('{') and data.endswith('}'):
+    elif data.startswith(b'{') and data.endswith(b'}'):
         obj = json.loads(data)
         edition_builder = import_edition_builder.import_edition_builder(init_dict=obj)
         format = 'json'


Test patch:
Repository: internetarchive/openlibrary

Issue:
Bug in bookshelves.py add method when changing edition
When calling the add method in openlibrary/core/bookshelves.py (e.g. POSTing to bookshelves.json) with a specific edition_id, the result is not as expected if another edition of the same work already exists on the user's bookshelves. 

### Steps to Reproduce
Submit a POST to bookshelves.json in a work context (e.g. https://openlibrary.org/works/OL460810W/bookshelves.json) with an add action and a specific edition_id: {'action':'add', 'redir':false, 'bookshelf_id':1, 'edition_id':'/books/OL7656518M', 'dont_remove':true}

If the user already has a different edition of the work on any bookshelf:

* If dont_remove is true, no change is made
* If dont_remove is false, the existing edition is removed from the bookshelf

Expected behavior:

The record for the work on the user's bookshelf should be updated to reflect the passed edition_id if dont_remove is true.

### Details

- **Logged in (Y/N)?**  Y
- **Browser type/version?** N/A
- **Operating system?** N/A
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints

Changing the update call at line 189 from

`return oldb.update('bookshelves_books', where=where, bookshelf_id=bookshelf_id, vars=data)`

to 

`return oldb.update('bookshelves_books', where=where, bookshelf_id=bookshelf_id, edition_id=edition_id, vars=data)`

should create the expected behavior without affecting other use cases.

### Stakeholders
@mheiman
@mekarpeles



Discussion:

Resolving patch:
diff --git a/openlibrary/core/bookshelves.py b/openlibrary/core/bookshelves.py
--- a/openlibrary/core/bookshelves.py
+++ b/openlibrary/core/bookshelves.py
@@ -187,7 +187,7 @@ def add(cls, username, bookshelf_id, work_id, edition_id=None):
         else:
             where = "work_id=$work_id AND username=$username"
             return oldb.update('bookshelves_books', where=where,
-                               bookshelf_id=bookshelf_id, vars=data)
+                               bookshelf_id=bookshelf_id, edition_id=edition_id, vars=data)
 
     @classmethod
     def remove(cls, username, work_id, bookshelf_id=None):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Blog posts unavailable on local build, causing failure to fetch web pages
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
On the local build, an error occurred upon connecting to the server which caused the blog posts to not be retrieved and the thus prevented the rest of the web pages to load. Upon further investigation, it was just a single line change which fixed the issue.  
### Evidence / Screenshot (if possible)
![issue_absPathOnElement](https://user-images.githubusercontent.com/46735016/112989967-86736080-9133-11eb-87ed-a8676a8cdc71.png)  

### Relevant url?
<!-- `https://openlibrary.org/...` -->
localhost:8080  

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...  localhost:8080
2. Do ...  nothing

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: Error in screenshot
* Expected: Home web page to load. None of the other web pages loaded either though.

### Details

- **Logged in (Y/N)?** N  
- **Browser type/version?** Firefox  
- **Operating system?** Windows 10 Home  
- **Environment (prod/dev/local)?** Local  
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
Using the solution found [here](https://stackoverflow.com/questions/5501118/python-elementtree-cannot-use-absolute-path-on-element), I will make a PR very shortly to fix the issue.

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->
openlibrary\plugins\upstream\utils.py  

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/utils.py b/openlibrary/plugins/upstream/utils.py
--- a/openlibrary/plugins/upstream/utils.py
+++ b/openlibrary/plugins/upstream/utils.py
@@ -745,7 +745,7 @@ def parse_item(item):
             link=item.find("link").text,
             pubdate=pubdate
         )
-    return [parse_item(item) for item in tree.findall("//item")]
+    return [parse_item(item) for item in tree.findall(".//item")]
 
 _get_blog_feeds = cache.memcache_memoize(_get_blog_feeds, key_prefix="upstream.get_blog_feeds", timeout=5*60)
 


Test patch:
Repository: internetarchive/openlibrary

Issue:
Add name and 'best way to contact you' fields to contact form
Per request by Liz R. in Patron Services, please add two additional fields to contact form as this is a primary way for donors to reach out.
Add the following as optional:

1. patron name
2. best way to contact you

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
https://openlibrary.org/contact

### Stakeholders
<!-- @ tag stakeholders of this bug -->





Discussion:
@seabelis: For the `best way to contact you` field, do you expect patron to request an answer per phone or even per post?
The email address is already requested at the top of the form for the support team to be able to reply.
Wouldn't it be better to add a small paragraph like:

> If you prefer to be contacted by other means please append your contact preference to your question.

or is a particular field required to ease the request management for the support team?
@lephemere For the "best way to contact you" field I would expect either an email or a phone number to contact the patron. You could add that paragraph instead since email is already provided. The required field should be Patron Name as this is what we're using to connect these submissions with donation shipments received with just a name/return address on the label. Thanks for your help!
@seabelis can you please drive prioritization of this one and let us know during some community call what the desired approach is? Struggling to find the bandwidth to include this in my usual triage.

Resolving patch:
diff --git a/openlibrary/plugins/openlibrary/support.py b/openlibrary/plugins/openlibrary/support.py
--- a/openlibrary/plugins/openlibrary/support.py
+++ b/openlibrary/plugins/openlibrary/support.py
@@ -29,6 +29,7 @@ def GET(self):
 
     def POST(self):
         form = web.input()
+        patron_name = form.get("name", "")
         email = form.get("email", "")
         topic = form.get("topic", "")
         description = form.get("question", "")
@@ -95,6 +96,7 @@ def sendmail(from_address, to_address, subject, message):
 URL: %(url)s
 User-Agent: %(useragent)s
 OL-username: %(username)s
+Patron-name: %(patron_name)s
 """
 
 def setup():


Test patch:
Repository: internetarchive/openlibrary

Issue:
Is there a way to limit the page-size of search API requests?
### Question
Is there a way to limit the page-size of search API requests?

The default Search-API page-size is 100 items: http://openlibrary.org/search.json?q=the+lord+of+the+rings

I would like to reduce the page-size (limit) for Search API calls, since the user can just 'page' through the results if he/she wants. Fetching more results also requires more processing on the client-side.

Side notes:
- The number is 20 for the search-inside API: http://openlibrary.org/search/inside.json?q=thanks%20for%20all%20the%20fish
- I think both default page-sizes should probably be the same (20 seems like a reasonable number to me).
- The Archive.org API has the "limit" parameter to do this.

Thanks!




Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/inside/code.py b/openlibrary/plugins/inside/code.py
--- a/openlibrary/plugins/inside/code.py
+++ b/openlibrary/plugins/inside/code.py
@@ -4,7 +4,7 @@
 import web
 
 from infogami.utils import delegate
-from infogami.utils.view import render_template
+from infogami.utils.view import render_template, safeint
 
 from openlibrary.core.fulltext import fulltext_search
 
@@ -12,7 +12,6 @@
 
 
 class search_inside(delegate.page):
-
     path = '/search/inside'
 
     def GET(self):
@@ -25,8 +24,6 @@ def GET(self):
 
         return render_template('search/inside.tmpl', query, results, search_time,
                                page=page, results_per_page=RESULTS_PER_PAGE)
-        page.v2 = True  # page is mobile-first
-        return page
 
 
 class search_inside_json(delegate.page):
@@ -35,7 +32,7 @@ class search_inside_json(delegate.page):
 
     def GET(self):
         i = web.input(q='', page=1, limit=RESULTS_PER_PAGE)
-        limit = min(i.limit, RESULTS_PER_PAGE) if i.limit else RESULTS_PER_PAGE
+        limit = min(safeint(i.limit, RESULTS_PER_PAGE), RESULTS_PER_PAGE)
         query = i.q
         page = int(i.page)
         results = fulltext_search(query, page=page, limit=limit, js=True)


Test patch:
Repository: internetarchive/openlibrary

Issue:
Search API language-filtering does not work currently
The web interface is able to do language filtering, but the search API is currently not. 

As the search API documentation implies you can just add ".json" to the search end-point, combined with the fact that language-filtering is important for many readers, I marked this a bug (instead of a feature request / enhancement).

**Evidence / Screenshot (if possible)**

Example 1:

https://openlibrary.org/search?mode=ebooks&q=romeinen&has_fulltext=true&language=dut (5 results)
https://openlibrary.org/search.json?mode=ebooks&q=romeinen&has_fulltext=true&language=dut (0 results)

Example 2:

https://openlibrary.org/search?mode=ebooks&q=%22Remy%22&has_fulltext=true&language=fre (424 results)
https://openlibrary.org/search.json?mode=ebooks&q=%22Remy%22&has_fulltext=true&language=fre (0 results)

**Proposal & Constraints**

Add the same language-filtering logic as seen in the web-interface to the search API.

There may be other missing Search API filters (I will check them later), but the language filter is quite important IMHO.


Discussion:
I tested other filters available in the search web interface and the following didn't work with the search API as well:
- author_key
- subject_facet
- person_facet
- place_facet
- time_facet
- first_publish_year
- publisher_facet

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/code.py b/openlibrary/plugins/worksearch/code.py
--- a/openlibrary/plugins/worksearch/code.py
+++ b/openlibrary/plugins/worksearch/code.py
@@ -353,7 +353,10 @@ def build_q_list(param):
                 if v:
                     q_list.append("(author_name:(%(name)s) OR author_alternative_name:(%(name)s))" % {'name': v})
 
-        check_params = ['title', 'publisher', 'oclc', 'lccn', 'contribtor', 'subject', 'place', 'person', 'time']
+        check_params = [
+            'title', 'publisher', 'oclc', 'lccn', 'contributor', 'subject', 'place',
+            'person', 'time'
+        ]
         q_list += [
             '%s:(%s)' % (k, re_to_esc.sub(r'\\\g<0>', param[k]))
             for k in check_params if k in param
@@ -1034,7 +1037,15 @@ class search_json(delegate.page):
     encoding = "json"
 
     def GET(self):
-        i = web.input()
+        i = web.input(author_key=[],
+                      subject_facet=[],
+                      person_facet=[],
+                      place_facet=[],
+                      time_facet=[],
+                      first_publish_year=[],
+                      publisher_facet=[],
+                      language=[],
+                      public_scan_b=[])
         if 'query' in i:
             query = json.loads(i.query)
         else:


Test patch:
Repository: internetarchive/openlibrary

Issue:
Add OpenSearch so that open library can easily be searched from FireFox
### Describe the problem that you'd like solved
Right now, chrome automatically adds open library as a search engine using it's [tab search](https://www.chromium.org/tab-to-search) feature.

I like to use Firefox and right now it does not detect search on openlibrary and as such doesn't offer to add OL as a search engine.

### Proposal & Constraints
I propose that we follow [these docs](https://developer.mozilla.org/en-US/docs/Web/OpenSearch) to make it easy to add OL as a search engine.
It seems like the implementation would be just adding one xml file and one line to the head element of OL pages.

### Additional context
At the end we can have something like this
![image](https://user-images.githubusercontent.com/921217/116029743-ac3a3b00-a5f5-11eb-9d8a-957e78967b87.png)

Here are some examples of how this is used:
https://groups.google.com/g/opensearch/c/bSM_Z-Ds7L4

### Stakeholders
<!-- @ tag stakeholders of this bug -->





Discussion:
Go for it, if you'd like to try!!

Resolving patch:
diff --git a/openlibrary/plugins/openlibrary/code.py b/openlibrary/plugins/openlibrary/code.py
--- a/openlibrary/plugins/openlibrary/code.py
+++ b/openlibrary/plugins/openlibrary/code.py
@@ -351,6 +351,18 @@ def GET(self):
             raise web.notfound()
 
 
+class opensearchxml(delegate.page):
+    path = '/opensearch.xml'
+
+    def GET(self):
+        web.header('Content-Type', 'text/plain')
+        try:
+            data = open('static/opensearch.xml').read()
+            raise web.HTTPError('200 OK', {}, data)
+        except IOError:
+            raise web.notfound()
+
+
 class health(delegate.page):
     path = '/health'
     def GET(self):


Test patch: