internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: internetarchive/openlibrary

Issue:
Archive another 50k covers to free up storage on server
The goal is to free up enough space on ol-covers0 that there is no longer a nagios alert.

And to have a safe processes for achieving this moving forward. 


Discussion:

Resolving patch:
diff --git a/openlibrary/coverstore/code.py b/openlibrary/coverstore/code.py
--- a/openlibrary/coverstore/code.py
+++ b/openlibrary/coverstore/code.py
@@ -275,9 +275,9 @@ def redirect(id):
             url = zipview_url_from_id(int(value), size)
             raise web.found(url)
 
-        # covers_0008 partials [_00, _23] are tar'd in archive.org items
+        # covers_0008 partials [_00, _80] are tar'd in archive.org items
         if isinstance(value, int) or value.isnumeric():
-            if 8240000 > int(value) >= 8000000:
+            if 8810000 > int(value) >= 8000000:
                 prefix = f"{size.lower()}_" if size else ""
                 pid = "%010d" % int(value)
                 item_id = f"{prefix}covers_{pid[:4]}"


Test patch:
Repository: internetarchive/openlibrary

Issue:
Covers server not returning default author images
Previously it would return a dummy image. Now it return alt text because the endpoint errors?

### Evidence / Screenshot (if possible)

![image](https://user-images.githubusercontent.com/6251786/210642102-bda36372-24d9-4f9a-9e88-e2245c403254.png)


### Relevant url?
eg https://openlibrary.org/search/authors?q=e+h+smith&mode=everything

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to https://covers.openlibrary.org/a/olid/OL291179A-M.jpg

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: `not found` response
* Expected: Empty single pixel image, or default author image? Whatever it used to do in the past.

### Details

- **Logged in (Y/N)?**N
- **Browser type/version?**FF
- **Operating system?**Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@onnotasler



Discussion:
This is also happening with cover images for book titles, i.e. this used to return a default blank image and now returns a 404 error: https://covers.openlibrary.org/b/isbn/9780063031616-L.jpg

Resolving patch:
diff --git a/openlibrary/coverstore/code.py b/openlibrary/coverstore/code.py
--- a/openlibrary/coverstore/code.py
+++ b/openlibrary/coverstore/code.py
@@ -268,7 +268,7 @@ def redirect(id):
             value = self.query(category, key, value)
 
         if not value or (value and safeint(value) in config.blocked_covers):
-            raise web.notfound()
+            return notfound()
 
         # redirect to archive.org cluster for large size and original images whenever possible
         if size in ("L", "") and self.is_cover_in_cluster(value):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Fix Matched Edition Imports: Expand set of accepted fields to update
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

See #7489. Affects fields in #7318.

When we try to import and improve an existing/matched edition in Open Library, the import code seems to be overly selective about which fields are important enough to merit a modification and ignores many of the fields (number_of_pages) we'd like to be added (if missing):

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/catalog/add_book/__init__.py#L797-L822

Support (at minimum) the fields in our import schema:
https://github.com/internetarchive/openlibrary-client/blob/master/olclient/schemata/import.schema.json 

### Evidence / Screenshot (if possible)

Pre-check says we need fields, some import attempts failing to import these new fields, instead only adding source_record:
https://openlibrary.org/books/OL22760588M/Introduction_to_business_information_systems?m=diff&b=4


### Steps to Reproduce
<!-- What steps caused you to find the bug? -->

See #7489

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: editions either matched (instead of modified) or modified but only the `source_record` changed
* Expected: New fields added to these editions

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini @hornc @seabelis @jimchamp @bfalling 


Discussion:

Resolving patch:
diff --git a/openlibrary/catalog/add_book/__init__.py b/openlibrary/catalog/add_book/__init__.py
--- a/openlibrary/catalog/add_book/__init__.py
+++ b/openlibrary/catalog/add_book/__init__.py
@@ -800,14 +800,15 @@ def load(rec, account_key=None):
         need_edition_save = True
 
     # Add list fields to edition as needed
-    edition_fields = [
+    edition_list_fields = [
         'local_id',
         'lccn',
         'lc_classifications',
         'oclc_numbers',
         'source_records',
+        'publishers',
     ]
-    for f in edition_fields:
+    for f in edition_list_fields:
         if f not in rec or not rec[f]:
             continue
         # ensure values is a list
@@ -821,6 +822,17 @@ def load(rec, account_key=None):
         if to_add:
             need_edition_save = True
 
+    other_edition_fields = [
+        'number_of_pages',
+        'publish_date',
+    ]
+    for f in other_edition_fields:
+        if f not in rec or not rec[f]:
+            continue
+        if f not in e:
+            e[f] = rec[f]
+            need_edition_save = True
+
     # Add new identifiers
     if 'identifiers' in rec:
         identifiers = defaultdict(list, e.dict().get('identifiers', {}))


Test patch:
Repository: internetarchive/openlibrary

Issue:
Add sort options to author search results
There should be two options: `Relevance | Work Count | Random` on the UI. The default will now be relevance instead of work count.


Discussion:
I'd love to see an alphabetical option as well. Even if the first iteration isn't sophisticated (i.e. a raw string sort) it would make it a lot easier to spot author duplications.
Hi everyone, I would like to work on this issue if possible? TIA :D 
Hi @DebbieSan , absolutely!

The rough approach will be:

Update https://github.com/internetarchive/openlibrary/blob/15393af2524139f2b041b9db840c060547c1561e/openlibrary/templates/search/sort_options.html to take in a `search_scheme` parameter that defaults to `works`. Then switch the `sort_options` variable depending on whether that parameter is `authors`.

Then add the sort options to the /search/authors template https://github.com/internetarchive/openlibrary/blob/889506b829ed9ea12df855883480505bdfb1e4be/openlibrary/templates/search/authors.html in a similar positioning as on the main search page.

Let me know if you have any questions!
@mheiman I think that would be best tackled out of scope of this one!

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/code.py b/openlibrary/plugins/worksearch/code.py
--- a/openlibrary/plugins/worksearch/code.py
+++ b/openlibrary/plugins/worksearch/code.py
@@ -603,13 +603,14 @@ class author_search(delegate.page):
     def GET(self):
         return render_template('search/authors', self.get_results)
 
-    def get_results(self, q, offset=0, limit=100, fields='*'):
+    def get_results(self, q, offset=0, limit=100, fields='*', sort=''):
         resp = run_solr_query(
             AuthorSearchScheme(),
             {'q': q},
             offset=offset,
             rows=limit,
             fields=fields,
+            sort=sort,
         )
 
         return resp
@@ -620,12 +621,14 @@ class author_search_json(author_search):
     encoding = 'json'
 
     def GET(self):
-        i = web.input(q='', offset=0, limit=100, fields='*')
+        i = web.input(q='', offset=0, limit=100, fields='*', sort='')
         offset = safeint(i.offset, 0)
         limit = safeint(i.limit, 100)
         limit = min(1000, limit)  # limit limit to 1000.
 
-        response = self.get_results(i.q, offset=offset, limit=limit, fields=i.fields)
+        response = self.get_results(
+            i.q, offset=offset, limit=limit, fields=i.fields, sort=i.sort
+        )
         raw_resp = response.raw_resp['response']
         for doc in raw_resp['docs']:
             # SIGH the public API exposes the key like this :(


Test patch:
Repository: internetarchive/openlibrary

Issue:
Update solrupdater/solrbuilder to specify new fields for IA search
IA search now has ratelimiting, and we'll need to specify new parameters / fields when making requests to it. Make sure that this is only done for non-user triggered requests. Those _should_ be rate limited presumably.


Discussion:
Wait, does this imply that you can't rebuild the OpenLibrary search index without special internal access to IA? That would be pretty bad for external developers.

Resolving patch:
diff --git a/openlibrary/solr/data_provider.py b/openlibrary/solr/data_provider.py
--- a/openlibrary/solr/data_provider.py
+++ b/openlibrary/solr/data_provider.py
@@ -143,6 +143,9 @@ async def _get_lite_metadata(ocaids: list[str], _recur_depth=0, _max_recur_depth
                 r = await client.get(
                     "https://archive.org/advancedsearch.php",
                     timeout=30,  # The default is silly short
+                    headers={
+                        'x-application-id': 'ol-solr',
+                    },
                     params={
                         'q': f"identifier:({' OR '.join(ocaids)})",
                         'rows': len(ocaids),
@@ -150,6 +153,7 @@ async def _get_lite_metadata(ocaids: list[str], _recur_depth=0, _max_recur_depth
                         'page': 1,
                         'output': 'json',
                         'save': 'yes',
+                        'service': 'metadata__unlimited',
                     },
                 )
             r.raise_for_status()


Test patch:
Repository: internetarchive/openlibrary

Issue:
Editor should be returned to same URL they started from after saving edit(s).
Since the new book page launch, after saving an edit, the editor is always returned to an edition URL rather than the original URL they started from.
### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...any work url, for example https://openlibrary.org/works/OL20745722W/Raybearer
2. Do ...Click edit. Save.

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: You are returned to an edition url.
* Expected: To be returned to the same url user started from.

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
Related to #3561 

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/addbook.py b/openlibrary/plugins/upstream/addbook.py
--- a/openlibrary/plugins/upstream/addbook.py
+++ b/openlibrary/plugins/upstream/addbook.py
@@ -861,7 +861,7 @@ def GET(self, key):
         return render_template('books/edit', work, edition, recaptcha=get_recaptcha())
 
     def POST(self, key):
-        i = web.input(v=None, _method="GET")
+        i = web.input(v=None, work_key=None, _method="GET")
 
         if spamcheck.is_spam(allow_privileged_edits=True):
             return render_template(
@@ -901,7 +901,12 @@ def POST(self, key):
             else:
                 add_flash_message("info", utils.get_message("flash_book_updated"))
 
-            raise safe_seeother(edition.url())
+            if i.work_key and i.work_key.startswith('/works/'):
+                url = i.work_key
+            else:
+                url = edition.url()
+
+            raise safe_seeother(url)
         except ClientException as e:
             add_flash_message('error', e.args[-1] or e.json)
             return self.GET(key)


Test patch:
Repository: internetarchive/openlibrary

Issue:
Improving the return button and expiry in LoanStatus
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
Go to [testing.openlibrary.org](https://testing.openlibrary.org/) and borrow a book.

<img width="388" alt="image" src="https://user-images.githubusercontent.com/90945854/224814061-eb3643d6-4bf5-4231-bf6e-45c2b4bc5649.png">

Improvements to make:
1. Investigate making the date relative. (under `FormatExpiry.html`)
2. Fix spacing on work page between the "Return eBook" button and the expiry; looks funky. (Under `LoanStatus.html`)
3. After returning book, don't go to /loans page. Return to referrer page.


### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
Please note that this is an extension to #7597, and do reference the comments made in there for more context
### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini 
I will work on this and aim to have a PR before the end of the week!




Discussion:
@cdrini Would you mind clarifying what you mean by "return to referrer page" after returning book?


Resolving patch:
diff --git a/openlibrary/plugins/upstream/borrow.py b/openlibrary/plugins/upstream/borrow.py
--- a/openlibrary/plugins/upstream/borrow.py
+++ b/openlibrary/plugins/upstream/borrow.py
@@ -158,7 +158,7 @@ def POST(self, key):
             stats.increment('ol.loans.return')
             edition.update_loan_status()
             user.update_loan_status()
-            raise web.seeother('/account/loans')
+            raise web.seeother(web.ctx.path)
         elif action == 'join-waitinglist':
             lending.s3_loan_api(edition.ocaid, s3_keys, action='join_waitlist')
             stats.increment('ol.loans.joinWaitlist')


Test patch:
Repository: internetarchive/openlibrary

Issue:
Publisher search endpoint solr performance
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

Our /search/publishers endpoint is doing a strange roll-up and submitting many solr select queries causing performance issues. Solution presumably is to not make more than 1 solr query on /search/publishers.

### Proposal

Change the backend call for /search/publishers to make a single query to solr `publisher:(...)` query.

### Evidence / Screenshot (if possible)
![image](https://user-images.githubusercontent.com/978325/227324438-3fe82d6b-c8b3-4f11-bb51-855898bb62a5.png)
<img width="775" alt="Screenshot 2023-03-23 at 12 18 55 PM" src="https://user-images.githubusercontent.com/978325/227324919-d19b91c5-d19b-4746-9908-43e0f7cf1cbd.png">

### Relevant url?
<!-- `https://openlibrary.org/...` -->

http://testing.openlibrary.org/search/publishers?q=Black%20Dolls%20And%20White%20Dolls%20From%201940%20Through%201970%3A%20Their%20Impact%20Then%20On%20Black%20And%20White%20Children%27s%20Development%20

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

https://github.com/internetarchive/openlibrary/blob/b897c8c51a79308e38f9825fac82864a5cc7d3ae/openlibrary/plugins/worksearch/publishers.py#L82

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/publishers.py b/openlibrary/plugins/worksearch/publishers.py
--- a/openlibrary/plugins/worksearch/publishers.py
+++ b/openlibrary/plugins/worksearch/publishers.py
@@ -57,30 +57,28 @@ class publisher_search(delegate.page):
 
     def GET(self):
         i = web.input(q="")
-        solr = search.get_solr()
-        q = {"publisher": i.q}
-
-        result = solr.select(
-            q,
+        result = search.get_solr().select(
+            {"publisher": i.q, "type": "work"},
             facets=["publisher_facet"],
-            fields=["publisher", "publisher_facet"],
+            facet_mincount=1,
+            facet_limit=25,
+            facet_contains=i.q,
+            facet_contains_ignoreCase='true',
             rows=0,
         )
         result = self.process_result(result)
         return render_template('search/publishers', i.q, result)
 
     def process_result(self, result):
-        solr = search.get_solr()
-
-        def process(p):
-            return web.storage(
+        publisher_facets = result['facets']['publisher_facet']
+        return [
+            web.storage(
                 name=p.value,
                 key="/publishers/" + p.value.replace(" ", "_"),
-                count=solr.select({"publisher_facet": p.value}, rows=0)['num_found'],
+                count=p.count,
             )
-
-        publisher_facets = result['facets']['publisher_facet'][:25]
-        return [process(p) for p in publisher_facets]
+            for p in publisher_facets
+        ]
 
 
 class PublisherEngine(subjects.SubjectEngine):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Enable single-field password entry for email sign-ups
Currently, the email sign-up screen asks the user to create their password and re-enter it for confirmation. This results in more labor and therefore cumbersome UX.

Delete the password confirmation field and instead introduce an eye icon within the password entry field that would allow the user to toggle password visibility on and off.

Design details specified in: https://www.figma.com/file/kiaTbWxdPsyqz6eeaT4M8l/Open-Library---Sign-In-Flow?node-id=73%3A4143&t=gm520FiKGEaKFNWJ-1

Screenshot of the Figma frame: 
<img width="1440" alt="Phase 6" src="https://user-images.githubusercontent.com/99151794/225096314-aa77ba26-6162-472a-a01c-669fe4856f49.png">

This issue belongs to the epic https://github.com/internetarchive/openlibrary/issues/7694


## Code

Here are the files which may need to be updated:

* Here is the front-end html view (the thing the patron sees) where the 2 passwords are implemented on the account creation page. https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/account/create.html#L50-L51
* Here is the logic and rules which help us build the form which gets plugged in to the account creation page view: https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/forms.py#L98-L112
* Here is the back-end code where all the information the patron enters gets submitted upon a registration action (e.g. the passwords will be sent here) https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/account.py#L234-L298



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/forms.py b/openlibrary/plugins/upstream/forms.py
--- a/openlibrary/plugins/upstream/forms.py
+++ b/openlibrary/plugins/upstream/forms.py
@@ -102,15 +102,6 @@ class RegisterForm(Form):
             klass='required',
             validators=[vpass],
         ),
-        Password(
-            'password2',
-            description=_('Confirm password'),
-            klass='required',
-            validators=[
-                vpass,
-                EqualToValidator('password', _("Passwords didn't match.")),
-            ],
-        ),
         Checkbox(
             'ia_newsletter',
             description=_(


Test patch:
Repository: internetarchive/openlibrary

Issue:
Fix git SHA on website + status page (dupe?)
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

The footer + https://openlibrary.org/status should both show the website's git sha. Somewhere down the road we had a regression.

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual:
* Expected:

### Details

- **Logged in (Y/N)?**
- **Browser type/version?**
- **Operating system?**
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:

Resolving patch:
diff --git a/openlibrary/utils/__init__.py b/openlibrary/utils/__init__.py
--- a/openlibrary/utils/__init__.py
+++ b/openlibrary/utils/__init__.py
@@ -193,8 +193,11 @@ def is_number(s):
 
 
 def get_software_version() -> str:
+    """
+    assert get_software_version()  # Should never return a falsy value
+    """
     cmd = "git rev-parse --short HEAD --".split()
-    return run(cmd, text=True).stdout
+    return run(cmd, capture_output=True, text=True).stdout.strip()
 
 
 # See https://docs.python.org/3/library/enum.html#orderedenum


Test patch:
Repository: internetarchive/openlibrary

Issue:
Members of `/usergroup/read-only` can edit covers
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
Members of the `read-only` usergroup are still able to upload cover images and URLs via the manage covers modal.

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
While logged-in as a member of the read-only group:
1. Go to any book page
2. Attempt to add a new cover via the manage covers modal

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: The cover is updated.
* Expected: The update fails gracefully.

### Details

- **Logged in (Y/N)?**Y
- **Browser type/version?**Any
- **Operating system?**Any
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
Check for membership in the read-only group before persisting the new cover.  If submitter is a member, do not store the image.

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->
POST handler for manage cover form: https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/covers.py#L34-L53


### Stakeholders
<!-- @ tag stakeholders of this bug -->
@seabelis 


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/covers.py b/openlibrary/plugins/upstream/covers.py
--- a/openlibrary/plugins/upstream/covers.py
+++ b/openlibrary/plugins/upstream/covers.py
@@ -36,6 +36,10 @@ def POST(self, key):
         if not book:
             raise web.notfound("")
 
+        user = accounts.get_current_user()
+        if user and user.is_read_only():
+            raise web.forbidden(message="Patron not permitted to upload images")
+
         i = web.input(file={}, url="")
 
         # remove references to field storage objects


Test patch: