internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: internetarchive/openlibrary

Issue:
Imports importing 4x
### Summary

- **What is wrong?** At some the Affiliate server imports starting importing the same record 4 times!

<!-- Complete these sections once the problem is resolved. Be brief. -->

- **What caused it?** Unknown

- **What fixed it?** Turning off the affiliate server requests.

- **Followup actions:**
    - Find a proper fix to this
    - Make it easy to turn of affiliate-server imports without having to disable things like prices.

### Steps to close
<!-- For instructions, please see: https://github.com/internetarchive/openlibrary/wiki/Disaster-History-Log#how-to -->
1. [x] Assignment: Is someone assigned to this issue? (notetaker, responder)
2. [x] Labels: Is there an `Affects:` label applied?
3. [x] Diagnosis: Add a description and scope of the issue
4. [x] Updates: As events unfold, is notable **provenance** documented in issue comments? (i.e. useful debug commands / steps / learnings / reference links)
5. [x] "What caused it?" - please answer in summary
6. [x] "What fixed it?" - please answer in summary
7. [x] "Followup actions:" actions added to summary



Discussion:
Note: The only way I eventually got the thing to stop importing was to turn off the link to the imports
The problem may be either:
1. /isbn endpoint in plugins/openlibrary/code.py is attempting imports even though the batch_import is also attempting same import via affiliate-server
2. affiliate-server is not checking what's in the batch prior to re-submitting the import

Because we turned batch-import off and still had a problem, I'd like to guess it's [1] so let's test turning off imports from core/models.py Edition -> core/vendors.py:
https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/vendors.py#L431

Resolving patch:
diff --git a/openlibrary/core/vendors.py b/openlibrary/core/vendors.py
--- a/openlibrary/core/vendors.py
+++ b/openlibrary/core/vendors.py
@@ -322,6 +322,8 @@ def _get_amazon_metadata(
     :param float sleep_sec: Delay time.sleep(sleep_sec) seconds before each retry
     :return: A single book item's metadata, or None.
     """
+    # TMP: This is causing a bunch of duplicate imports
+    return None
     if not affiliate_server_url:
         return None
 


Test patch:
Repository: internetarchive/openlibrary

Issue:
Update solr Docker tag to v8.11.2
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [solr](https://togithub.com/apache/solr) | minor | `8.10.1` -> `8.11.2` |

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/internetarchive/openlibrary).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xNTQuMiIsInVwZGF0ZWRJblZlciI6IjM2LjQwLjMiLCJ0YXJnZXRCcmFuY2giOiJtYXN0ZXIifQ==-->



Discussion:

Resolving patch:
diff --git a/openlibrary/solr/types_generator.py b/openlibrary/solr/types_generator.py
--- a/openlibrary/solr/types_generator.py
+++ b/openlibrary/solr/types_generator.py
@@ -10,7 +10,9 @@ def generate():
     import xml.etree.ElementTree as ET
 
     # read the managed-schema xml file
-    solr_schema = ET.parse(os.path.join(root, '../../conf/solr/conf/managed-schema'))
+    solr_schema = ET.parse(
+        os.path.join(root, '../../conf/solr/conf/managed-schema.xml')
+    )
     python_fields: list[str] = []
     seen_names: set[str] = set()
     for field in solr_schema.getroot().findall('field'):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Unique visitors graph on homepage numbers appear too large
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

### Evidence / Screenshot (if possible)
![image](https://github.com/internetarchive/openlibrary/assets/6251786/4aa4a554-416b-46c8-be0b-4f8d3c704d5c)

### Relevant url?
https://openlibrary.org/

### Proposal & Constraints
Mek thinks these might be off by a factor of something? Drini thinks they might be correct just include a tooooon of requests we weren't including before (images, script tags).

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@mekarpeles @cclauss 


Discussion:

Resolving patch:
diff --git a/openlibrary/core/admin.py b/openlibrary/core/admin.py
--- a/openlibrary/core/admin.py
+++ b/openlibrary/core/admin.py
@@ -116,7 +116,7 @@ def _get_visitor_counts_from_graphite(self, ndays: int = 28) -> list[list[int]]:
         response = requests.get(
             "http://graphite.us.archive.org/render/",
             params={
-                "target": "hitcount(stats.uniqueips.openlibrary, '1d')",
+                "target": "summarize(stats.uniqueips.openlibrary, '1d')",
                 "from": f"-{ndays}days",
                 "tz": "UTC",
                 "format": "json",
@@ -132,7 +132,7 @@ def _get_visitor_counts_from_graphite(self, ndays: int = 28) -> list[list[int]]:
 class VisitorStats(Stats):
     def get_counts(self, ndays: int = 28, times: bool = False) -> list[tuple[int, int]]:
         visitors = _get_visitor_counts_from_graphite(ndays)
-        # Flip the order, convert timestamp to msec and convert count==None to zero
+        # Flip the order, convert timestamp to msec, and convert count==None to zero
         return [
             (int(timestamp * 1000), int(count or 0)) for count, timestamp in visitors
         ]


Test patch:
Repository: internetarchive/openlibrary

Issue:
Lists are not stored in any of the dumps
The full dump on https://openlibrary.org/developers/dumps does not include lists.

### Expectation
Since they are public, they should be included. Note there are concerns because the key for a list includes the user's name (e.g. https://openlibrary.org/people/ScarTissue/lists/OL139466L/Test_Sep_2019 ).

### Proposal & Constraints
Have a separate dump with keys that do not include username; e.g. https://openlibrary.org/people/ScarTissue/lists/OL139466L/Test_Sep_2019 can use as a key `/lists/OL139466L` (maybe)

### Related Files
https://github.com/internetarchive/openlibrary/blob/60175f143dcdc7db15992f29e0ab5cc2d620ab72/scripts/oldump.py

### Stakeholders
@hornc 



Discussion:
Private lists (#3456) and reading logs should **NOT** be included in the dump.

Resolving patch:
diff --git a/openlibrary/data/dump.py b/openlibrary/data/dump.py
--- a/openlibrary/data/dump.py
+++ b/openlibrary/data/dump.py
@@ -45,8 +45,14 @@ def print_dump(json_records, filter=None):
 
         key = web.safestr(d["key"])
 
-        # skip user and admin pages
-        if key.startswith(("/people/", "/admin/")):
+        # skip user pages
+        if key.startswith("/people/") and not re.match(
+            r"^/people/[^/]+/lists/OL\d+L$", key
+        ):
+            continue
+
+        # skip admin pages
+        if key.startswith("/admin/"):
             continue
 
         # skip obsolete pages. Obsolete pages include volumes, scan_records and users
@@ -206,7 +212,13 @@ def split_dump(dump_file=None, format="oldump_%s.txt"):
     """Split dump into authors, editions and works."""
     log(f"split_dump({dump_file}, format={format})")
     start_time = datetime.now()
-    types = ("/type/edition", "/type/author", "/type/work", "/type/redirect")
+    types = (
+        "/type/edition",
+        "/type/author",
+        "/type/work",
+        "/type/redirect",
+        "/type/list",
+    )
     files = {}
     for t in types:
         tname = t.split("/")[-1] + "s"
@@ -236,7 +248,7 @@ def make_index(dump_file):
         if type in ("/type/edition", "/type/work"):
             title = data.get("title", "untitled")
             path = key + "/" + urlsafe(title)
-        elif type == "/type/author":
+        elif type in ("/type/author", "/type/list"):
             title = data.get("name", "unnamed")
             path = key + "/" + urlsafe(title)
         else:


Test patch:
Repository: internetarchive/openlibrary

Issue:
Some user-created books created with lccn=[null], causing rendering error
Unclear how in the flow this is happening, but it's been happening for a while. Likely related to the add book form? Maybe it allows an empty LCCN? Not happening super often though, there were 106 total such errors in the db.

### Evidence / Screenshot (if possible)

### Relevant url?
Eg https://openlibrary.org/books/OL47807201M/Mesopotamian_Coronation_and_Accession_Rites?v=1 (Note since fixed)

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
The import bot also creates empty LCCN entries, but they render okay. https://openlibrary.org/books/OL29452901M/Dracula's_Child?b=2&a=1&_compare=Compare&m=diff 
The two examples above are different. The first has an LCCN array with a single null entry and the second a LC classification array with a single entry of an empty string ("").

https://openlibrary.org/books/OL47807201M.json?v=1
https://openlibrary.org/books/OL29452901M.json?v=1

In `catalog.add_book` **and** `/addbook`, if a record has lccn of [] or [''] or [None] then discard the value? 

One test case is to select an ID (e.g. LCCN) but not to add a value and ensure this empty id doesn't get passed through to `save_many`

Moving forward we may want `/addbook` to also use the `catalog.add_book` path or some sort of safe DRYing of the code.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/models.py b/openlibrary/plugins/upstream/models.py
--- a/openlibrary/plugins/upstream/models.py
+++ b/openlibrary/plugins/upstream/models.py
@@ -331,7 +331,10 @@ def set_identifiers(self, identifiers):
             name, value = id['name'], id['value']
             if name == 'lccn':
                 value = normalize_lccn(value)
-            d.setdefault(name, []).append(value)
+            # `None` in this field causes errors. See #7999.
+            # We should surface to the patron that an invalid LCCN is dropped. See #8092.
+            if value is not None:
+                d.setdefault(name, []).append(value)
 
         # clear existing value first
         for name in names:
@@ -463,8 +466,8 @@ def wp_citation_fields(self):
             }
         )
 
-        if self.lccn:
-            citation['lccn'] = normalize_lccn(self.lccn[0])
+        if self.lccn and (lccn := normalize_lccn(self.lccn[0])):
+            citation['lccn'] = lccn
         if self.get('oclc_numbers'):
             citation['oclc'] = self.oclc_numbers[0]
         citation['ol'] = str(self.get_olid())[2:]


Test patch:
Repository: internetarchive/openlibrary

Issue:
Author dropdown not working as expected
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
Since the last deployment on July 13, the author dropdown on the edit form behaves differently. 
### Evidence / Screenshot (if possible)
<img width="1012" alt="Screenshot 2023-07-13 at 08 35 17" src="https://github.com/internetarchive/openlibrary/assets/17739465/389b1544-9d04-4de1-b218-0145867ec284">


### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...add book or edit book form
2. Do ... try to add Plato as an author

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: the most obvious choice is missing (Plato, the philosopher). Instead, there are authors that have plato as part of the spelling of their names or less prolific authors with the last name Plato.
* Expected: The most likely choice, probably determined by spelling and number of works, should appear on the list.

### Details

- **Logged in (Y/N)?**
- **Browser type/version?**
- **Operating system?**
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini 


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/autocomplete.py b/openlibrary/plugins/worksearch/autocomplete.py
--- a/openlibrary/plugins/worksearch/autocomplete.py
+++ b/openlibrary/plugins/worksearch/autocomplete.py
@@ -113,9 +113,9 @@ def doc_wrap(self, doc: dict):
 class authors_autocomplete(autocomplete):
     path = "/authors/_autocomplete"
     fq = ['type:author']
-    fl = 'key,name,alternate_names,birth_date,death_date,work_count,works,subjects'
+    fl = 'key,name,alternate_names,birth_date,death_date,work_count,top_work,top_subjects'
     olid_suffix = 'A'
-    query = 'name:({q}*) OR alternate_names:({q}*)'
+    query = 'name:({q}*) OR alternate_names:({q}*) OR name:"{q}"^2 OR alternate_names:"{q}"^2'
 
     def doc_wrap(self, doc: dict):
         if 'top_work' in doc:


Test patch:
Repository: internetarchive/openlibrary

Issue:
Languages API endpoint only returns 15 languages
The languages [page](https://openlibrary.org/languages) lists hundreds of languages, but the [JSON API](https://openlibrary.org/languages.json) only returns 15 and neither `limit` nor `offset parameters appear to have any affect.

### Evidence / Screenshot (if possible)

### Relevant url?
https://openlibrary.org/languages
https://openlibrary.org/languages.json?limit=1000
https://openlibrary.org/languages.json?offset=100

It looks like the limit of 15 is intentional, which is *very* odd.
https://github.com/internetarchive/openlibrary/blob/7e197beeca9e4802fc96101d122766e5f52e8265/openlibrary/plugins/worksearch/languages.py#L71-L73



Discussion:
Yep, that's a bad API! :P Here is a workaround in the interim, if needed: http://openlibrary.org/query.json?type=/type/language&name
> a workaround in the interim, if needed: http://openlibrary.org/query.json?type=/type/language&name

Thanks for the hint. I ended up parsing the HTML page, but if I need to do it again before this is fixed, the hint will be useful, although I'll probably use:

http://openlibrary.org/query.json?type=/type/language&name&name_translated

to get access to all the name forms.
I can look into this and get a fix together
@tfmorris You can also add `&*` to fetch all the data fields: http://openlibrary.org/query.json?type=/type/language&*=
@tfmorris @cdrini  I have a working solution on my local. Before I create a PR I wanted to ask if you would want the default limit remain at 15 for no limit/invalid value. 
Yep! 15 should be the default :+1: 

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/languages.py b/openlibrary/plugins/worksearch/languages.py
--- a/openlibrary/plugins/worksearch/languages.py
+++ b/openlibrary/plugins/worksearch/languages.py
@@ -70,7 +70,8 @@ class index_json(delegate.page):
 
     @jsonapi
     def GET(self):
-        return json.dumps(get_top_languages(15))
+        i = web.input(limit=15)
+        return json.dumps(get_top_languages(safeint(i.limit, 15)))
 
 
 class language_search(delegate.page):


Test patch:
Repository: internetarchive/openlibrary

Issue:
last_login field broken since xauthn
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

Since ~2017 when we moved from OL to IA login, we broke `last_login` field in `store` db under `account/{person}`.

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

The logic in https://github.com/internetarchive/openlibrary/blob/master/openlibrary/accounts/model.py#L237-L238 needs to be added to https://github.com/internetarchive/openlibrary/blob/master/openlibrary/accounts/model.py#L895

The solution is a new function which performs...

```
account_key = 'account/{person}'
account = web.ctx.site.get(account_key)
account['last_login'] = datetime.datetime.utcnow().isoformat()
web.ctx.site.store[account_key] = account
```


Discussion:

Resolving patch:
diff --git a/openlibrary/accounts/model.py b/openlibrary/accounts/model.py
--- a/openlibrary/accounts/model.py
+++ b/openlibrary/accounts/model.py
@@ -573,6 +573,13 @@ def save_s3_keys(self, s3_keys):
         web.ctx.site.store[self._key] = _ol_account
         self.s3_keys = s3_keys
 
+    def update_last_login(self):
+        _ol_account = web.ctx.site.store.get(self._key)
+        last_login = datetime.datetime.utcnow().isoformat()
+        _ol_account['last_login'] = last_login
+        web.ctx.site.store[self._key] = _ol_account
+        self.last_login = last_login
+
     @classmethod
     def authenticate(cls, email, password, test=False):
         ol_account = cls.get(email=email, test=test)
@@ -885,6 +892,7 @@ def audit_accounts(
     # web.ctx.site.login method (which requires OL credentials), and directly set an
     # auth_token to enable the user's session.
     web.ctx.conn.set_auth_token(ol_account.generate_login_code())
+    ol_account.update_last_login()
     return {
         'authenticated': True,
         'special_access': getattr(ia_account, 'has_disability_access', False),


Test patch:
Repository: internetarchive/openlibrary

Issue:
DOCKER: git rev-parse --short HEAD --` returns a non-zero exit status 128. web container won't load
### Evidence / Screenshot (if possible)
![image](https://github.com/internetarchive/openlibrary/assets/126498180/a75ff91b-4477-4a98-96b4-4b78c12bb202)

### Steps to Reproduce

1. Follow the instructions from openlibrary/docker/README.md to build a new docker setup
2. Run docker compose up

* Actual:
The errors shown on the screen capture are thrown, and the _web_ container fails to get up.
* Expected:
_web_ container successfully brought up

### Proposal & Constraints
The issue seems to be on the `get_software_version()` function called on `openlibrary/plugins/openlibrary/status.py`, when `git rev-parse --short HEAD --` returns a non-zero exit status 128.

I put the function call between quotation marks as a test, so that "Software version" becomes a hardcoded string, and after that everything seems to load and work just fine.




Discussion:

Resolving patch:
diff --git a/openlibrary/utils/__init__.py b/openlibrary/utils/__init__.py
--- a/openlibrary/utils/__init__.py
+++ b/openlibrary/utils/__init__.py
@@ -2,7 +2,7 @@
 
 from enum import Enum
 import re
-from subprocess import run
+from subprocess import CalledProcessError, run
 from typing import TypeVar, Literal, Optional
 from collections.abc import Iterable, Callable
 
@@ -212,7 +212,10 @@ def get_software_version() -> str:
     assert get_software_version()  # Should never return a falsy value
     """
     cmd = "git rev-parse --short HEAD --".split()
-    return run(cmd, capture_output=True, text=True, check=True).stdout.strip()
+    try:
+        return run(cmd, capture_output=True, text=True, check=True).stdout.strip()
+    except CalledProcessError:
+        return "unknown"
 
 
 # See https://docs.python.org/3/library/enum.html#orderedenum


Test patch:
Repository: internetarchive/openlibrary

Issue:
Invisible reCaptcha
We use recaptcha in a number of places throughout the site. archive.org uses an Invisible recatpcha, which does not prompt the user unless it detect suspicious activity -- we've decided to follow suit.

https://developers.google.com/recaptcha/docs/invisible

**Checklist**
1. The invisible reCAPTCHA requires new keys, so we'll need a new entry in [openlibrary.yml](https://github.com/internetarchive/openlibrary/blob/master/conf/openlibrary.yml) for the keys.  Name the new entry `plugin_invisible_recaptcha`, and model the entry after the existing `plugin_recaptcha` entry (with empty strings as the key values).
2. Update the registration form handler to use the new keys.  These keys are set in the `get_recap` function, [here](https://github.com/internetarchive/openlibrary/blob/f3c7db292127774a2aed5f87448b0239cc2c8cb6/openlibrary/plugins/upstream/account.py#L254-L258).
3. Make the following updates to the account registration template ([here](https://github.com/internetarchive/openlibrary/blob/f3c7db292127774a2aed5f87448b0239cc2c8cb6/openlibrary/templates/account/create.html)):
  - Remove the existing reCAPTCHA, and related copy
  - Add the `g-recaptcha` class to the submit button
  - Add `data-sitekey` to the submit button.  If the form has a reCAPTCHA, use the new public key as the value.  Otherwise, use an empty string.
4. Update the existing submission button click handler, [here](https://github.com/internetarchive/openlibrary/blob/bee11c445c5d2d37f6d91c56b89791bb138e80cd/openlibrary/plugins/openlibrary/js/realtime_account_validation.js#L88-L98),  to programmatically invoke the reCAPTCHA challenge.  Form submission should not be triggered if the challenge was failed.


Discussion:
Hey @mekarpeles , could you assign this task to me please? I would like to work on this issue.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -253,9 +253,10 @@ def get_form(self) -> forms.RegisterForm:
 
     def get_recap(self):
         if self.is_plugin_enabled('recaptcha'):
-            public_key = config.plugin_recaptcha.public_key
-            private_key = config.plugin_recaptcha.private_key
-            return recaptcha.Recaptcha(public_key, private_key)
+            public_key = config.plugin_invisible_recaptcha.public_key
+            private_key = config.plugin_invisible_recaptcha.private_key
+            if public_key and private_key:
+                return recaptcha.Recaptcha(public_key, private_key)
 
     def is_plugin_enabled(self, name):
         return (


Test patch: