Repository: internetarchive/openlibrary

Issue:
Fix /isbn to retry Affiliate API up to 3 times (w/ 1 sec timeout)
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
Should be fixed in #6332

Right now if one goes to a page like https://openlibrary.org/isbn/9781797210285 and the book isn't in OL, we enqueue a request for it in the Affiliate Server.

However, after enqueued, Open Library will return a Not Found page (and import the book behind the scenes).

What we should do is if one hits the /isbn endpoint directly, that we should assume they're willing for the import to complete and we should retry up to e.g. 3 times.

### Relevant url?
<!-- `https://openlibrary.org/...` -->

https://openlibrary.org/isbn/9781797210285

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

Alternatively, we could have GET parameter which lets people skip the retry (if they're trying to just submit books for import)

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/code.py#L374

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/vendors.py#L340

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@jimchamp @cclauss 



Discussion:
Not sure about what level of overhead this may require but it could be nice to perhaps return a status like "pending" or "failed" when you call that endpoint. Something that could just be stored in memory and only kept around for an hour or something. That's perhaps a nicer approach for many tools than just having much longer response times.
Possibly related to #2205 
The current implementation needs to be fixed / rolled-back because it's spamming the service without exponentially backing-off:

https://github.com/internetarchive/openlibrary/pull/6332/files#diff-94c90a299da1e9c940e56c4d68e767b02163a4c9f49b24133426a77a68444977R331-R336

We should update this code so that we're modifying the source of the call: the affiliate-server `/submit` endpoint so that by default it retries=0 times. The /isbn/{isbn} endpoint sould override to use retries=3 with exponential back-off starting at 1 sec (e.g. 1 sec, 2 sec, 4 sec).

https://github.com/internetarchive/openlibrary/blob/master/scripts/affiliate-server#L188-L202

Resolving patch:
diff --git a/openlibrary/core/models.py b/openlibrary/core/models.py
--- a/openlibrary/core/models.py
+++ b/openlibrary/core/models.py
@@ -365,7 +365,7 @@ def get_ia_download_link(self, suffix):
                 return f"https://archive.org/download/{self.ocaid}/{filename}"
 
     @classmethod
-    def from_isbn(cls, isbn: str):
+    def from_isbn(cls, isbn: str, retry: bool = False):
         """Attempts to fetch an edition by isbn, or if no edition is found,
         attempts to import from amazon
         :rtype: edition|None
@@ -391,8 +391,9 @@ def from_isbn(cls, isbn: str):
                     return web.ctx.site.get(matches[0])
 
         # Attempt to create from amazon, then fetch from OL
+        retries = 5 if retry else 0
         key = (isbn10 or isbn13) and create_edition_from_amazon_metadata(
-            isbn10 or isbn13
+            isbn10 or isbn13, retries=retries
         )
         if key:
             return web.ctx.site.get(key)
diff --git a/openlibrary/core/vendors.py b/openlibrary/core/vendors.py
--- a/openlibrary/core/vendors.py
+++ b/openlibrary/core/vendors.py
@@ -278,7 +278,10 @@ def serialize(product: Any) -> dict:
 
 @public
 def get_amazon_metadata(
-    id_: str, id_type: Literal['asin', 'isbn'] = 'isbn', resources: Any = None
+    id_: str,
+    id_type: Literal['asin', 'isbn'] = 'isbn',
+    resources: Any = None,
+    retries: int = 0,
 ) -> dict | None:
     """Main interface to Amazon LookupItem API. Will cache results.
 
@@ -286,7 +289,9 @@ def get_amazon_metadata(
     :param str id_type: 'isbn' or 'asin'.
     :return: A single book item's metadata, or None.
     """
-    return cached_get_amazon_metadata(id_, id_type=id_type, resources=resources)
+    return cached_get_amazon_metadata(
+        id_, id_type=id_type, resources=resources, retries=retries
+    )
 
 
 def search_amazon(title: str = '', author: str = '') -> dict:  # type: ignore[empty-body]
@@ -303,8 +308,8 @@ def _get_amazon_metadata(
     id_: str,
     id_type: Literal['asin', 'isbn'] = 'isbn',
     resources: Any = None,
-    retries: int = 3,
-    sleep_sec: float = 0.1,
+    retries: int = 0,
+    sleep_sec: float = 1,
 ) -> dict | None:
     """Uses the Amazon Product Advertising API ItemLookup operation to locate a
     specific book by identifier; either 'isbn' or 'asin'.
@@ -409,7 +414,7 @@ def clean_amazon_metadata_for_load(metadata: dict) -> dict:
 
 
 def create_edition_from_amazon_metadata(
-    id_: str, id_type: Literal['asin', 'isbn'] = 'isbn'
+    id_: str, id_type: Literal['asin', 'isbn'] = 'isbn', retries: int = 0
 ) -> str | None:
     """Fetches Amazon metadata by id from Amazon Product Advertising API, attempts to
     create OL edition from metadata, and returns the resulting edition key `/key/OL..M`
@@ -420,7 +425,7 @@ def create_edition_from_amazon_metadata(
     :return: Edition key '/key/OL..M' or None
     """
 
-    md = get_amazon_metadata(id_, id_type=id_type)
+    md = get_amazon_metadata(id_, id_type=id_type, retries=retries)
 
     if md and md.get('product_group') == 'Book':
         with accounts.RunAs('ImportBot'):
diff --git a/openlibrary/plugins/openlibrary/code.py b/openlibrary/plugins/openlibrary/code.py
--- a/openlibrary/plugins/openlibrary/code.py
+++ b/openlibrary/plugins/openlibrary/code.py
@@ -465,8 +465,7 @@ def GET(self, isbn):
             ext += '?' + web.ctx.env['QUERY_STRING']
 
         try:
-            ed = Edition.from_isbn(isbn)
-            if ed:
+            if ed := Edition.from_isbn(isbn, retry=True):
                 return web.found(ed.key + ext)
         except Exception as e:
             logger.error(e)


Test patch: