Repository: internetarchive/openlibrary

Issue:
Full Text Search Inside: Excess requests to individual book metadata
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

As seen in the screenshot below, currently, when we do a full text search, we are making archive.org metadata requests for each book. Instead we should make a **single** request to solr or ES to fetch all of these books.

Based on results, we can eliminate performance overhead by ~2/3. 

### Evidence / Screenshot (if possible)

<img width="768" alt="Screenshot 2023-03-23 at 12 25 47 PM" src="https://user-images.githubusercontent.com/978325/227326833-949258a9-845a-42fc-9df3-0572f7e3537d.png">

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual:
* Expected:

### Details

- **Logged in (Y/N)?**
- **Browser type/version?**
- **Operating system?**
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
@JaydenTeoh another one that could be a great fit for Performance improvements!
> @JaydenTeoh another one that could be a great fit for Performance improvements!

Cool! Will add it to the document list, thanks!
👋 @mekarpeles , I'm excited to work on this issue and get hands-on experience with Apache Solr! My plan is to:

- Set up a Solr instance and create a core to store archive metadata.
- Improve search performance by making a single Solr request instead of many archive.org metadata requests.
- Parse the Solr response to display relevant metadata fields as search results.
- Optimize the Solr query and indexing settings for better performance.

Let me know if this works for you, and I'll get started right away! 💻🔍📚

If @JaydenTeoh is already working on this issue, may I please help with anything that may be tedious? This looks like a perfect issue to get started.
@wonhyeongseo I believe this issue would not involving setting up a Solr instance. It would require the batching of requests for multiple IA identifiers via a single request to the endpoint and parsing the response. With that said, although this issue was assigned to me, if you feel that this is the issue that would best kickstart your journey as a volunteer in Open Library, I would be more than glad to let you have a go at it. Welcome to the Open Library community!
I am working on this issue and think `ia.py` should be modified to make this work. I will post a draft PR this weekend at the latest. Thank you for your patience, and I hope you have a wonderful day ☺️ 
@wonhyeongseo May I check if you are still working on this issue? Do you need any help with it?
Hello @JaydenTeoh , I am working on this, but sadly, I can't figure out what to change in the code base. May you please help me? I'm very sorry for the lack of progress.
@mekarpeles May I confirm if it's the Full Text search that is triggering multiple IA loads or is it the Advanced Search feature? I can't seem to see multiple requests to IA for full text search but it is appearing as such for advanced search.
The problem is that https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/fulltext.py#L31-L62 calls format_book_data with an `ocaid` which is causing core.ia.metadata lookups here:
https://github.com/internetarchive/openlibrary/blob/678dd06aa9a084fcdf9a8200237207a3517ea8cb/openlibrary/plugins/openlibrary/home.py#L283-L290 in order to get collections in order to determine availability (which we already have in the code).

**CAUTION** full text search is not the only code that uses `format_book_data` so we want to be careful that fixing full text search doesn't break these other paths.

**Proposal**: Modify `format_book_data` so it can take an extra parameter defaulting to `True` which lets full text search override the `if` statement for whether we fetch collections / ia.get_metadata. Let's investigate whether this block of code is needed anywhere of if it can be removed.

For testing, you may have to load in some borrowable books which you can do using this guide:
https://github.com/internetarchive/openlibrary/wiki/Loading-Production-Book-Data



Resolving patch:
diff --git a/openlibrary/core/fulltext.py b/openlibrary/core/fulltext.py
--- a/openlibrary/core/fulltext.py
+++ b/openlibrary/core/fulltext.py
@@ -56,7 +56,7 @@ def fulltext_search(q, page=1, limit=100, js=False, facets=False):
             if ed.ocaid in ocaids:
                 idx = ocaids.index(ed.ocaid)
                 ia_results['hits']['hits'][idx]['edition'] = (
-                    format_book_data(ed) if js else ed
+                    format_book_data(ed, False) if js else ed
                 )
                 ia_results['hits']['hits'][idx]['availability'] = availability[ed.ocaid]
     return ia_results
diff --git a/openlibrary/plugins/openlibrary/home.py b/openlibrary/plugins/openlibrary/home.py
--- a/openlibrary/plugins/openlibrary/home.py
+++ b/openlibrary/plugins/openlibrary/home.py
@@ -691,7 +691,9 @@ def get_ia_carousel_books(query=None, subject=None, sorts=None, limit=None):
         sorts=sorts,
         query=query,
     )
-    formatted_books = [format_book_data(book) for book in books if book != 'error']
+    formatted_books = [
+        format_book_data(book, False) for book in books if book != 'error'
+    ]
     return formatted_books
 
 
@@ -824,7 +826,7 @@ def format_work_data(work):
     return d
 
 
-def format_book_data(book):
+def format_book_data(book, fetch_availability=True):
     d = web.storage()
     d.key = book.get('key')
     d.url = book.url()
@@ -846,13 +848,14 @@ def get_authors(doc):
     elif d.ocaid:
         d.cover_url = 'https://archive.org/services/img/%s' % d.ocaid
 
-    if d.ocaid:
-        collections = ia.get_metadata(d.ocaid).get('collection', [])
+    if fetch_availability:
+        if d.ocaid:
+            collections = ia.get_metadata(d.ocaid).get('collection', [])
 
-        if 'lendinglibrary' in collections or 'inlibrary' in collections:
-            d.borrow_url = book.url("/borrow")
-        else:
-            d.read_url = book.url("/borrow")
+            if 'lendinglibrary' in collections or 'inlibrary' in collections:
+                d.borrow_url = book.url("/borrow")
+            else:
+                d.read_url = book.url("/borrow")
     return d
 
 


Test patch: