Repository: internetarchive/openlibrary

Issue:
My Books: Borrow History Page
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Background

Please refer to the **`My Books` Development Tutorial** to provide background on which changes may be required:
https://github.com/internetarchive/openlibrary/wiki/Developing-the-Reading-Log

### Problem

At the time of authoring this issue, there was no Archive.org API for fetching a patron's loan history so, instead of rendering a `loan_history` page on Open Library under the **my books** page, we had to link patrons to their archive.org account:

<img width="812" alt="Screenshot 2023-09-12 at 8 59 00 AM" src="https://github.com/internetarchive/openlibrary/assets/978325/ace637bc-fb0a-47cd-b623-82039b4c13b0">

**On Archive.org - My Loan History experience**
1. Going to https://archive.org/account/?tab=loans#loans-history did not switch to history tab (no way to navigate directly)
2. History was not sorted from most recent borrow to least
3. No way to change the sort order
4. No way to search for a title to re-borrow
5. No way to segment by year borrowed
6. No API (I can see) to improve this experience

**The things I'm trying to do are:**
1. See books I've borrowed (preferably recently, to continue)
2. Search for a specific title
3. See whether these titles are available 

### Unofficial Docs

We should eventually create an endpoint `https://archive.org/services/loans/loan/?action=history` which takes s3_secret and s3_access (as POST / headers). 

![image](https://user-images.githubusercontent.com/978325/208777406-3c3662ec-0431-47a8-971f-e9dc65bf6f64.png)

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->
Implement borrow history (private) on Open Library which uses s3 keys + archive.org API

Add search (by title / author), year separators, show recently borrowed first.

## Developer Notes

`/plugins` is a directory containing python code which allows us to define new pages or URLs that we want to register for Open Library. So, imagine we want to create a new page that doesn't exist yet called `/{account-name}/books/loan-history` -- we can register a python function in a plugin that:
1. defines a URL (e.g. `/{account-name}/books/loan-history`) the patron will go
2. defines what data models will be used to fetch the data our page will need
    * our model will likely be the [borrow_history API](https://github.com/internetarchive/openlibrary/pull/8192/files#diff-896c18b3ccfadfa425c7e24a86226343314fce7957d14a8023d3e159156b224bR194-R211)
3. calls and renders an html template, giving it the data we've fetched

## Steps:
1. Add a new plugin for /{account-name}/books/loan-history to the my books: https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/mybooks.py
2. In this plugin function, we're going to fetch the logged in patron, use the borrow_history API to fetch their historical loans
3. Add a new html template into https://github.com/internetarchive/openlibrary/tree/master/openlibrary/templates/account directory for loan_history.html  (you may use loans.html as a reference because its very similar to what we want to do)
4. The plugin will fetch borrow_history API and pass the result into loan_history.html 
    * There may be 1 "gotcha" which is, in templates/account there's a file called mybooks.html which wraps around the loans page and the loan_history page we're going to be creating -- it's what renders the sidebar for the my books page.

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@bfalling 




Discussion:
We've...
1. Built an API for this on archive.org:
https://git.archive.org/ia/petabox/-/merge_requests/3764
2. Updated openlibrary.org APIs to fetch `user_borrow_history`:
https://github.com/internetarchive/openlibrary/pull/8192/files#diff-896c18b3ccfadfa425c7e24a86226343314fce7957d14a8023d3e159156b224bR194-R211:
    * `core.lending.s3_loan_api(action='user_borrow_history')`

The remaining step is to build a basic UI, similar to the `My Already Read` on OpenLibrary.org
Hi @mekarpeles , I have started working on this issue. Can you please assign this to me? Thanks.

Resolving patch:
diff --git a/openlibrary/core/lending.py b/openlibrary/core/lending.py
--- a/openlibrary/core/lending.py
+++ b/openlibrary/core/lending.py
@@ -1,6 +1,6 @@
 """Module for providing core functionality of lending on Open Library.
 """
-from typing import Literal, TypedDict, cast
+from typing import TYPE_CHECKING, Literal, TypedDict, cast
 
 import web
 import datetime
@@ -21,6 +21,11 @@
 from . import ia
 from . import helpers as h
 
+
+if TYPE_CHECKING:
+    from openlibrary.plugins.upstream.models import Edition
+
+
 logger = logging.getLogger(__name__)
 
 S3_LOAN_URL = 'https://%s/services/loans/loan/'
@@ -504,13 +509,36 @@ def get_availability_of_ocaid(ocaid):
     return get_availability('identifier', [ocaid])
 
 
-def get_availability_of_ocaids(ocaids: list[str]) -> dict:
+def get_availability_of_ocaids(ocaids: list[str]) -> dict[str, AvailabilityStatusV2]:
     """
     Retrieves availability based on ocaids/archive.org identifiers
     """
     return get_availability('identifier', ocaids)
 
 
+def get_items_and_add_availability(ocaids: list[str]) -> dict[str, "Edition"]:
+    """
+    Get Editions from OCAIDs and attach their availabiliity.
+
+    Returns a dict of the form: `{"ocaid1": edition1, "ocaid2": edition2, ...}`
+    """
+    ocaid_availability = get_availability_of_ocaids(ocaids=ocaids)
+    editions = web.ctx.site.get_many(
+        [
+            f"/books/{item.get('openlibrary_edition')}"
+            for item in ocaid_availability.values()
+            if item.get('openlibrary_edition')
+        ]
+    )
+
+    # Attach availability
+    for edition in editions:
+        if edition.ocaid in ocaids:
+            edition.availability = ocaid_availability.get(edition.ocaid)
+
+    return {edition.ocaid: edition for edition in editions if edition.ocaid}
+
+
 def is_loaned_out(identifier):
     """Returns True if the given identifier is loaned out.
 
diff --git a/openlibrary/plugins/upstream/account.py b/openlibrary/plugins/upstream/account.py
--- a/openlibrary/plugins/upstream/account.py
+++ b/openlibrary/plugins/upstream/account.py
@@ -2,7 +2,7 @@
 import json
 import logging
 import re
-from typing import Any
+from typing import Any, Final
 from collections.abc import Callable
 from collections.abc import Iterable, Mapping
 
@@ -25,6 +25,10 @@
 from openlibrary.core import helpers as h, lending
 from openlibrary.core.booknotes import Booknotes
 from openlibrary.core.bookshelves import Bookshelves
+from openlibrary.core.lending import (
+    get_items_and_add_availability,
+    s3_loan_api,
+)
 from openlibrary.core.observations import Observations
 from openlibrary.core.ratings import Ratings
 from openlibrary.plugins.recaptcha import recaptcha
@@ -41,9 +45,12 @@
 from openlibrary.plugins.upstream import borrow, forms, utils
 from openlibrary.utils.dateutil import elapsed_time
 
+
 logger = logging.getLogger("openlibrary.account")
 
+CONFIG_IA_DOMAIN: Final = config.get('ia_base_url', 'https://archive.org')
 USERNAME_RETRIES = 3
+RESULTS_PER_PAGE: Final = 25
 
 # XXX: These need to be cleaned up
 send_verification_email = accounts.send_verification_email
@@ -1077,6 +1084,48 @@ def GET(self):
         return delegate.RawText(json.dumps({"loans": loans}))
 
 
+class account_loan_history(delegate.page):
+    path = "/account/loan-history"
+
+    @require_login
+    def GET(self):
+        i = web.input(page=1)
+        page = int(i.page)
+        user = accounts.get_current_user()
+        username = user['key'].split('/')[-1]
+        mb = MyBooksTemplate(username, key='loan_history')
+        loan_history_data = get_loan_history_data(page=page, mb=mb)
+        template = render['account/loan_history'](
+            docs=loan_history_data['docs'],
+            current_page=page,
+            show_next=loan_history_data['show_next'],
+            ia_base_url=CONFIG_IA_DOMAIN,
+        )
+        return mb.render(header_title=_("Loan History"), template=template)
+
+
+class account_loan_history_json(delegate.page):
+    encoding = "json"
+    path = "/account/loan-history"
+
+    @require_login
+    def GET(self):
+        i = web.input(page=1)
+        page = int(i.page)
+        user = accounts.get_current_user()
+        username = user['key'].split('/')[-1]
+        mb = MyBooksTemplate(username, key='loan_history')
+        loan_history_data = get_loan_history_data(page=page, mb=mb)
+        # Ensure all `docs` are `dicts`, as some are `Edition`s.
+        loan_history_data['docs'] = [
+            loan.dict() if not isinstance(loan, dict) else loan
+            for loan in loan_history_data['docs']
+        ]
+        web.header('Content-Type', 'application/json')
+
+        return delegate.RawText(json.dumps({"loans_history": loan_history_data}))
+
+
 class account_waitlist(delegate.page):
     path = "/account/waitlist"
 
@@ -1140,3 +1189,75 @@ def process_goodreads_csv(i):
         else:
             books_wo_isbns[_book['Book Id']] = _book
     return books, books_wo_isbns
+
+
+def get_loan_history_data(page: int, mb: "MyBooksTemplate") -> dict[str, Any]:
+    """
+    Retrieve IA loan history data for page `page` of the patron's history.
+
+    This will use a patron's S3 keys to query the IA loan history API,
+    get the IA IDs, get the OLIDs if available, and and then convert this
+    into editions and IA-only items for display in the loan history.
+
+    This returns both editions and IA-only items because the loan history API
+    includes items that are not in Open Library, and displaying only IA
+    items creates pagination and navigation issues. For further discussion,
+    see https://github.com/internetarchive/openlibrary/pull/8375.
+    """
+    if not (account := OpenLibraryAccount.get(username=mb.username)):
+        raise render.notfound(
+            "Account for not found for %s" % mb.username, create=False
+        )
+    s3_keys = web.ctx.site.store.get(account._key).get('s3_keys')
+    limit = RESULTS_PER_PAGE
+    offset = page * limit - limit
+    loan_history = s3_loan_api(
+        s3_keys=s3_keys,
+        action='user_borrow_history',
+        limit=limit + 1,
+        offset=offset,
+        newest=True,
+    ).json()['history']['items']
+
+    # We request limit+1 to see if there is another page of history to display,
+    # and then pop the +1 off if it's present.
+    show_next = len(loan_history) == limit + 1
+    if show_next:
+        loan_history.pop()
+
+    ocaids = [loan_record['identifier'] for loan_record in loan_history]
+    loan_history_map = {
+        loan_record['identifier']: loan_record for loan_record in loan_history
+    }
+
+    # Get editions and attach their loan history.
+    editions_map = get_items_and_add_availability(ocaids=ocaids)
+    for edition in editions_map.values():
+        edition_loan_history = loan_history_map.get(edition.get('ocaid'))
+        edition['last_loan_date'] = (
+            edition_loan_history.get('updatedate') if edition_loan_history else ''
+        )
+
+    # Create 'placeholders' dicts for items in the Internet Archive loan history,
+    # but absent from Open Library, and then add loan history.
+    # ia_only['loan'] isn't set because `LoanStatus.html` reads it as a current
+    # loan. No apparenty way to distinguish between current and past loans with
+    # this API call.
+    ia_only_loans = [{'ocaid': ocaid} for ocaid in ocaids if ocaid not in editions_map]
+    for ia_only_loan in ia_only_loans:
+        loan_data = loan_history_map[ia_only_loan['ocaid']]
+        ia_only_loan['last_loan_date'] = loan_data.get('updatedate', '')
+        # Determine the macro to load for loan-history items only.
+        ia_only_loan['ia_only'] = True  # type: ignore[typeddict-unknown-key]
+
+    editions_and_ia_loans = list(editions_map.values()) + ia_only_loans
+    editions_and_ia_loans.sort(
+        key=lambda item: item.get('last_loan_date', ''), reverse=True
+    )
+
+    return {
+        'docs': editions_and_ia_loans,
+        'show_next': show_next,
+        'limit': limit,
+        'page': page,
+    }
diff --git a/openlibrary/plugins/upstream/mybooks.py b/openlibrary/plugins/upstream/mybooks.py
--- a/openlibrary/plugins/upstream/mybooks.py
+++ b/openlibrary/plugins/upstream/mybooks.py
@@ -3,17 +3,22 @@
 
 from typing import Final, Literal
 
+from infogami import config
 from infogami.utils import delegate
 from infogami.utils.view import public, safeint, render
 
 from openlibrary.i18n import gettext as _
 
 from openlibrary import accounts
+from openlibrary.accounts.model import OpenLibraryAccount
 from openlibrary.utils import extract_numeric_id_from_olid
 from openlibrary.utils.dateutil import current_year
 from openlibrary.core.booknotes import Booknotes
 from openlibrary.core.bookshelves import Bookshelves
-from openlibrary.core.lending import add_availability, get_loans_of_user
+from openlibrary.core.lending import (
+    add_availability,
+    get_loans_of_user,
+)
 from openlibrary.core.observations import Observations, convert_observation_ids
 from openlibrary.core.sponsorships import get_sponsored_editions
 from openlibrary.core.models import LoggedBooksData


Test patch: