Repository: internetarchive/openlibrary

Issue:
Trending Books Carousel
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

## Background

For #6014 we created a new Trending page https://openlibrary.org/trending ([code](https://github.com/internetarchive/openlibrary/blob/a67cedf3dcdb417b7980c034984d8f85ad2a9cc3/openlibrary/views/loanstats.py)] and several APIs changes to enable us to query for the most recently logged books (i.e. those added to e.g. one's Want to Read list).

## Goal

The objective of this github issue is to adapt and enhance these APIs (with pagination support) so we can show a Trending or Activity carousel on the homepage.

## Steps

1. Update the `get_most_logged_books` API to allow for an `offset` parameter (to compliment the existing `limit` parameter) https://github.com/internetarchive/openlibrary/blob/a67cedf3dcdb417b7980c034984d8f85ad2a9cc3/openlibrary/views/loanstats.py#L40-L54. This will allow us to load additional pages of results for the carousel.
2. We need to add a new API endpoint to plugins/openlibrary/api.py for getting the `most_logged_books` which takes a `limit` (default 20, max 100), `offset` (default 0), and `since_days` (default 1) which returns json of books.
3. Using the `QueryCarousel` as a template https://github.com/internetarchive/openlibrary/blob/master/openlibrary/macros/QueryCarousel.html create a new macro called a macros/ModelCarousel.html which uses a database model (such as ratings or reading logs). This will call `get_most_logged_books(since_days=7, limit=20, offset=0)` and a `load_more` which in
4. Update the homepage template https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/home/index.html#L22 to include a call to `ModelCarousel` which will show a carousel of the most recently logged books.

## Notes

Closes #6147




Discussion:
@mekarpeles I'm beginner at openlibrary but i have enough development skills to work on enhancement/issue like this so if possible so assign this issue to me, i would like to work on it.
## Background

Currently, the homepage carousels are powered by
https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/home/index.html#L39

```
$:render_template("home/custom_ia_carousel", title=_('Books We Love'), key="staff_picks", query='languageSorter:("English")', subject="openlibrary_staff_picks", sorts=["lending___last_browse desc"], limit=18, test=test)
```

Which under the hood calls:
https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/books/custom_carousel.html

## Step 1

The solution will be to add a new carousel to the templates/home/index.html which looks like:

```
$:render_template("books/custom_carousel", books, title=title, url=url, key=key, min_books=min_books, load_more={"url": load_more_url, "mode": "page", "limit": limit}, test=test, compact_mode=compact_mode)
```

The `books` variable will need to be replaced with a `public` function which is capable of fetching trending books.

Such a function exists here:
https://github.com/internetarchive/openlibrary/blob/master/openlibrary/views/loanstats.py#L40-L54
We'll need to import `from infogami.utils.view import public` add a `@public` decorator above line 40 so this function can be called from within our html templating language

This will accomplish the first half of what we need -- it will produce a carousel on the homepage with 20 trending books.

## Step 2

Eventually, in order to configure the carousel to dynamically load more pages, we'll need to create a new API to fetch more trending books and pass this url in as `load_more_url`.

An example of such a `load_more` API is openlibrary.org/browse.json which is defined in https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/api.py#L64-L89.

We may wish to extend this `browse.json` endpoint to include a `trending` flag like:

```
        i = web.input(
            q='', page=1, limit=100, subject='', work_id='', _type='', sorts='', trending=False
        )
        if i.trending:
            get_most_logged_books(...)
```

But this will require one final step because `get_most_logged_books` currently only returns the first limit (e.g. 20) books. We want to update this function so it can take a limit and a page number, e.g. `get_most_logged_books(limit=20, page=2). This requires us to update the function beneath get_most_logged_books which is called most_logged_books in order to not only have a limit and since but to also have an offset (where the offset will basically be the `limit * page`
https://github.com/internetarchive/openlibrary/blob/07f80c5a8d1c402d4d20d278662a65cf09b5dd8d/openlibrary/core/bookshelves.py#L79-L98

Resolving patch:
diff --git a/openlibrary/core/bookshelves.py b/openlibrary/core/bookshelves.py
--- a/openlibrary/core/bookshelves.py
+++ b/openlibrary/core/bookshelves.py
@@ -76,26 +76,55 @@ def total_unique_users(cls, since=None):
         return results[0] if results else None
 
     @classmethod
-    def most_logged_books(cls, shelf_id=None, limit=10, since=False):
+    def most_logged_books(cls, shelf_id=None, limit=10, since=False, page=1, fetch=False):
         """Returns a ranked list of work OLIDs (in the form of an integer --
         i.e. OL123W would be 123) which have been most logged by
         users. This query is limited to a specific shelf_id (e.g. 1
         for "Want to Read").
         """
+        offset = (page - 1) * limit
         oldb = db.get_db()
         where = 'WHERE bookshelf_id' + ('=$shelf_id' if shelf_id else ' IS NOT NULL ')
         if since:
             where += ' AND created >= $since'
         query = f'select work_id, count(*) as cnt from bookshelves_books {where}'
-        query += ' group by work_id order by cnt desc limit $limit'
+        query += ' group by work_id order by cnt desc limit $limit offset $offset'
         logger.info("Query: %s", query)
         logged_books = list(
             oldb.query(
-                query, vars={'shelf_id': shelf_id, 'limit': limit, 'since': since}
+                query, vars={'shelf_id': shelf_id, 'limit': limit, 'offset': offset, 'since': since}
             )
         )
-        logger.info("Results: %s", logged_books)
-        return logged_books
+        return cls.fetch(logged_books) if fetch else logged_books
+
+    @classmethod
+    def fetch(cls, readinglog_items):
+        """Given a list of readinglog_items, such as those returned by
+        Bookshelves.most_logged_books, fetch the corresponding Open Library
+        book records from solr with availability
+        """
+        from openlibrary.plugins.worksearch.code import get_solr_works
+        from openlibrary.core.lending import get_availabilities
+
+        # This gives us a dict of all the works representing
+        # the logged_books, keyed by work_id
+        work_index = get_solr_works(
+            f"/works/OL{i['work_id']}W"
+            for i in readinglog_items
+        )
+
+        # Loop over each work in the index and inject its availability
+        availability_index = get_availabilities(work_index.values())
+        for work_key in availability_index:
+            work_index[work_key]['availability'] = availability_index[work_key]
+
+        # Return items from the work_index in the order
+        # they are represented by the trending logged books
+        for i, item in enumerate(readinglog_items):
+            key = f"/works/OL{item['work_id']}W"
+            if key in work_index:
+                readinglog_items[i]['work'] = work_index[key]
+        return readinglog_items
 
     @classmethod
     def count_total_books_logged_by_user(cls, username, bookshelf_ids=None):
@@ -185,7 +214,7 @@ def get_users_logged_books(
 
 
     @classmethod
-    def get_recently_logged_books(cls, bookshelf_id=None, limit=50, page=1):
+    def get_recently_logged_books(cls, bookshelf_id=None, limit=50, page=1, fetch=False):
         oldb = db.get_db()
         page = int(page) if page else 1
         data = {
@@ -198,7 +227,8 @@ def get_recently_logged_books(cls, bookshelf_id=None, limit=50, page=1):
             f"SELECT * from bookshelves_books {where} "
             "ORDER BY created DESC LIMIT $limit OFFSET $offset"
         )
-        return list(oldb.query(query, vars=data))
+        logged_books = list(oldb.query(query, vars=data))
+        return cls.fetch(logged_books) if fetch else logged_books
 
 
     @classmethod
diff --git a/openlibrary/plugins/openlibrary/api.py b/openlibrary/plugins/openlibrary/api.py
--- a/openlibrary/plugins/openlibrary/api.py
+++ b/openlibrary/plugins/openlibrary/api.py
@@ -8,7 +8,7 @@
 import re
 import json
 from collections import defaultdict
-
+from openlibrary.views.loanstats import get_trending_books
 from infogami import config
 from infogami.utils import delegate
 from infogami.utils.view import render_template  # noqa: F401 used for its side effects
@@ -60,6 +60,29 @@ def get_book_availability(self, id_type, ids):
             else []
         )
 
+class trending_books_api(delegate.page):
+    path = "/trending(/?.*)"
+    #path = "/trending/(now|daily|weekly|monthly|yearly|forever)"
+    encoding = "json"
+
+    def GET(self, period="/daily"):
+        from openlibrary.views.loanstats import SINCE_DAYS
+        period = period[1:]  # remove slash
+        i = web.input(page=1, limit=100)
+        works = get_trending_books(
+            since_days=SINCE_DAYS[period],
+            limit=int(i.limit),
+            page=int(i.page),
+            books_only=True
+        )
+        result = {
+            'query': f"/trending/{period}",
+            'works': [dict(work) for work in works],
+        }
+        return delegate.RawText(
+            json.dumps(result),
+            content_type="application/json"
+        )
 
 class browse(delegate.page):
     path = "/browse"
@@ -67,7 +90,8 @@ class browse(delegate.page):
 
     def GET(self):
         i = web.input(
-            q='', page=1, limit=100, subject='', work_id='', _type='', sorts=''
+            q='', page=1, limit=100, subject='',
+            work_id='', _type='', sorts=''
         )
         sorts = i.sorts.split(',')
         page = int(i.page)
diff --git a/openlibrary/views/loanstats.py b/openlibrary/views/loanstats.py
--- a/openlibrary/views/loanstats.py
+++ b/openlibrary/views/loanstats.py
@@ -3,7 +3,8 @@
 import web
 from infogami.utils import delegate
 from ..core.lending import get_availabilities
-from ..plugins.worksearch.code import get_solr_works
+
+from infogami.utils.view import public
 
 from ..utils import dateutil
 from .. import app
@@ -12,11 +13,20 @@
 from ..core.bookshelves import Bookshelves
 from ..core.ratings import Ratings
 from ..plugins.admin.code import get_counts
-
+from ..plugins.worksearch.code import get_solr_works
 
 LENDING_TYPES = '(libraries|regions|countries|collections|subjects|format)'
 
 
+SINCE_DAYS = {
+    'now': 0,
+    'daily': 1,
+    'weekly': 7,
+    'monthly': 30,
+    'yearly': 365,
+    'forever': None,
+}
+
 def reading_log_summary():
     # enable to work w/ cached
     if 'env' not in web.ctx:
@@ -32,27 +42,40 @@ def reading_log_summary():
     reading_log_summary, 'stats.readling_log_summary', timeout=dateutil.HOUR_SECS
 )
 
-def cached_get_most_logged_books(shelf_id=None, since_days=1, limit=20):
-    return cache.memcache_memoize(
-        get_most_logged_books, 'stats.trending', timeout=dateutil.HOUR_SECS
-    )(shelf_id=shelf_id, since_days=since_days, limit=limit)
-
-def get_most_logged_books(shelf_id=None, since_days=1, limit=20):
-    """
-    shelf_id: Bookshelves.PRESET_BOOKSHELVES['Want to Read'|'Already Read'|'Currently Reading']
-    since: DATE_ONE_YEAR_AGO, DATE_ONE_MONTH_AGO, DATE_ONE_WEEK_AGO, DATE_ONE_DAY_AGO
-    """
-    # enable to work w/ cached
-    if 'env' not in web.ctx:
-        delegate.fakeload()
+@public
+def get_trending_books(since_days=1, limit=18, page=1, books_only=False):
+    logged_books = (
+        Bookshelves.fetch(get_activity_stream(limit=limit, page=page))  # i.e. "now"
+        if since_days == 0 else
+        Bookshelves.most_logged_books(
+            since=dateutil.date_n_days_ago(since_days),
+            limit=limit,
+            page=page,
+            fetch=True)
+    )
+    return (
+        [book['work'] for book in logged_books if book.get('work')]
+        if books_only else logged_books
+    )
 
-    # Return as dict to enable cache serialization
-    return [dict(book) for book in
-            Bookshelves.most_logged_books(
-                shelf_id=shelf_id,
-                since=dateutil.date_n_days_ago(since_days),
-                limit=limit)]
 
+def cached_get_most_logged_books(shelf_id=None, since_days=1, limit=20, page=1):
+    def get_cachable_trending_books(shelf_id=None, since_days=1, limit=20, page=1):
+        # enable to work w/ cached
+        if 'env' not in web.ctx:
+            delegate.fakeload()
+        # Return as dict to enable cache serialization
+        return [dict(book) for book in
+                Bookshelves.most_logged_books(
+                    shelf_id=shelf_id,
+                    since=dateutil.date_n_days_ago(since_days),
+                    limit=limit,
+                    page=page
+                )]
+    return cache.memcache_memoize(
+        get_cachable_trending_books, 'stats.trending',
+        timeout=dateutil.HOUR_SECS
+    )(shelf_id=shelf_id, since_days=since_days, limit=limit, page=page)
 
 def reading_log_leaderboard(limit=None):
     # enable to work w/ cached
@@ -108,11 +131,11 @@ class lending_stats(app.view):
     def GET(self, key, value):
         raise web.seeother("/")
 
-def get_activity_stream(limit=None):
+def get_activity_stream(limit=None, page=1):
     # enable to work w/ cached
     if 'env' not in web.ctx:
         delegate.fakeload()
-    return Bookshelves.get_recently_logged_books(limit=limit)
+    return Bookshelves.get_recently_logged_books(limit=limit, page=page)
 
 def get_cached_activity_stream(limit):
     return cache.memcache_memoize(
@@ -127,29 +150,22 @@ class activity_stream(app.view):
     def GET(self, page=''):
         if not page:
             raise web.seeother("/trending/now")
-        page = page[1:]
+        page = page[1:]  # remove slash
         limit = 20
         if page == "now":
-            logged_books = get_activity_stream(limit=limit)
+            logged_books = Bookshelves.fetch(get_activity_stream(limit=limit))
         else:
             shelf_id = None  # optional; get from web.input()?
-            logged_books = cached_get_most_logged_books(since_days={
-                'daily': 1,
-                'weekly': 7,
-                'monthly': 30,
-                'yearly': 365,
-                'forever': None,
-            }[page], limit=limit)
-
-        work_index = get_solr_works(f"/works/OL{book['work_id']}W" for book in logged_books)
-        availability_index = get_availabilities(work_index.values())
-        for work_key in availability_index:
-            work_index[work_key]['availability'] = availability_index[work_key]
-        for i, logged_book in enumerate(logged_books):
-            key = f"/works/OL{logged_book['work_id']}W"
-            if key in work_index:
-                logged_books[i]['work'] = work_index[key]
-        return app.render_template("trending", logged_books=logged_books, mode=page)
+            logged_books = Bookshelves.fetch(
+                cached_get_most_logged_books(
+                    since_days=SINCE_DAYS[page],
+                    limit=limit)
+            )
+        return app.render_template(
+            "trending",
+            logged_books=logged_books,
+            mode=page
+        )
 
 
 class readinglog_stats(app.view):


Test patch: