Repository: bookwyrm-social/bookwyrm

Issue:
Querying Book Data From Remote Catalogues Causes Slowdown
Currently, requesting book data from remote catalogues (a rather common occurrence considering the size of the local library) causes the site to grind to a halt for all users, especially when done multiple times in a row. Seeing as new users would want to add all their books at once, this isn't that rare of an issue.

On the issue, @mouse-reeve had the following to say:
> oh shoot I've been meaning to fix that. The problem is that as your instance federates, it adds more and more search endpoints, and ends up making more and more http requests every time there's a search. If one of the other instances is slow to respond, the whole thing waits on that. A temporary fix is to go into the connectors admin panel (/admin/bookwyrm/connector/) and unchecked the option for all the connectors besides openlibrary and inventaire
you'll still be sharing data with other servers, just not querying them for search





Discussion:
Some of the potential solutions that I came up with off the top of my head are:

- Having a time limit on the request itself, in which it would stop if waiting for the endpoints takes too long. It'd probably work short-term but I'd imagine this wouldn't solve the issue of the amount of requests scaling exponentially with the network.
- Splitting up the buttons as one for querying Inventaire/OpenLibrary and one for other instances, since most people use the button to access the former. It'd help but I imagine it'd lead users to neglect the other instances' data which kind of defeats the purpose of federation. 
- Maybe some sort of instance following system so instance admins can decide which other instances to query data from? Would require instance administrators to take the initiative though and would probably be a lot of work to implement. 

> * Maybe some sort of instance following system so instance admins can decide which other instances to query data from? Would require instance administrators to take the initiative though and would probably be a lot of work to implement.

This does currently exist (/admin/bookwyrm/connector/), but it's a hassle to use - it's a temporary fix right now, but not _the_ solution
The solution I'm leaning towards is to have only inventaire, openlibrary, and bookwyrm search by default, and let admins optionally enable search to other instances. So new instances wouldn't be automatically added as search connectors, which is what's causing this problem.

To be clear, books will still federate between instances when users' posts reach other instances, so it wouldn't cut off federation overall. I do agree that it makes less use of the neatness of federating book data, though!
Could we possibly have it as a setting during the instance setup, maybe with a disclaimer regarding network speeds? I think that would strike a good balance,

Resolving patch:
diff --git a/bookwyrm/connectors/abstract_connector.py b/bookwyrm/connectors/abstract_connector.py
--- a/bookwyrm/connectors/abstract_connector.py
+++ b/bookwyrm/connectors/abstract_connector.py
@@ -37,7 +37,7 @@ def __init__(self, identifier):
         for field in self_fields:
             setattr(self, field, getattr(info, field))
 
-    def search(self, query, min_confidence=None):
+    def search(self, query, min_confidence=None, timeout=5):
         """free text search"""
         params = {}
         if min_confidence:
@@ -46,6 +46,7 @@ def search(self, query, min_confidence=None):
         data = self.get_search_data(
             "%s%s" % (self.search_url, query),
             params=params,
+            timeout=timeout,
         )
         results = []
 
@@ -218,7 +219,7 @@ def dict_from_mappings(data, mappings):
     return result
 
 
-def get_data(url, params=None):
+def get_data(url, params=None, timeout=10):
     """wrapper for request.get"""
     # check if the url is blocked
     if models.FederatedServer.is_blocked(url):
@@ -234,6 +235,7 @@ def get_data(url, params=None):
                 "Accept": "application/json; charset=utf-8",
                 "User-Agent": settings.USER_AGENT,
             },
+            timeout=timeout,
         )
     except (RequestError, SSLError, ConnectionError) as e:
         logger.exception(e)
@@ -250,7 +252,7 @@ def get_data(url, params=None):
     return data
 
 
-def get_image(url):
+def get_image(url, timeout=10):
     """wrapper for requesting an image"""
     try:
         resp = requests.get(
@@ -258,6 +260,7 @@ def get_image(url):
             headers={
                 "User-Agent": settings.USER_AGENT,
             },
+            timeout=timeout,
         )
     except (RequestError, SSLError) as e:
         logger.exception(e)
diff --git a/bookwyrm/connectors/connector_manager.py b/bookwyrm/connectors/connector_manager.py
--- a/bookwyrm/connectors/connector_manager.py
+++ b/bookwyrm/connectors/connector_manager.py
@@ -1,4 +1,5 @@
 """ interface with whatever connectors the app has """
+from datetime import datetime
 import importlib
 import logging
 import re
@@ -29,9 +30,11 @@ def search(query, min_confidence=0.1, return_first=False):
     isbn = re.sub(r"[\W_]", "", query)
     maybe_isbn = len(isbn) in [10, 13]  # ISBN10 or ISBN13
 
+    timeout = 15
+    start_time = datetime.now()
     for connector in get_connectors():
         result_set = None
-        if maybe_isbn and connector.isbn_search_url and connector.isbn_search_url == "":
+        if maybe_isbn and connector.isbn_search_url and connector.isbn_search_url != "":
             # Search on ISBN
             try:
                 result_set = connector.isbn_search(isbn)
@@ -59,6 +62,8 @@ def search(query, min_confidence=0.1, return_first=False):
                     "results": result_set,
                 }
             )
+        if (datetime.now() - start_time).seconds >= timeout:
+            break
 
     if return_first:
         return None


Test patch: