Repository: DDMAL/CantusDB

Issue:
We need a new API that displays concordances information for all chants in the database
In an email from Jan:

> The intensive process of getting all the data from CD via individual json-cid requests (59.000+) is running already on the 3rd day (and not finished yet) but this will not keep the Cantus data fresh in the Cantus Index API in the long term.
> 
> The solution would be to regularly create a large JSON file export of all the CD chants (with the same fields as in json-cid exports) and make it available as a file to download. An example of such json export is here: https://austriamanus.org/files/concordances-export.json
> This kind of data transfer works also with the MMMO database which has approximately half the amount of data compared to a CD. I believe it would also be the best solution for CD.

This will not be difficult. We can use the code in our `json-con` API, but return all chants rather than filtering them by Cantus ID.

What's a good path for this API to live at? `/json-concordances-export`?


Discussion:
I just tried setting this up along the lines of our existing json-cid API, the only substantial change being I didn't filter by Cantus ID. And go figure, `runserver` times out before our backend can finish processing the request.

A couple of possible solutions come to mind:
- we can occasionally generate the entire file and store it, along the lines of what Jan suggested (i.e., cache the file _in toto_)
- we can create a new `concordances_export` (or something) field on our BaseChant model which we use to store the dictionary objects needed for our `json-cid` and `json-concordances-export` APIs, and keep it updated with a hook in `signals.py` (i.e., cache the important information on individual objects in the database).
  - I'm not certain that this is the case, but I think this could be enough to calculate the entire response in one shot. Either way, it would make both APIs faster, even if we ended up needing to cache an entire file too.

Thoughts on both of these options?
> we can create a new `concordances_export` (or something) field on our BaseChant model which we use to store the dictionary objects needed for our `json-cid` and `json-concordances-export` APIs, and keep it updated with a hook in `signals.py` (i.e., cache the important information on individual objects in the database).

Hm... it occurs to me that the values in `["chantlink"]` and `["srclink"]` will need to be configured properly to use the correct domain, which is something Django gives to us in our views (`request.build_absolute_uri()`), but I don't think it will simply hand us in `signals.py`. So caching these values on chant objects in our database will maybe be a bit tricky.
Have you benchmarked your sql queries to figure out why it’s slow?
A few thoughts:

Your first priority when writing a web service, always and forever, is to return **as fast as possible.** Nothing on the web likes waiting: users, your upstream servers, other machines. You will be bitten by timeouts in inexplicable places if you don’t design your services to return control and close the request as fast as possible.

You also open yourself wide for DDoS if you have slow servers.

If you absolutely need to do things at request time, then it needs to return in 2 seconds or less. And, even if it returns under your development environment in that window, you have to think if it will also do so with a different server, or whether it will continue to do so as your data grows.

If you need to calculate something expensive, but triggered by a request, then you need to set up a queue system where the request simply adds it to a queue and returns with a URL so that clients can check on the progress of their job.

If you can’t do that, you need to cache it offline. With Django this is super easy: write a management script and run it with cron. 

For the URLs you can add a command-line parameter to a manage.py script to vary the ones that get written, if you are going in that direction.

IMO, your data isn’t changing fast enough to justify request-time processing. Dump it out once a day, at most. I don’t know how often Jan harvests this data, but a one day lag time is probably about right.

An added benefit, if your service infrastructure supports it, is that you can put the file somewhere where nginx can serve it directly, and doesn’t need to even bother Django for it.
> Have you benchmarked your sql queries to figure out why it’s slow?

I haven't - the page already doesn't load, and turning on DDT would make it run slower.

But thank you for this comment - it occurs to me that there is at least one place where a `select_related` would help. The main culprit, I suspect, is that for each chant in the database, we are building a dictionary that we then return as part of our JSON response:
https://github.com/DDMAL/CantusDB/blob/3020ef2ae79310dcec35d0c061f7e28ca2fea5c9/django/cantusdb_project/main_app/views/views.py#L597-L635
> the page already doesn't load, and turning on DDT would make it run slower.

For the purposes of development, select only a subset of the output (~100 records) and view the console. That will give you a good idea on how to proceed. 
> If you can’t [return a query quickly or set up a queue system], you need to cache it offline. With Django this is super easy: write a management script and run it with cron.

Question: when you say "cache it offline", where would "offline" be? My first thought (since I already know how to set this up) is to create a `CachedJSONResponse` model and save the JSON to a table in the database. Perhaps I could figure out how to write it to a file and then return the file upon receipt of a request. In a Django app, where would you expect a cached response to be put?

> IMO, your data isn’t changing fast enough to justify request-time processing. Dump it out once a day, at most. I don’t know how often Jan harvests this data, but a one day lag time is probably about right.

Sounds about right.

> An added benefit, if your service infrastructure supports it, is that you can put the file somewhere where nginx can serve it directly, and doesn’t need to even bother Django for it.

Following up on "cach[ing] it offline", we could potentially save the file to a volume (would it be necessary to put it in a volume?), and instruct nginx to serve it from there I suppose.
Sorry -- I wasn't very clear. I just meant it should be cached in a separate process that wasn't triggered by an incoming request. So "offline" meant some server-side process.

Not sure about your setup with nginx, but it should be as simple as putting a JSON file in a place where nginx can access it, and modifying the nginx config to look there to serve an incoming API request. 


Resolving patch:
diff --git a/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py b/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
--- a/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
+++ b/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
@@ -1,6 +1,7 @@
 import ujson
 import os
 from sys import stdout
+from typing import Optional
 from datetime import datetime
 from collections import defaultdict
 from django.db.models.query import QuerySet
@@ -8,10 +9,27 @@
 from main_app.models import Chant
 
 
+# Usage: `python manage.py update_cached_concordances`
+# or `python manage.py update_cached_concordances -d "/path/to/directory/in/which/to/save/concordances"`
+
+
 class Command(BaseCommand):
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "-d",
+            "--directory",
+            help="Optional filepath specifying a directory to output concordances",
+            type=str,
+        )
+
     def handle(self, *args, **kwargs) -> None:
-        CACHE_DIR: str = "api_cache"
-        FILEPATH: str = f"{CACHE_DIR}/concordances.json"
+        cache_dir: Optional[str] = kwargs["directory"]
+        if not cache_dir:
+            # this default directory should match the value in docker-compose.yml,
+            # at services:django:volumes:api_cache_volume
+            cache_dir = "/resources/api_cache"
+
+        filepath: str = f"{cache_dir}/concordances.json"
         start_time: str = datetime.now().isoformat()
         stdout.write(f"Running update_cached_concordances at {start_time}.\n")
         concordances: dict = get_concordances()
@@ -23,22 +41,29 @@ def handle(self, *args, **kwargs) -> None:
             "data": concordances,
             "metadata": metadata,
         }
-        stdout.write(f"Attempting to make directory at {CACHE_DIR} to hold cache: ")
+        stdout.write(f"Attempting to make directory at {cache_dir} to hold cache: ")
         try:
-            os.mkdir(CACHE_DIR)
-            stdout.write(f"successfully created directory at {CACHE_DIR}.\n")
+            os.mkdir(cache_dir)
+            stdout.write(f"successfully created directory at {cache_dir}.\n")
         except FileExistsError:
-            stdout.write(f"directory at {CACHE_DIR} already exists.\n")
-        stdout.write(f"Writing concordances to {FILEPATH} at {write_time}.\n")
-        with open(FILEPATH, "w") as json_file:
+            stdout.write(f"directory at {cache_dir} already exists.\n")
+        stdout.write(f"Writing concordances to {filepath} at {write_time}.\n")
+        with open(filepath, "w") as json_file:
             ujson.dump(data_and_metadata, json_file)
         end_time = datetime.now().isoformat()
         stdout.write(
-            f"Concordances successfully written to {FILEPATH} at {end_time}.\n\n"
+            f"Concordances successfully written to {filepath} at {end_time}.\n\n"
         )
 
 
 def get_concordances() -> dict:
+    """Fetch all published chants in the database, group them by Cantus ID, and return
+    a dictionary containing information on each of these chants.
+
+    Returns:
+        dict: A dictionary where each key is a Cantus ID and each value is a list all
+          published chants in the database with that Cantus ID.
+    """
     DOMAIN: str = "https://cantusdatabase.org"
 
     stdout.write("Querying database for published chants\n")


Test patch: