Repository: DDMAL/CantusDB

Issue:
We need to update our concordances export API for new specification
In an email from Jan, who's working to use our [concordances API](https://github.com/DDMAL/CantusDB/wiki/APIs#concordances) on Cantus Index:

> There is a small issue with parsing your JSON: 
> You have a 'data' element and a Cantus ID element ("004650" in the following snippet) which is not necessary and not compatible with the current CI parser. 
> ```
> ["data":{"004650":[{"siglum":"A-KR VI\/258","srclink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/source\/123660\/","chantlink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/chant\/307346\/","folio":"060r","sequence":11,"incipit":"Rex autem David cooperto","feast":"De Regum","genre":"A","office":"H","position":"9","cantus_id":"004650","image":null,"mode":"?","full_text":null,"melody":null,"db":"CD"},{"siglum":"SI-Lna 19 (olim 18)","srclink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/source\/123659\/","chantlink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/chant\/356859\/","folio":"021v","sequence":2,"incipit":"Rex autem David cooperto","feast":"De Regum","genre":"A","office":"H","position":"10","cantus_id":"004650","image":null,"mode":"8","full_text":null,"melody":null,"db":"CD"},
>```
> 
> Is it possible to remove these elements and keep only what follows?
> ```
> [{"siglum":"A-KR VI\/258","srclink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/source\/123660\/","chantlink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/chant\/307346\/","folio":"060r","sequence":11,"incipit":"Rex autem David cooperto","feast":"De Regum","genre":"A","office":"H","position":"9","cantus_id":"004650","image":null,"mode":"?","full_text":null,"melody":null,"db":"CD"},{"siglum":"SI-Lna 19 (olim 18)","srclink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/source\/123659\/","chantlink":"https:\/\/[cantusdatabase.org](http://cantusdatabase.org/)\/chant\/356859\/","folio":"021v","sequence":2,"incipit":"Rex autem David cooperto","feast":"De Regum","genre":"A","office":"H","position":"10","cantus_id":"004650","image":null,"mode":"8","full_text":null,"melody":null,"db":"CD"},
>```
> 
> I understand that it's possible to develop a unique code for parsing the specific CD data format on my side, but for a sustainable solution, it would be beneficial if all CI-connected databases used the same system and format. 
> 
> Please let me know if you have some serious reasons for including the mentioned elements.

In a resulting email chain, it was noted that if we got rid of `"data"`, it would result in us losing `"metadata"`, the latter of which might be useful. @dchiller noted that we could include this data in a response header rather than including it in the file.

It sounds as if Jan just wants an array of all the chants, in no particular order, and not grouped by Cantus ID. Whatever we decide to do with `"data"` and `"metadata"`, this will be an easy change to make.


Discussion:
If we stop grouping by Cantus ID, perhaps `/concordances` stops being an accurate URI, since it just becomes a list of all published chants in the database. We could implement an updated version without touching what's currently there, and it could live at `/chants/json`? (this could also be a step forward toward implementing #1305, where we could create `/sources/json`, `/genres/json`, etc.)
> If we stop grouping by Cantus ID, perhaps `/concordances` stops being an accurate URI, since it just becomes a list of all published chants in the database.

Hmmm...good point. Maybe we can clarify with Jan that we don't want to group? I agree with your reading of his email request that it looks like he doesn't want grouping, but since it then becomes just a list of all chants I become confused by what we are trying to achieve here...
Necessary next step: clarify with Jan what format he wants. Once we've done that, we can just implement it.
> If we stop grouping by Cantus ID, perhaps `/concordances` stops being an accurate URI, since it just becomes a list of all published chants in the database. We could implement an updated version without touching what's currently there, and it could live at `/chants/json`? (this could also be a step forward toward implementing #1305, where we could create `/sources/json`, `/genres/json`, etc.)

It occurs to me that we will only be including certain information for each chant, whereas we would want a `/chants/json` API to include basically all the information for each chant. So `/concordances` should remain.
Having checked in with Jan, he confirmed that the file should be a simple list - not grouped by Cantus ID, and not within a "data" element:

> I agree with "an array of JSON objects, each representing one chant in the database, and not in any particular order - you don't want them to be grouped by Cantus ID?" - yes. 
> 
> "data" and "metadata" elements - In the case of these very large datasets, parsing would be easier if the data were structured as a single-level array, without any additional elements. 
> 
> Checking the Date/Last-Modified in the response headers is a great idea - I believe we will not need any other metadata!


Current plan for order of operations:
- update our `update_cached_concordances` command, and open PR to merge that change into develop
- test the changes currently on the staging server (#1326)
- deploy the updated API to staging, and show it to Jan for his approval.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py b/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
--- a/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
+++ b/django/cantusdb_project/main_app/management/commands/update_cached_concordances.py
@@ -1,17 +1,17 @@
 import ujson
 import os
 from sys import stdout
-from typing import Optional
 from datetime import datetime
-from collections import defaultdict
 from django.db.models.query import QuerySet
 from django.core.management.base import BaseCommand
 from main_app.models import Chant
 
-
 # Usage: `python manage.py update_cached_concordances`
 # or `python manage.py update_cached_concordances -d "/path/to/directory/in/which/to/save/concordances"`
 
+# Used for creating URIs for chant and source detail pages
+CANTUSDB_DOMAIN: str = "https://cantusdatabase.org"
+
 
 class Command(BaseCommand):
     def add_arguments(self, parser):
@@ -32,13 +32,6 @@ def handle(self, *args, **kwargs) -> None:
         stdout.write(f"Running update_cached_concordances at {start_time}.\n")
         concordances: dict = get_concordances()
         write_time: str = datetime.now().isoformat()
-        metadata: dict = {
-            "last_updated": write_time,
-        }
-        data_and_metadata: dict = {
-            "data": concordances,
-            "metadata": metadata,
-        }
         stdout.write(f"Attempting to make directory at {cache_dir} to hold cache: ")
         try:
             os.mkdir(cache_dir)
@@ -47,22 +40,21 @@ def handle(self, *args, **kwargs) -> None:
             stdout.write(f"directory at {cache_dir} already exists.\n")
         stdout.write(f"Writing concordances to {filepath} at {write_time}.\n")
         with open(filepath, "w") as json_file:
-            ujson.dump(data_and_metadata, json_file)
+            ujson.dump(concordances, json_file)
         end_time = datetime.now().isoformat()
         stdout.write(
             f"Concordances successfully written to {filepath} at {end_time}.\n\n"
         )
 
 
-def get_concordances() -> dict:
-    """Fetch all published chants in the database, group them by Cantus ID, and return
-    a dictionary containing information on each of these chants.
+def get_concordances() -> list[dict]:
+    """Fetch all published chants in the database, and return a list
+    of dictionaries containing information on these chants.
 
     Returns:
-        dict: A dictionary where each key is a Cantus ID and each value is a list all
-          published chants in the database with that Cantus ID.
+        list[dict]: A list of dictionaries, each representing a single chant
+            from the database
     """
-    DOMAIN: str = "https://cantusdatabase.org"
 
     stdout.write("Querying database for published chants\n")
     published_chants: QuerySet[Chant] = Chant.objects.filter(source__published=True)
@@ -89,35 +81,45 @@ def get_concordances() -> dict:
         "volpiano",
     )
 
-    stdout.write("Processing chants\n")
-    concordances: defaultdict = defaultdict(list)
-    for chant in values:
-        source_id: int = chant["source_id"]
-        source_absolute_url: str = f"{DOMAIN}/source/{source_id}/"
-        chant_id: int = chant["id"]
-        chant_absolute_url: str = f"{DOMAIN}/chant/{chant_id}/"
+    concordances: list[dict] = [make_chant_dict(chant) for chant in values]
+    return concordances
 
-        concordances[chant["cantus_id"]].append(
-            {
-                "siglum": chant["source__siglum"],
-                "srclink": source_absolute_url,
-                "chantlink": chant_absolute_url,
-                "folio": chant["folio"],
-                "sequence": chant["c_sequence"],
-                "incipit": chant["incipit"],
-                "feast": chant["feast__name"],
-                "genre": chant["genre__name"],
-                "office": chant["office__name"],
-                "position": chant["position"],
-                "cantus_id": chant["cantus_id"],
-                "image": chant["image_link"],
-                "mode": chant["mode"],
-                "full_text": chant["manuscript_full_text_std_spelling"],
-                "melody": chant["volpiano"],
-                "db": "CD",
-            }
-        )
 
-    stdout.write(f"All chants processed - found {len(concordances)} Cantus IDs\n")
+def make_chant_dict(chant: dict) -> dict:
+    """Given a dictionary representing a chant from the database,
+    return a chant with the keys specified at
+    https://github.com/DDMAL/CantusDB/wiki/APIs#concordances
+
+    Args:
+        chant (dict): A dictionary representing a chant from the database,
+            from a QuerySet.values() call, with several non-standardized
+            keys from database traversals (e.g., `source__siglum`)
+
+    Returns:
+        dict: A dictionary representing a chant, with several values standardized
+            (e.g., `siglum`) and several values calculated (e.g., `srclink`)
+    """
+    source_id: int = chant["source_id"]
+    source_uri: str = f"{CANTUSDB_DOMAIN}/source/{source_id}/"
+    chant_id: int = chant["id"]
+    chant_uri: str = f"{CANTUSDB_DOMAIN}/chant/{chant_id}/"
+    processed_chant: dict = {
+        "siglum": chant["source__siglum"],
+        "srclink": source_uri,
+        "chantlink": chant_uri,
+        "folio": chant["folio"],
+        "sequence": chant["c_sequence"],
+        "incipit": chant["incipit"],
+        "feast": chant["feast__name"],
+        "genre": chant["genre__name"],
+        "office": chant["office__name"],
+        "position": chant["position"],
+        "cantus_id": chant["cantus_id"],
+        "image": chant["image_link"],
+        "mode": chant["mode"],
+        "full_text": chant["manuscript_full_text_std_spelling"],
+        "melody": chant["volpiano"],
+        "db": "CD",
+    }
 
-    return dict(concordances)
+    return processed_chant


Test patch:
diff --git a/django/cantusdb_project/main_app/tests/test_functions.py b/django/cantusdb_project/main_app/tests/test_functions.py
--- a/django/cantusdb_project/main_app/tests/test_functions.py
+++ b/django/cantusdb_project/main_app/tests/test_functions.py
@@ -17,18 +17,13 @@
 class UpdateCachedConcordancesCommandTest(TestCase):
     def test_concordances_structure(self):
         chant: Chant = make_fake_chant(cantus_id="123456")
-        concordances: dict = update_cached_concordances.get_concordances()
+        concordances: list = update_cached_concordances.get_concordances()
 
-        with self.subTest(test="Ensure get_concordances returns dict"):
-            self.assertIsInstance(concordances, dict)
+        with self.subTest(test="Ensure get_concordances returns list"):
+            self.assertIsInstance(concordances, list)
 
-        concordances_for_single_cantus_id: list = concordances["123456"]
-        with self.subTest(test="Ensure values are lists"):
-            self.assertIsInstance(concordances_for_single_cantus_id, list)
-
-        single_concordance = concordances_for_single_cantus_id[0]
+        single_concordance = concordances[0]
         with self.subTest(test="Ensure each concordance is a dict"):
-            single_concordance: dict = concordances_for_single_cantus_id[0]
             self.assertIsInstance(single_concordance, dict)
 
         expected_keys = (
@@ -56,57 +51,31 @@ def test_concordances_structure(self):
         with self.subTest(test="Ensure no unexpected keys present"):
             self.assertEqual(len(concordance_keys), len(expected_keys))
 
-    def test_number_of_concordances_returned(self):
-        cantus_ids: tuple[tuple[str, int]] = (
-            ("000002", 2),
-            ("000003", 3),
-            ("000005", 5),
-            ("000007", 7),
-            ("000011", 11),
-        )
-        for cantus_id, n in cantus_ids:
-            for _ in range(n):
-                make_fake_chant(cantus_id=cantus_id)
-
-        concordances: dict = update_cached_concordances.get_concordances()
-        with self.subTest(test="Test all Cantus IDs present"):
-            self.assertEqual(len(concordances), len(cantus_ids))
-
-        for cantus_id, n in cantus_ids:
-            concordances_for_id: list = concordances[cantus_id]
-            with self.subTest(n=n):
-                self.assertEqual(len(concordances_for_id), n)
-
     def test_published_vs_unpublished(self):
         published_source: Source = make_fake_source(published=True)
         published_chant: Chant = make_fake_chant(
             source=published_source,
-            cantus_id="123456",
-            incipit="chant in a published source",
+            manuscript_full_text_std_spelling="chant in a published source",
         )
         unpublished_source: Source = make_fake_source(published=False)
         unpublished_chant: Chant = make_fake_chant(
             source=unpublished_source,
-            cantus_id="123456",
-            incipit="chant in an unpublished source",
+            manuscript_full_text_std_spelling="chant in an unpublished source",
         )
 
-        concordances: dict = update_cached_concordances.get_concordances()
-        concordances_for_single_id: list = concordances["123456"]
+        concordances: list = update_cached_concordances.get_concordances()
         self.assertEqual(len(concordances), 1)
 
-        single_concordance: dict = concordances_for_single_id[0]
-        expected_incipit: str = published_chant.incipit
-        observed_incipit: str = single_concordance["incipit"]
-        self.assertEqual(expected_incipit, observed_incipit)
+        single_concordance: dict = concordances[0]
+        expected_fulltext: str = published_chant.manuscript_full_text_std_spelling
+        observed_fulltext: str = single_concordance["full_text"]
+        self.assertEqual(expected_fulltext, observed_fulltext)
 
     def test_concordances_values(self):
         chant: Chant = make_fake_chant()
-        cantus_id: str = chant.cantus_id
 
-        concordances: dict = update_cached_concordances.get_concordances()
-        concordances_for_single_id: list = concordances[cantus_id]
-        single_concordance: dict = concordances_for_single_id[0]
+        concordances: list = update_cached_concordances.get_concordances()
+        single_concordance: dict = concordances[0]
 
         expected_items: tuple = (
             ("siglum", chant.source.siglum),