Repository: DDMAL/CantusDB

Issue:
We need some method to track which Cantus IDs have been merged
Directly related to https://github.com/DDMAL/CantusDB/issues/877. For each Chant object in the database, it would be a good idea to keep track of each `cantus_id` it has had in the past. In case there is a problem with Cantus Index's https://cantusindex.org/json-merged-chants API, this would allow us to roll back erroneous changes.

There are several ways I can imagine setting this up.

One way would be to create a `CantusIDMergeEvent` model. I imagine it looking something like:
```python
class CantusIDMergeEvent(BaseModel):
    old_cantus_id = CharField() # or ForeignKey to CantusIdentifier model?
    new_cantus_id = CharField() # or ForeignKey to CantusIdentifier model?
    applied = BoolField() # has the merge been applied yet, y/n
    date_applied = DateTimeField() # if this merge has been applied, when did it happen
    ci_timestamp = CharField() # we've asked Jan to attach a timestamp to each merge event in the
        # json-merged-chants API, which we're hoping we can use as a unique identifier. This would
        # be the time that the merge was submitted on CantusIndex, i.e., when it was posted in the
        # json-merged-chants API
    applied_by = ForeignKey(model=User)

    chants = OneToMany(model=Chant)
    sequences = OneToMany(model=Sequence)
    # when a merge occurs, we use these two fields to record which chants/sequences were affected

    # we could consider adding a apply() and revert() methods, which would switch the `cantus_id` of
    # chant/sequence in the `chants`/`sequences` from `old_cantus_id` to `new_cantus_id` and back

    # N.B.: an apply() method would certainly require more thought - is `chants` calculated when a new
    # merger gets posted to CI's API, or when we commit the merge?
    # In any case, you get the general idea.
```

Since we're worried about Cantus IDs being changed programmatically when new merge events get posted to https://cantusindex.org/json-merged-chants (see https://github.com/DDMAL/CantusDB/pull/1141#issuecomment-1812749551 etc.), it would be easy to create a view that lists all the chants/sequences that would be affected by a given Merge Event, and would allow Debra to approve the merge. I'm hoping this would not be too much work for Debra to press "approve" on CD as well as on CI, and this would ensure that there's always a human in the loop when we're applying changes specified by Cantus Index.

Worth noting: this record will need to be attached to individual Chant/Sequence objects, and not to CantusIdentifier objects themselves (in case we go ahead with implementing them - see https://github.com/DDMAL/CantusDB/pull/1141 and https://github.com/DDMAL/CantusDB/issues/410). In case mutiple IDs get folded into a single ID and we need to roll back a single merge event, we'll need to know which of the chants associated with the new ID needs to be reset to the earlier ID.


Discussion:
This looks very complicated to get right. 

Store the chant ids as strings.
Use Django reversion to store model changes.

If something goes wrong, the history can be seen through the version history and reverted.

You can create a non-model based view that will analyze the output of the merge API and give a list of chants to be updated. Nothing actually needs to be stored in the database for this to happen. Simply query your database for chants matching the old id, and offer to update the chant to the new id. You could do “bulk approve” if you want, otherwise an “update chant” next to each change would serve as an implicit approval (I’m hoping only certain users would have access to this view)
I asked Debra about a bulk approve page for each merger, and she said that it wouldn't work for her workflow.
> Store the chant ids as strings.

i.e., you're suggesting that we do create a CantusIDMergeEvent, but that we keep track of the chants that were affected as a list of strings?

> Use Django reversion to store model changes.

i.e., if we have the list of IDs, a someone could look at all the chants and set them back to what they were before the merge event? This would also reset any other edits that may have been made since, correct?

Resolving patch:
diff --git a/django/cantusdb_project/cantusindex.py b/django/cantusdb_project/cantusindex.py
--- a/django/cantusdb_project/cantusindex.py
+++ b/django/cantusdb_project/cantusindex.py
@@ -6,6 +6,7 @@
 import requests
 from typing import Optional, Union, Callable
 from main_app.models import Genre
+import json
 
 CANTUS_INDEX_DOMAIN: str = "https://cantusindex.uwaterloo.ca"
 DEFAULT_TIMEOUT: float = 2  # seconds
@@ -112,7 +113,7 @@ def get_suggested_chant(
     }
 
 
-def get_suggested_fulltext(cantus_id: str) -> str:
+def get_suggested_fulltext(cantus_id: str) -> Optional[str]:
     endpoint_path: str = f"/json-cid/{cantus_id}"
     json: Union[dict, list, None] = get_json_from_ci_api(endpoint_path)
 
@@ -128,6 +129,41 @@ def get_suggested_fulltext(cantus_id: str) -> str:
     return suggested_fulltext
 
 
+def get_merged_cantus_ids() -> Optional[list]:
+    """Retrieve merged Cantus IDs from the Cantus Index API (/json-merged-chants)
+
+    This function sends a request to the Cantus Index API endpoint for merged chants
+    and retrieves the response. The response is expected to be a list of dictionaries,
+    each containing information about a merged Cantus ID, including the old Cantus ID,
+    the new Cantus ID, and the date of the merge.
+
+    Returns:
+        Optional[list]: A list of dictionaries representing merged chant information,
+    or None if there was an error retrieving the data or the response format is invalid.
+
+    """
+    endpoint_path: str = "/json-merged-chants"
+
+    # We have to use the old CI domain since the API is still not available on
+    # cantusindex.uwaterloo.ca. Once it's available, we can use get_json_from_ci_api
+    # json: Union[dict, list, None] = get_json_from_ci_api(endpoint_path)
+    uri: str = f"https://cantusindex.org{endpoint_path}"
+    try:
+        response: requests.Response = requests.get(uri, timeout=DEFAULT_TIMEOUT)
+    except requests.exceptions.Timeout:
+        return None
+    if not response.status_code == 200:
+        return None
+    response.encoding = "utf-8-sig"
+    raw_text: str = response.text
+    text_without_bom: str = raw_text.encode().decode("utf-8-sig")
+    merge_events: list = json.loads(text_without_bom)
+
+    if not isinstance(merge_events, list):
+        return None
+    return merge_events
+
+
 def get_json_from_ci_api(
     path: str, timeout: float = DEFAULT_TIMEOUT
 ) -> Union[dict, list, None]:
diff --git a/django/cantusdb_project/main_app/management/commands/add_cantus_index_merge_events.py b/django/cantusdb_project/main_app/management/commands/add_cantus_index_merge_events.py
new file mode 100644
--- /dev/null
+++ b/django/cantusdb_project/main_app/management/commands/add_cantus_index_merge_events.py
@@ -0,0 +1,96 @@
+from typing import Optional
+from datetime import datetime
+from django.core.management.base import BaseCommand
+from main_app.models import Chant
+from cantusindex import get_merged_cantus_ids
+
+
+class Command(BaseCommand):
+    help = (
+        "Fetch and apply merge events from the /json-merged-chants API on Cantus Index"
+    )
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "date",
+            nargs="?",
+            type=str,
+            help="Filter merges by date in the format YYYY-MM-DD",
+        )
+
+    def handle(self, *args, **kwargs):
+        date_filter: str = kwargs["date"]
+        if date_filter:
+            try:
+                date_filter = datetime.strptime(date_filter, "%Y-%m-%d")
+            except ValueError:
+                self.stdout.write(
+                    self.style.ERROR("Invalid date format. Please use YYYY-MM-DD.")
+                )
+                return
+
+        # Fetch the merge events from the Cantus Index API
+        merge_events: Optional[list] = get_merged_cantus_ids()
+
+        if not merge_events:
+            self.stdout.write(
+                self.style.ERROR("Failed to fetch data from Cantus Index.")
+            )
+            return
+
+        # Filter merge events by the provided date
+        if date_filter:
+            merge_events = self.filter_merge_events_by_date(merge_events, date_filter)
+
+        # Apply the new merge events on Cantus DB
+        for tx in merge_events:
+            self.apply_transaction(tx)
+
+    def filter_merge_events_by_date(
+        self, merge_events: list, date_filter: datetime
+    ) -> list:
+        filtered_merge_events = []
+        for tx in merge_events:
+            try:
+                tx_date = datetime.strptime(tx["date"], "%Y-%m-%d")
+                if tx_date > date_filter:
+                    filtered_merge_events.append(tx)
+            except ValueError:
+                # Handle invalid date format (usually 0000-00-00)
+                self.stdout.write(
+                    self.style.WARNING(
+                        f"Ignoring merge event with invalid date format: {tx['date']}"
+                    )
+                )
+        return filtered_merge_events
+
+    def apply_transaction(self, transaction: dict) -> None:
+        old_cantus_id: str = transaction["old"]
+        new_cantus_id: str = transaction["new"]
+        if not old_cantus_id or not new_cantus_id:
+            self.stdout.write(
+                self.style.WARNING(
+                    f"Skipping transaction with missing cantus_id. Old: {old_cantus_id}, New: {new_cantus_id}"
+                )
+            )
+            return
+
+        affected_chants = Chant.objects.filter(cantus_id=old_cantus_id)
+        if affected_chants:
+            try:
+                affected_chants.update(cantus_id=new_cantus_id)
+                self.stdout.write(
+                    self.style.SUCCESS(
+                        f"Old Cantus ID: {old_cantus_id} -> New Cantus ID: {new_cantus_id}\n"
+                    )
+                )
+            except Exception as e:
+                self.stdout.write(
+                    self.style.ERROR(
+                        f"An error occurred while updating Chants with old cantus_id {old_cantus_id}: {e}"
+                    )
+                )
+        else:
+            self.stdout.write(
+                self.style.WARNING(f"No Chants found with cantus_id: {old_cantus_id}")
+            )


Test patch: