Repository: DDMAL/CantusDB

Issue:
Chant Edit/Proofread pages - UnboundLocalError
discovered at http://206.12.88.113/edit-chants/123704?pk=316285&folio=003r. Doesn't seem to affect that chant's Detail page, but it is also present at http://206.12.88.113/proofread-chant/123704?pk=316285&folio=003r.


Discussion:
It's because the chant has neither `manuscript_syllabized_full_text`, `manuscript_full_text`, nor `manuscript_full_text_std_spelling`. Both OldCantus and New validate to make sure every chant has at least a `manuscript_full_text_std_spelling`, but I guess this is a case we have to deal with. Currently working to fix the functions in `align_text_mel.py` to account for this edge case.
Current plan of attack:

- [ ] Add type annotations and comments to the functions in `align_text_mel.py` (in progress - the functions in this file are already fairly well commented, but the values returned are quite complex in some cases, so this will help me figure out exactly where things are going wrong and how to fix it).
- [ ] Fix the relevant function(s) (question @annamorphism: if a chant has volpiano but no fulltext, in the text-with-melody preview, we want to display the volpiano without any text beneath it, yes?)
- [ ] Write more exhaustive tests for the functions in the file, and for the Chant Edit and Proofread views.
Hmm. In principle I would say that it's still possible to match the volpiano incipits to the text incipits, which seems to be what was attempted in these cases (the several volumes DK-Kk 3449 8o), but the Volpiano was entered wrong and it doesn't work.  So yes, I think Volpiano with no text is probably the way to go. 
Incidentally, is there any way of finding out how many sources have this problem of not having full texts?

> In principle I would say that it's still possible to match the volpiano incipits to the text incipits, which seems to be what was attempted in these cases (the several volumes DK-Kk 3449 8o), but the Volpiano was entered wrong and it doesn't work. So yes, I think Volpiano with no text is probably the way to go.

Currently, we check syllabized fulltext, then MS spelling fulltext, then STD spelling fulltext. It's certainly possible to add the chant's incipit to this list (as this appears to be what OldCantus does), and then use an empty string as a last resort, but I can leave the incipit out of this list of things to check if you think it would be best.

> Incidentally, is there any way of finding out how many sources have this problem of not having full texts?

I'm just looking into this, and it seems to be surprisingly (even frighteningly) common. I ran the following commands on the staging server, whose data should be basically the same as OldCantus's was a few months ago...

```python
>>> from main_app.models import Chant
>>> chants_w_empty_fulltext = Chant.objects.filter(manuscript_full_text_std_spelling__isnull=True)
>>> chants_w_empty_fulltext.count()
277655
```
Like, that's about half of the chants in the database (!!)

I tried to tally up the sources that contain these chants, and python died rather than completing the calculation:

```python
>>> empty_fulltext_sources = {c.source for c in chants_w_empty_fulltext if not c.manuscript_full_text_std_spelling}
Killed
```
I don't understand why so many chants have no fulltext - OldCantus doesn't let you save a chant if it doesn't have a fulltext, correct? I guess a bunch of these chant records could have originally been from indexes of sources where the researcher only copied down the chant's incipit.

One thing to take from this: this seems like a strong argument not to simply get rid of the `incipit` field and rely on the first few words of a chant's fulltexts, as was floated in #361...

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -86,9 +86,17 @@ def get_context_data(self, **kwargs):
                     chant.manuscript_full_text, pre_syllabized=False
                 )
                 syls_text, syls_melody = postprocess(syls_text, syls_melody)
-            else:
-                syls_text = syllabize_text(chant.incipit, pre_syllabized=False)
+            elif chant.manuscript_full_text_std_spelling:
+                syls_text = syllabize_text(
+                    chant.manuscript_full_text_std_spelling, pre_syllabized=False
+                )
+            elif chant.incipit:
+                syls_text = syllabize_text(
+                    chant.incipit, pre_syllabized=False
+                )
                 syls_text, syls_melody = postprocess(syls_text, syls_melody)
+            else:
+                syls_text = [[""]]
 
             word_zip = align(syls_text, syls_melody)
             context["syllabized_text_with_melody"] = word_zip
@@ -1599,10 +1607,13 @@ def get_chants_with_folios(chants_in_feast):
         chant = self.get_object()
 
         # Preview of melody and text:
-        # in the old CantusDB,
+        # in OldCantus,
         # 'manuscript_syllabized_full_text' exists => preview constructed from 'manuscript_syllabized_full_text'
         # no 'manuscript_syllabized_full_text', but 'manuscript_full_text' exists => preview constructed from 'manuscript_full_text'
         # no 'manuscript_syllabized_full_text' and no 'manuscript_full_text' => preview constructed from 'manuscript_full_text_std_spelling'
+        # to this we add:
+        # no full text of any kind => preview constructed from `incipit`
+        # none of the above => show message explaining why melody preview has no text
 
         if chant.volpiano:
             syls_melody = syllabize_melody(chant.volpiano)
@@ -1619,6 +1630,11 @@ def get_chants_with_folios(chants_in_feast):
             elif chant.manuscript_full_text_std_spelling:
                 syls_text = syllabize_text(chant.manuscript_full_text_std_spelling, pre_syllabized=False)
                 syls_text, syls_melody = postprocess(syls_text, syls_melody)
+            elif chant.incipit:
+                syls_text = syllabize_text(chant.incipit, pre_syllabized=False)
+                syls_text, syls_melody = postprocess(syls_text, syls_melody)
+            else:
+                syls_text = [[""]]
 
             word_zip = align(syls_text, syls_melody)
             context["syllabized_text_with_melody"] = word_zip


Test patch:
diff --git a/django/cantusdb_project/main_app/tests/make_fakes.py b/django/cantusdb_project/main_app/tests/make_fakes.py
--- a/django/cantusdb_project/main_app/tests/make_fakes.py
+++ b/django/cantusdb_project/main_app/tests/make_fakes.py
@@ -69,6 +69,7 @@ def make_fake_chant(source=None,
     cantus_id=None,
     feast=None,
     manuscript_full_text_std_spelling=None,
+    incipit=None,
     manuscript_full_text_std_proofread=None,
     manuscript_full_text=None,
     volpiano=None,
@@ -97,6 +98,8 @@ def make_fake_chant(source=None,
         manuscript_full_text_std_spelling = make_fake_text(
             max_size=MAX_NUMBER_TEXT_CHARS, min_size=MIN_NUMBER_TEXT_CHARS
         )
+    if incipit is None:
+        incipit = manuscript_full_text_std_spelling[0:INCIPIT_LENGTH]
     if manuscript_full_text_std_proofread is None:
         manuscript_full_text_std_proofread = False
     if manuscript_full_text is None:
@@ -127,7 +130,7 @@ def make_fake_chant(source=None,
         chant_range=make_fake_text(LONG_CHAR_FIELD_MAX),
         addendum=make_fake_text(LONG_CHAR_FIELD_MAX),
         manuscript_full_text_std_spelling=manuscript_full_text_std_spelling,
-        incipit=manuscript_full_text_std_spelling[0:INCIPIT_LENGTH],
+        incipit=incipit,
         manuscript_full_text_std_proofread=manuscript_full_text_std_proofread,
         manuscript_full_text=manuscript_full_text,
         manuscript_full_text_proofread=faker.boolean(),
diff --git a/django/cantusdb_project/main_app/tests/test_views.py b/django/cantusdb_project/main_app/tests/test_views.py
--- a/django/cantusdb_project/main_app/tests/test_views.py
+++ b/django/cantusdb_project/main_app/tests/test_views.py
@@ -590,6 +590,38 @@ def test_chant_edit_link(self):
         expected_url_fragment = f"edit-chants/{source.id}?pk={chant.id}&folio={chant.folio}"
 
         self.assertIn(expected_url_fragment, str(response.content))
+    
+    def test_chant_with_volpiano_with_no_fulltext(self):
+        # in the past, a Chant Detail page will error rather than loading properly when the chant has volpiano but no fulltext
+        source = make_fake_source()
+        chant = make_fake_chant(
+            source=source,
+            volpiano="1---c--g--e---e---d---c---c---f---e---e--d---d---c",
+            incipit="somebody"
+        )
+        chant.manuscript_full_text = None
+        chant.manuscript_full_text_std_spelling = None
+        chant.save()
+        response = self.client.get(
+            reverse("chant-detail", args=[chant.id])
+        )
+        self.assertEqual(response.status_code, 200)
+
+    def test_chant_with_volpiano_with_no_incipit(self):
+        # in the past, a Chant Detail page will error rather than loading properly when the chant has volpiano but no fulltext/incipit
+        source = make_fake_source()
+        chant = make_fake_chant(
+            source=source,
+            volpiano="1---g---a---g--a---g---e---c--e---|---f---g--f--g---f--d---9--d",
+        )
+        chant.manuscript_full_text = None
+        chant.manuscript_full_text_std_spelling = None
+        chant.incipit = None
+        chant.save()
+        response = self.client.get(
+            reverse("chant-detail", args=[chant.id])
+        )
+        self.assertEqual(response.status_code, 200)
 
 
 class ChantByCantusIDViewTest(TestCase):
@@ -1513,6 +1545,40 @@ def test_volpiano_signal(self):
         )
         self.assertEqual(chant_2.volpiano, "abacadaeafagahaja")
         self.assertEqual(chant_2.volpiano_intervals, "1-12-23-34-45-56-67-78-8")
+    
+    def test_chant_with_volpiano_with_no_fulltext(self):
+        # in the past, a Chant Edit page will error rather than loading properly when the chant has volpiano but no fulltext
+        source = make_fake_source()
+        chant = make_fake_chant(
+            source=source,
+            volpiano="1---f--e---f--d---e--c---d--d",
+            incipit="dies irae"
+        )
+        chant.manuscript_full_text = None
+        chant.manuscript_full_text_std_spelling = None
+        chant.save()
+        response = self.client.get(
+            reverse('source-edit-chants', args=[source.id]), 
+            {'pk': chant.id}
+        )
+        self.assertEqual(response.status_code, 200)
+
+    def test_chant_with_volpiano_with_no_incipit(self):
+        # in the past, a Chant Edit page will error rather than loading properly when the chant has volpiano but no fulltext/incipit
+        source = make_fake_source()
+        chant = make_fake_chant(
+            source=source,
+            volpiano="1---d---da--g--f--e---e-eg--fed---d-nmn-mn---d",
+        )
+        chant.manuscript_full_text = None
+        chant.manuscript_full_text_std_spelling = None
+        chant.incipit = None
+        chant.save()
+        response = self.client.get(
+            reverse('source-edit-chants', args=[source.id]), 
+            {'pk': chant.id}
+        )
+        self.assertEqual(response.status_code, 200)
 
 
 class ChantProofreadViewTest(TestCase):
@@ -1579,6 +1645,23 @@ def test_proofread_chant(self):
         chant.refresh_from_db()
         self.assertIs(chant.manuscript_full_text_std_proofread, True)
 
+    def test_chant_with_volpiano_with_no_incipit(self):
+        # in the past, a Chant Proofread page will error rather than loading properly when the chant has volpiano but no fulltext/incipit
+        source = make_fake_source()
+        chant = make_fake_chant(
+            source=source,
+            volpiano="1---m---l---k---m---h",
+        )
+        chant.manuscript_full_text = None
+        chant.manuscript_full_text_std_spelling = None
+        chant.incipit = None
+        chant.save()
+        response = self.client.get(
+            reverse('source-edit-chants', args=[source.id]), 
+            {'pk': chant.id}
+        )
+        self.assertEqual(response.status_code, 200)
+
 
 class ChantEditSyllabificationViewTest(TestCase):
     @classmethod