Repository: DDMAL/CantusDB

Issue:
sort headers don't work 
When in chant search, the headers of each column LOOK like you can sort by them, and clicking on them gives you a dutiful little arrow pointing up or down. It doesn't, however, actually sort anything:

<img width="1086" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/57175726-095d-4631-9cec-609c5ac8ef51">

https://cantusdatabase.org/chant-search/?search_bar=notum%20fecit&order=office&sort=desc


Discussion:
Looking into this a bit now, this seems to work in some contexts, but not others?

- https://cantusdatabase.org/chant-search/?op=contains&keyword=principium%20et (query returns 43 results, so they all fit on one page) - All the column headers seem to work as expected (though I can't test the Genre column header for this search, as all the results are antiphons).
- However, looking at a search with more results https://cantusdatabase.org/chant-search/?op=contains&keyword=princip, something weird is going on. Sorting by melody, we get two melodyless chants appearing up top when melodyful chants should be at the top of the list: https://cantusdatabase.org/chant-search/?op=contains&keyword=princip&order=has_melody&sort=asc

Something is certainly still messed up in the initial Notum Fecit search you posted, @annamorphism.
I think I've identified why @annamorphism's initial search using https://cantusdatabase.org/chant-search/?search_bar=notum%20fecit&order=office&sort=desc didn't work.

If a user searches for chants using the global search bar (which adds the `?search_bar=` parameter), the logic that filters the queryset using the column headers (Line 388) is completely skipped:
https://github.com/DDMAL/CantusDB/blob/fcf31afe26527180e7ede0afa80e8bb760830806/django/cantusdb_project/main_app/views/chant.py#L346-L392

So if the user uses the "keyword search" on the chant search page (which adds `?op=contains&keyword=` to the URL) then everything works as expected.

> However, looking at a search with more results https://cantusdatabase.org/chant-search/?op=contains&keyword=princip, something weird is going on. Sorting by melody, we get two melodyless chants appearing up top when melodyful chants should be at the top of the list: https://cantusdatabase.org/chant-search/?op=contains&keyword=princip&order=has_melody&sort=asc

In my mind, sorting by ascending order makes sense to place the melodyless chants first, however, I think that most people would expect the descending order to be the default behaviour when the melody header is clicked.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -334,14 +334,14 @@ def get_context_data(self, **kwargs) -> dict:
         return context
 
     def get_queryset(self) -> QuerySet:
-        # if user has just arrived on the Chant Search page, there will be no
-        # GET parameters.
+        # if user has just arrived on the Chant Search page, there will be no GET parameters.
         if not self.request.GET:
             return Chant.objects.none()
 
         # Create a Q object to filter the QuerySet of Chants
         q_obj_filter = Q()
         display_unpublished = self.request.user.is_authenticated
+
         # if the search is accessed by the global search bar
         if self.request.GET.get("search_bar"):
             if display_unpublished:
@@ -383,13 +383,10 @@ def get_queryset(self) -> QuerySet:
                     *CHANT_SEARCH_TEMPLATE_VALUES
                 )
                 queryset = chant_set.union(sequence_set, all=True)
-            queryset = queryset.order_by("source__siglum", "id")
-
         else:
             # The field names should be keys in the "GET" QueryDict if the search button has been
-            # clicked, even if the user put nothing into the search form and hit "apply"
-            # immediately. In that case, we return the all chants + seqs filtered by the search
-            # form. For every GET parameter other than incipit, add to the Q object
+            # clicked, even if the user put nothing into the search form and hit "apply" immediately.
+            # In that case, we return all chants + seqs filtered by the search form.
             if self.request.GET.get("office"):
                 office_id = self.request.GET.get("office")
                 q_obj_filter &= Q(office__id=office_id)
@@ -412,38 +409,9 @@ def get_queryset(self) -> QuerySet:
                     q_obj_filter &= Q(volpiano__isnull=False)
             if self.request.GET.get("feast"):
                 feast = self.request.GET.get("feast")
-                # This will match any feast whose name contains the feast parameter
-                # as a substring
+                # This will match any feast whose name contains the feast parameter as a substring
                 feasts = Feast.objects.filter(name__icontains=feast)
                 q_obj_filter &= Q(feast__in=feasts)
-            order_get_param: Optional[str] = self.request.GET.get("order")
-            sort_get_param: Optional[str] = self.request.GET.get("sort")
-
-            order_param_options = (
-                "incipit",
-                "office",
-                "genre",
-                "cantus_id",
-                "mode",
-                "has_fulltext",
-                "has_melody",
-                "has_image",
-            )
-            if order_get_param in order_param_options:
-                if order_get_param == "has_fulltext":
-                    order = "manuscript_full_text"
-                elif order_get_param == "has_melody":
-                    order = "volpiano"
-                elif order_get_param == "has_image":
-                    order = "image_link"
-                else:
-                    order = order_get_param
-            else:
-                order = "source__siglum"
-
-            # sort values: "asc" and "desc". Default is "asc"
-            if sort_get_param and sort_get_param == "desc":
-                order = f"-{order}"
 
             if not display_unpublished:
                 chant_set: QuerySet = Chant.objects.filter(source__published=True)
@@ -451,12 +419,14 @@ def get_queryset(self) -> QuerySet:
             else:
                 chant_set: QuerySet = Chant.objects.all()
                 sequence_set: QuerySet = Sequence.objects.all()
+
             # Filter the QuerySet with Q object
             chant_set = chant_set.filter(q_obj_filter)
             sequence_set = sequence_set.filter(q_obj_filter)
             # Fetch only the values necessary for rendering the template
             chant_set = chant_set.values(*CHANT_SEARCH_TEMPLATE_VALUES)
             sequence_set = sequence_set.values(*CHANT_SEARCH_TEMPLATE_VALUES)
+
             # Finally, do keyword searching over the querySet
             if self.request.GET.get("keyword"):
                 keyword = self.request.GET.get("keyword")
@@ -479,10 +449,40 @@ def get_queryset(self) -> QuerySet:
                 chant_set = chant_set.filter(keyword_filter)
                 sequence_set = sequence_set.filter(keyword_filter)
 
-            # once unioned, the queryset cannot be filtered/annotated anymore, so we put
-            # union to the last
+            # once unioned, the queryset cannot be filtered/annotated anymore, so we put union to the last
             queryset = chant_set.union(sequence_set, all=True)
-            queryset = queryset.order_by(order, "id")
+
+        # Apply sorting
+        order_get_param: Optional[str] = self.request.GET.get("order")
+        sort_get_param: Optional[str] = self.request.GET.get("sort")
+
+        order_param_options = (
+            "incipit",
+            "office",
+            "genre",
+            "cantus_id",
+            "mode",
+            "has_fulltext",
+            "has_melody",
+            "has_image",
+        )
+        if order_get_param in order_param_options:
+            if order_get_param == "has_fulltext":
+                order = "manuscript_full_text"
+            elif order_get_param == "has_melody":
+                order = "volpiano"
+            elif order_get_param == "has_image":
+                order = "image_link"
+            else:
+                order = order_get_param
+        else:
+            order = "source__siglum"
+
+        # sort values: "asc" and "desc". Default is "asc"
+        if sort_get_param and sort_get_param == "desc":
+            order = f"-{order}"
+
+        queryset = queryset.order_by(order, "id")
 
         return queryset
 


Test patch:
diff --git a/django/cantusdb_project/main_app/tests/test_views.py b/django/cantusdb_project/main_app/tests/test_views.py
--- a/django/cantusdb_project/main_app/tests/test_views.py
+++ b/django/cantusdb_project/main_app/tests/test_views.py
@@ -1130,6 +1130,44 @@ def test_order_by_siglum(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_siglum_global_search(self):
+        source_1 = make_fake_source(published=True, siglum="sigl-1")
+        chant_1 = make_fake_chant(
+            manuscript_full_text_std_spelling="thing 1", source=source_1
+        )
+        source_2 = make_fake_source(published=True, siglum="sigl-2")
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="thing 2", source=source_2
+        )
+        search_term = "thing"
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "siglum",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "siglum",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_incipit(self):
         source = make_fake_source(published=True)
         chant_1 = make_fake_chant(
@@ -1171,6 +1209,44 @@ def test_order_by_incipit(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_incipit_global_search(self):
+        source = make_fake_source(published=True)
+        chant_1 = make_fake_chant(
+            source=source, manuscript_full_text_std_spelling="jiggle"
+        )
+        chant_2 = make_fake_chant(
+            source=source, manuscript_full_text_std_spelling="jigsaw"
+        )
+
+        search_term = "jig"
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "incipit",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "incipit",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_office(self):
         # currently, office sort works by ID rather than by name
         office_1 = make_fake_office()
@@ -1215,6 +1291,48 @@ def test_order_by_office(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_office_global_search(self):
+
+        # currently, office sort works by ID rather than by name
+        office_1 = make_fake_office()
+        office_2 = make_fake_office()
+        assert office_1.id < office_2.id
+        chant_1 = make_fake_chant(
+            office=office_1, manuscript_full_text_std_spelling="fluffy"
+        )
+        chant_2 = make_fake_chant(
+            office=office_2, manuscript_full_text_std_spelling="fluster"
+        )
+
+        search_term = "flu"
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "office",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "office",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_genre(self):
         # currently, genre sort works by ID rather than by name
         genre_1 = make_fake_genre()
@@ -1259,6 +1377,46 @@ def test_order_by_genre(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_genre_global_search(self):
+        # currently, genre sort works by ID rather than by name
+        genre_1 = make_fake_genre()
+        genre_2 = make_fake_genre()
+        assert genre_1.id < genre_2.id
+        chant_1 = make_fake_chant(
+            genre=genre_1, manuscript_full_text_std_spelling="chuckle"
+        )
+        chant_2 = make_fake_chant(
+            genre=genre_2, manuscript_full_text_std_spelling="chunky"
+        )
+
+        search_term = "chu"
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "genre",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "genre",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_cantus_id(self):
         chant_1 = make_fake_chant(
             manuscript_full_text_std_spelling="isaac", cantus_id="121393"
@@ -1299,6 +1457,43 @@ def test_order_by_cantus_id(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_cantus_id_global_search(self):
+        chant_1 = make_fake_chant(
+            manuscript_full_text_std_spelling="twinkle", cantus_id="121393"
+        )
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="twirl", cantus_id="196413"
+        )
+
+        search_term = "13"
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "cantus_id",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "cantus_id",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_mode(self):
         chant_1 = make_fake_chant(
             manuscript_full_text_std_spelling="For first he looks upon his forepaws to see if they are clean",
@@ -1341,6 +1536,46 @@ def test_order_by_mode(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_mode_global_search(self):
+        chant_1 = make_fake_chant(
+            manuscript_full_text_std_spelling="doodle",
+            mode="1",
+        )
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="doodad",
+            mode="2",
+        )
+
+        search_term = "doo"
+
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "mode",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "mode",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_ms_fulltext(self):
         chant_1 = make_fake_chant(
             manuscript_full_text="this is a chant with a MS spelling fylltexte",
@@ -1384,6 +1619,47 @@ def test_order_by_ms_fulltext(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_ms_fulltext_global_search(self):
+        chant_1 = make_fake_chant(
+            manuscript_full_text="this is a chant with a MS spelling fylltexte",
+            manuscript_full_text_std_spelling="this is a chant with a MS spelling fulltext",
+        )
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="this will become a chant without a MS spelling fulltext",
+        )
+        chant_2.manuscript_full_text = None
+        chant_2.save()
+
+        search_term = "this"
+
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_fulltext",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_fulltext",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_volpiano(self):
         chant_1 = make_fake_chant(
             manuscript_full_text_std_spelling="this is a chant with volpiano",
@@ -1427,6 +1703,47 @@ def test_order_by_volpiano(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_volpiano_global_search(self):
+        chant_1 = make_fake_chant(
+            manuscript_full_text_std_spelling="this is a chant with volpiano",
+            volpiano="1---d---d---a--a---a---e--f--e---d---4",
+        )
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="this is a chant about mushrooms",
+        )
+        chant_2.volpiano = None
+        chant_2.save()
+
+        search_term = "this is a chant"
+
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_melody",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_melody",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_order_by_image_link(self):
         chant_1 = make_fake_chant(
             manuscript_full_text_std_spelling="this is a chant with a link",
@@ -1470,6 +1787,47 @@ def test_order_by_image_link(self):
         last_result_incipit = descending_results[1]["incipit"]
         self.assertEqual(last_result_incipit, chant_1.incipit)
 
+    def test_order_by_image_link_global_search(self):
+        chant_1 = make_fake_chant(
+            manuscript_full_text_std_spelling="this is a chant with a link",
+            image_link="https://www.youtube.com/watch?v=hyCIpKAIFyo",
+        )
+        chant_2 = make_fake_chant(
+            manuscript_full_text_std_spelling="this is a chant without",
+        )
+        chant_2.image_link = None
+        chant_2.save()
+
+        search_term = "this is"
+
+        response_ascending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_image",
+                "sort": "asc",
+            },
+        )
+        ascending_results = response_ascending.context["chants"]
+        first_result_incipit = ascending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_1.incipit)
+        last_result_incipit = ascending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_2.incipit)
+
+        response_descending = self.client.get(
+            reverse("chant-search"),
+            {
+                "search_bar": search_term,
+                "order": "has_image",
+                "sort": "desc",
+            },
+        )
+        descending_results = response_descending.context["chants"]
+        first_result_incipit = descending_results[0]["incipit"]
+        self.assertEqual(first_result_incipit, chant_2.incipit)
+        last_result_incipit = descending_results[1]["incipit"]
+        self.assertEqual(last_result_incipit, chant_1.incipit)
+
     def test_column_header_links(self):
         # these are the 9 column headers users can order by:
         siglum = "glum-01"