internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DDMAL/CantusDB

Issue:
some chant suggestions don't work?
In trying to create some test chants to investigate #996 , I found that the chant suggestions sometimes autofill but not always.  For example, having created 
![image](https://github.com/DDMAL/CantusDB/assets/67451875/7b096a37-0551-4d87-ba66-b473d93f3e45)
 the suggestion for g00556.Tp doesn't populate the fields, while all the other suggestions do. And for an earlier chant, the first suggestion didn't work but the others did: 
![image](https://github.com/DDMAL/CantusDB/assets/67451875/b84d5277-ba70-4c8b-8be9-25a2412e82c8)
Still investigating why this is.


Discussion:
wondering if it might be relevant that both those suggestions contain dots (and in both cases have IDs that are too long to fit in the button.) Seems to be consistent for other cases where this doesn't work...
Interesting. I think this was also pointed out by someone to Debra, so we opened #815. It was closed because it seemed like it was working fine again, so we should figure out why this is still happening.
It's also possible that we were simply unable to reproduce the issue at the time because the problem seems to be specific to only some suggestions  (I think it's only suggestions where the ChantID has a decimal point in it.)
It's the decimal point. In our template, we're creating a function `autoFill{{ chant.cid }}`. But we're not allowed to create a function called `autoFillg00556.Tp2`, due to the dot.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -204,6 +204,11 @@ def make_suggested_chant_dict(
     genre_name = cid_dict["genre"]
     genre_id = Genre.objects.get(name=genre_name).id
     cid_dict["genre_id"] = genre_id
+    # in the template, we create a js function based on the cantus ID. The function name
+    # cannot include a "." or a ":", so we have to use a cleaned version
+    cid_clean = cantus_id.replace(".", "d").replace(":", "c")
+    #                                  "d"ot             "c"olon
+    cid_dict["cid_clean"] = cid_clean
     return cid_dict
 
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
Hide "Number of chants" and "Number of melodies" fields in Source admin
On the Source detail page in the admin area, we are currently displaying the number of melodies and number of chants for the source.
![image](https://github.com/DDMAL/CantusDB/assets/71031342/df7986a3-1536-4ec0-89c8-bd421cf45725)

We only use this information behind the scenes, so we should not allow users to edit this field since they will be automatically updated as chants or melodies are added/removed from the Source.

Earlier, I found an issue where these fields weren't being updated correctly. I found this because the only place we can see the number of chants and melodies is in the admin area. For this reason and for future situations like this, I think we should make these fields `read_only` instead of hidden altogether.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -57,6 +57,12 @@ def get_source_siglum(self, obj):
         "cantus_id",
         "id",
     )
+
+    readonly_fields = (
+        "date_created",
+        "date_updated",
+    )
+
     list_filter = (
         "genre",
         "office",
@@ -161,6 +167,12 @@ class SourceAdmin(BaseModelAdmin):
         "title",
         "id",
     )
+    readonly_fields = (
+        "number_of_chants",
+        "number_of_melodies",
+        "date_created",
+        "date_updated",
+    )
     # from the Django docs:
     # Adding a ManyToManyField to this list will instead use a nifty unobtrusive JavaScript “filter” interface
     # that allows searching within the options. The unselected and selected options appear in two boxes side by side.


Test patch:
Repository: DDMAL/CantusDB

Issue:
Admin Area: Hide "Volpiano notes" and "Volpiano interval" fields
We currently display these fields in the admin area, but they are only for internal use. Instead, we should hide these fields (i.e., exclude, instead of readonly_fields).


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -76,6 +76,8 @@ def get_source_siglum(self, obj):
         "is_last_chant_in_feast",
         "visible_status",
         "date",
+        "volpiano_notes",
+        "volpiano_intervals",
     )
     form = AdminChantForm
     raw_id_fields = (


Test patch:
Repository: DDMAL/CantusDB

Issue:
CDB is sending useless requests to Cantus Index
Experimenting with our views, our Chant Detail page is causing requests to be sent to `https://cantusindex.org/json-con/cantus_id/refresh`. I don't understand why this is the case, but I'm looking into it currently.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -155,7 +155,7 @@ def touch_ci_json_con_api(cantus_id: str) -> None:
     """
     t = threading.Thread(
         target=_refresh_ci_json_con_api,
-        args={"cantus_id": f"{cantus_id}"},
+        args=[cantus_id],
     )
     t.start()
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
json-sources API should only return sources in the Cantus segment.
As noted by @dchiller on https://github.com/DDMAL/cantus/pull/765, the json-sources API on OldCantus only returned sources in the CANTUS segment, whereas NewCantus's API returns all published sources. We need to update NewCantus to match the behavior seen in OldCantus.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -14,6 +14,7 @@
     Office,
     Provenance,
     RismSiglum,
+    Segment,
     Sequence,
     Source,
 )
@@ -525,7 +526,8 @@ def json_sources_export(request) -> JsonResponse:
     """
     Generate a json object of published sources with their IDs and CSV links
     """
-    sources = Source.objects.filter(published=True)
+    cantus_segment = Segment.objects.get(id=4063)
+    sources = cantus_segment.source_set.filter(published=True)
     ids = [source.id for source in sources]
 
     csv_links = {id: build_json_sources_export_dictionary(id, request) for id in ids}


Test patch:
Repository: DDMAL/CantusDB

Issue:
Admin Area, Chant/Sequence Change pages - duplicate names appearing
A few weeks ago, we fixed the issue where users' names would appear twice among the "Available indexers" etc.

The same issue occurs on the Chant and Sequence Change pages, in the "Available proofread by" field. We need to apply the same treatment to these models/forms.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -585,6 +585,7 @@ class Meta:
     proofread_by = forms.ModelMultipleChoiceField(
         queryset=get_user_model()
         .objects.filter(Q(groups__name="project manager") | Q(groups__name="editor"))
+        .distinct()
         .order_by("last_name"),
         required=False,
         widget=FilteredSelectMultiple(verbose_name="proofread by", is_stacked=False),
@@ -679,6 +680,7 @@ class Meta:
     proofread_by = forms.ModelMultipleChoiceField(
         queryset=get_user_model()
         .objects.filter(Q(groups__name="project manager") | Q(groups__name="editor"))
+        .distinct()
         .order_by("last_name"),
         required=False,
         widget=FilteredSelectMultiple(verbose_name="proofread by", is_stacked=False),
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -1026,6 +1026,7 @@ def get_queryset(self):
             .objects.filter(
                 Q(groups__name="project manager") | Q(groups__name="editor")
             )
+            .distinct()
             .order_by("full_name")
         )
         if self.q:


Test patch:
Repository: DDMAL/CantusDB

Issue:
Source Edit page missing "liturgical occasions" field
On OldCantus one of the fields when creating a source was "liturgical occasions."
![image](https://github.com/DDMAL/CantusDB/assets/67451875/e04cadd0-c0b6-4ae4-bda6-24adf7cb185c)

This does not seem to be present in NewCantus. I don't think this difference is documented in the list of known differences between OldCantus and New, so I don't remember if this was by design or not. I'm guessing not, since it also means there is no way for an editor to edit existing "liturgical occasions" on manuscripts created on OldCantus, unless I'm totally crazy.


Discussion:
I don't think this omission was intentional. I'll get to work adding this.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -378,6 +378,7 @@ class Meta:
             "cursus",
             "complete_inventory",
             "summary",
+            "liturgical_occasions",
             "description",
             "selected_bibliography",
             "image_link",
@@ -398,6 +399,7 @@ class Meta:
             "provenance_notes": TextInputWidget(),
             "date": TextInputWidget(),
             "summary": TextAreaWidget(),
+            "liturgical_occasions": TextAreaWidget(),
             "description": TextAreaWidget(),
             "selected_bibliography": TextAreaWidget(),
             "image_link": TextInputWidget(),


Test patch:
Repository: DDMAL/CantusDB

Issue:
Chant admin "Manuscript full text std spelling" widget is small
The Manuscript full text std spelling widget should be the same size as the Manuscript full text and Manuscript syllabized full text widgets below in the Admin chant detail page.

![image](https://github.com/DDMAL/CantusDB/assets/71031342/ce781942-e762-435b-8f6a-84185bc9e657)



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -556,6 +556,12 @@ class Meta:
         'the first word of each chant, and the first word after "Alleluia" for '
         "Mass Alleluias. Punctuation is omitted.",
     )
+    # Django's default text area widget selection for form inputs is non-intuitive
+    # and manual updates to fields (e.g., changing required=True) affect widget properties unexpectedly;
+    # this workaround is our current best solution.
+    manuscript_full_text_std_spelling.widget.attrs.update(
+        {"style": "width: 610px; height: 170px;"}
+    )
 
     folio = forms.CharField(
         required=True,


Test patch:
Repository: DDMAL/CantusDB

Issue:
Admin area, change chant page: "title" field should be hidden
The "title" field is only used for sequences and never for chants, so this field should be hidden from the Chant Change page in the Admin area.

Debra sent us a message asking us what this field was for, so since this is a simple fix, we should hide this field before it causes further confusion.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -78,6 +78,7 @@ def get_source_siglum(self, obj):
         "date",
         "volpiano_notes",
         "volpiano_intervals",
+        "title",
     )
     form = AdminChantForm
     raw_id_fields = (


Test patch:
Repository: DDMAL/CantusDB

Issue:
Admin Area, Change Source page: we should hide "Source Status" field
Debra got tripped up trying to publish a source - instead of checking the "Published" checkbox, she selected something in the "Source Status" dropdown.

~~Everything regarding source visibility is handled by our `published` field, so I think "Source Status" is vestigal and no longer needed. We may not even use it anywhere in our templates/views. With this in mind, it may be worth considering removing this field entirely. (Any data related to this field that was in OldCantus when we last synced is saved in sources' `json_info` field, so we wouldn't really need to worry about losing data.)~~ (it was decided that this information may eventually be useful, so we'll not remove the field from the model)


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -172,6 +172,8 @@ def get_source_siglum(self, obj):
 
 
 class SourceAdmin(BaseModelAdmin):
+    exclude = ("source_status",)
+
     # These search fields are also available on the user-source inline relationship in the user admin page
     search_fields = (
         "siglum",


Test patch:
Repository: DDMAL/CantusDB

Issue:
Chant Search: we should remove the option to search for chants without melodies
Currently, on the Chant Search page, there are options to search among all chants, among chants with melodies, and among chants without melodies. We should remove the option to search for chants without melodies:
- this option is not available on OldCantus
- why would you ever want to search specifically for chants without melodies?
- it's not clear what should appear if a user isn't logged in and searches for chants without melodies: should only chants with no volpiano appear, or only chants that lack proofread volpiano? See #1147

Considering this isn't on OldCantus, @annamorphism suggested we can go ahead and remove this without consulting Debra.


Discussion:
assigning a priority of High because this blocks the merging of #1146.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -596,8 +596,6 @@ def get_queryset(self) -> QuerySet:
                 melodies = self.request.GET.get("melodies")
                 if melodies == "true":
                     q_obj_filter &= Q(volpiano__isnull=False)
-                if melodies == "false":
-                    q_obj_filter &= Q(volpiano__isnull=True)
             if self.request.GET.get("feast"):
                 feast = self.request.GET.get("feast")
                 # This will match any feast whose name contains the feast parameter


Test patch:
Repository: DDMAL/CantusDB

Issue:
Chant Create/Edit: Feast/Office/Genre autocompletes should do icontains search rather than istartswith
Testing changes currently on staging, on the Chant Create/Edit pages. It would be nice to be able to find text from the middle of a field, rather than just at the beginning:

Feast field:
- searching for "mariae", I expect "Conceptio Mariae" to come up. It doesn't.

Office field:
- When I search for "V", First and Second Vespers come up. But if I search for "vespers", neither does - I need to search for "first" in order for First Vespers to appear.

Genre field:
- Similarly to Office, I expect Antiphon Verse and Alleluia Verse to appear when I search "verse", but neither does.

This is not important enough to block the merging of #1146, but it would be nice to get it fixed before too long.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -1013,9 +1013,7 @@ def get_queryset(self):
             return Feast.objects.none()
         qs = Feast.objects.all().order_by("name")
         if self.q:
-            qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(feast_code__istartswith=self.q)
-            )
+            qs = qs.filter(name__icontains=self.q)
         return qs
 
 
@@ -1029,7 +1027,7 @@ def get_queryset(self):
         qs = Office.objects.all().order_by("name")
         if self.q:
             qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(description__istartswith=self.q)
+                Q(name__istartswith=self.q) | Q(description__icontains=self.q)
             )
         return qs
 
@@ -1044,7 +1042,7 @@ def get_queryset(self):
         qs = Genre.objects.all().order_by("name")
         if self.q:
             qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(description__istartswith=self.q)
+                Q(name__istartswith=self.q) | Q(description__icontains=self.q)
             )
         return qs
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
Chant Create page: on staging, you can no longer see which source you're creating the chant in
It seems that when I removed the suggested fulltext and CI Search features, the entire sidebar disappeared. Now you can't see which source you're operating within.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -1013,9 +1013,7 @@ def get_queryset(self):
             return Feast.objects.none()
         qs = Feast.objects.all().order_by("name")
         if self.q:
-            qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(feast_code__istartswith=self.q)
-            )
+            qs = qs.filter(name__icontains=self.q)
         return qs
 
 
@@ -1029,7 +1027,7 @@ def get_queryset(self):
         qs = Office.objects.all().order_by("name")
         if self.q:
             qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(description__istartswith=self.q)
+                Q(name__istartswith=self.q) | Q(description__icontains=self.q)
             )
         return qs
 
@@ -1044,7 +1042,7 @@ def get_queryset(self):
         qs = Genre.objects.all().order_by("name")
         if self.q:
             qs = qs.filter(
-                Q(name__istartswith=self.q) | Q(description__istartswith=self.q)
+                Q(name__istartswith=self.q) | Q(description__icontains=self.q)
             )
         return qs
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
We should disable the remap_user_ids command for the time being
We have changes on Staging that need to make their way to Production soon.

The `remap_user_ids` command is not working properly (#1165).

We should disable the command for now so we can deploy recent changes to Production.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/management/commands/remap_user_ids.py b/django/cantusdb_project/main_app/management/commands/remap_user_ids.py
--- a/django/cantusdb_project/main_app/management/commands/remap_user_ids.py
+++ b/django/cantusdb_project/main_app/management/commands/remap_user_ids.py
@@ -124,6 +124,11 @@ def reassign_chants() -> None:
 
 class Command(BaseCommand):
     def handle(self, *args, **kwargs) -> None:
+        error_message = (
+            "As of late November 2023, this command is not working. "
+            "It has been temporarily disabled until the bugs have been worked out."
+        )
+        raise NotImplementedError(error_message)
         stdout.write("\n\n==== Reassigning Sources ====\n")
         reassign_sources()
         stdout.write("\n== All sources successfully remapped! ==\n")


Test patch:
Repository: DDMAL/CantusDB

Issue:
Source create/edit: Provenance autocompletes should be icontains
...rather than istartswith. "Berne" should find "Abdij ban Berne", etc.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -1063,7 +1063,7 @@ def get_queryset(self):
             return Provenance.objects.none()
         qs = Provenance.objects.all().order_by("name")
         if self.q:
-            qs = qs.filter(name__istartswith=self.q)
+            qs = qs.filter(name__icontains=self.q)
         return qs
 
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
`created_by` and `last_updated_by` fields should be read only on source admin change page
They should be readonly fields instead. I don't recall seeing them at the top of the page like they are now. Maybe previously they were hidden and now they are visible.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -16,14 +16,17 @@
 
 # these fields should not be editable by all classes
 EXCLUDE = (
-    "created_by",
-    "last_updated_by",
     "json_info",
 )
 
+READ_ONLY = (
+    "created_by",
+    "last_updated_by",
+)
 
 class BaseModelAdmin(admin.ModelAdmin):
     exclude = EXCLUDE
+    readonly_fields = READ_ONLY
 
     # if an object is created in the admin interface, assign the user to the created_by field
     # else if an object is updated in the admin interface, assign the user to the last_updated_by field
@@ -58,7 +61,7 @@ def get_source_siglum(self, obj):
         "id",
     )
 
-    readonly_fields = (
+    readonly_fields = READ_ONLY + (
         "date_created",
         "date_updated",
     )
@@ -172,7 +175,7 @@ def get_source_siglum(self, obj):
 
 
 class SourceAdmin(BaseModelAdmin):
-    exclude = ("source_status",)
+    exclude = EXCLUDE + ("source_status",)
 
     # These search fields are also available on the user-source inline relationship in the user admin page
     search_fields = (
@@ -180,7 +183,7 @@ class SourceAdmin(BaseModelAdmin):
         "title",
         "id",
     )
-    readonly_fields = (
+    readonly_fields = READ_ONLY + (
         "number_of_chants",
         "number_of_melodies",
         "date_created",


Test patch: