internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DDMAL/CantusDB

Issue:
Source.number_of_melodies is not updating properly
The `number_of_melodies` of a source should correspond to the number of chants in the source that contain a volpiano entry. When I check my test sources on the database that do not have any chants that contain a volpaino, the `number_of_melodies` field matches the total number of chants. I suspect the `update_source_melody_count()` function in `signals.py` is not working as expected.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/signals.py b/django/cantusdb_project/main_app/signals.py
--- a/django/cantusdb_project/main_app/signals.py
+++ b/django/cantusdb_project/main_app/signals.py
@@ -91,9 +91,11 @@ def update_source_melody_count(instance):
     except Source.DoesNotExist:
         source = None
     if source is not None:
-        source.number_of_melodies = source.chant_set.filter(
-            volpiano__isnull=False
-        ).count()
+        source.number_of_melodies = (
+            source.chant_set.exclude(volpiano__isnull=True)
+            .exclude(volpiano__exact="")
+            .count()
+        )
         source.save()
 
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
Source "View full inventory" load time is slow
On OldCantus, when you click on "View full index" of a source (particularily for a large source like https://cantus.uwaterloo.ca/source/123617) the results load almost instantly. On NewCantus, the equiavlent link "View full inventory" takes around 30 seconds to load. This is likely because there are thousands of queries being executed on this page:

![image](https://github.com/DDMAL/CantusDB/assets/71031342/46aa6697-0d54-4dab-b82e-0baa6abacaf7)

We should try optimizing this using Django's `select_related` or `prefetch_related`


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -1389,12 +1389,16 @@ def get_context_data(self, **kwargs):
 
         # 4064 is the id for the sequence database
         if source.segment.id == 4064:
-            queryset = source.sequence_set.annotate(
-                record_type=Value("sequence")
-            ).order_by("s_sequence")
+            queryset = (
+                source.sequence_set.annotate(record_type=Value("sequence"))
+                .order_by("s_sequence")
+                .select_related("genre")
+            )
         else:
-            queryset = source.chant_set.annotate(record_type=Value("chant")).order_by(
-                "folio", "c_sequence"
+            queryset = (
+                source.chant_set.annotate(record_type=Value("chant"))
+                .order_by("folio", "c_sequence")
+                .select_related("feast", "office", "genre")
             )
 
         context["source"] = source


Test patch:
Repository: DDMAL/CantusDB

Issue:
Chant Detail pages sometimes have 500 errors
The Chant Detail pages for certain chants were causing 500 errors earlier today on Staging and Production. Only chants with Cantus IDs were affected.

We've applied a temporary fix on Staging and Production (huge shout-out to @lucasmarchd01 and @jackyyzhang03 for helping to debug this).

When trying to load concordances from Cantus Index, we were hitting an exception we hadn't seen before and so didn't know to catch. It seems that something changed on cantusindex.org, which was causing it to not send back useful results. I'll look more closely into why this error might be occurring - more details coming soon! - but the upshot is that chant pages can load, but the concordances feature is broken (i.e. it always displays "Cantus Database encountered an error while loading concordances").


Discussion:
doing some working-out:

I can run `curl https://cantusindex.org/json-con/007033` successfully locally, and load it in my browser, but when I try to run the command on either staging or production, I get the following error message:
```shell
jacob@cantus-db-stag:~$ curl https://cantusindex.org/json-con/007033
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cantusindex.org:443 
```

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -31,7 +31,7 @@
 from collections import Counter
 from django.contrib.auth.mixins import UserPassesTestMixin
 from typing import Optional, Union
-from requests.exceptions import SSLError, Timeout
+from requests.exceptions import SSLError, Timeout, ConnectionError
 from requests import Response
 
 CHANT_SEARCH_TEMPLATE_VALUES = (
@@ -83,6 +83,7 @@ def parse_json_from_api(url: str) -> Union[list, None]:
     except (
         SSLError,
         Timeout,
+        ConnectionError,
     ) as exc:
         print(  # eventually, we should log this rather than printing it to the console
             "Encountered an error in parse_json_from_api",


Test patch:
Repository: DDMAL/CantusDB

Issue:
we need to re-add a restart policy to docker-compose.yml
A recent change to docker-compose.yml removed the `restart: always` policy we added to our containers a couple of weeks ago. We should re-instate this.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/widgets.py b/django/cantusdb_project/main_app/widgets.py
--- a/django/cantusdb_project/main_app/widgets.py
+++ b/django/cantusdb_project/main_app/widgets.py
@@ -1,6 +1,7 @@
 from django.forms.widgets import TextInput, Select, Textarea, CheckboxInput
 from django.utils.safestring import mark_safe
 
+
 class TextInputWidget(TextInput):
     def __init__(self):
         self.attrs = {"class": "form-control form-control-sm"}


Test patch:
Repository: DDMAL/CantusDB

Issue:
Concordances Summary is mixed up
![Screenshot 2023-07-31 at 3 02 01 PM](https://github.com/DDMAL/CantusDB/assets/58090591/2c7c6a28-2237-475c-9d7b-21e3d45343ac)

Notice that both Medieval Music Manuscripts Online and Portuguese Early Music Database are abbreviated MMMO. The links for both rows still link to their correct places.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -350,7 +350,7 @@ def get_context_data(self, **kwargs):
                 },
                 {
                     "name": "Portuguese Early Music Database",
-                    "initialism": "MMMO",
+                    "initialism": "PEM",
                     "base_url": "https://pemdatabase.eu",
                     "results_url": f"https://pemdatabase.eu/id/{chant.cantus_id}",
                     "results_count": len(


Test patch:
Repository: DDMAL/CantusDB

Issue:
Admin Chant Edit page: we should make "title" field longer
In an email from Debra:

> https://cantusdatabase.org/admin/main_app/source/123611/change/
For a page like this one, could I please have a longer box for the source title? The width of the "provenance note" field would be about right, I think, so that I can see the whole title without having to scroll.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -826,6 +826,7 @@ class Meta:
         widget=AdminTextInputWidget,
         help_text="Full Manuscript Identification (City, Archive, Shelf-mark)",
     )
+    title.widget.attrs.update({"style": "width: 610px;"})
 
     siglum = forms.CharField(
         required=True,


Test patch:
Repository: DDMAL/CantusDB

Issue:
admin area has some incorrect titles
Several of the lists for choosing editors say things about centuries instead:

<img width="818" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/bac6e198-741a-4a37-b6bc-f7466e19050f">

<img width="728" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/7819152f-7df3-47a0-9800-bb703ccadf3a">



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -863,15 +863,43 @@ class Meta:
             | Q(groups__name="editor")
             | Q(groups__name="contributor")
         )
-        .order_by("last_name"),
+        .order_by("full_name"),
         required=False,
-        widget=FilteredSelectMultiple(verbose_name="Century", is_stacked=False),
+        widget=FilteredSelectMultiple(verbose_name="current editors", is_stacked=False),
+    )
+
+    inventoried_by = forms.ModelMultipleChoiceField(
+        queryset=get_user_model().objects.all().order_by("full_name"),
+        required=False,
+        widget=FilteredSelectMultiple(verbose_name="inventoried by", is_stacked=False),
+    )
+
+    full_text_entered_by = forms.ModelMultipleChoiceField(
+        queryset=get_user_model().objects.all().order_by("full_name"),
+        required=False,
+        widget=FilteredSelectMultiple(
+            verbose_name="full text entered by", is_stacked=False
+        ),
     )
 
     melodies_entered_by = forms.ModelMultipleChoiceField(
         queryset=get_user_model().objects.all().order_by("full_name"),
         required=False,
-        widget=FilteredSelectMultiple(verbose_name="Century", is_stacked=False),
+        widget=FilteredSelectMultiple(
+            verbose_name="melodies entered by", is_stacked=False
+        ),
+    )
+
+    proofreaders = forms.ModelMultipleChoiceField(
+        queryset=get_user_model().objects.all().order_by("full_name"),
+        required=False,
+        widget=FilteredSelectMultiple(verbose_name="proofreaders", is_stacked=False),
+    )
+
+    other_editors = forms.ModelMultipleChoiceField(
+        queryset=get_user_model().objects.all().order_by("full_name"),
+        required=False,
+        widget=FilteredSelectMultiple(verbose_name="other editors", is_stacked=False),
     )
 
     TRUE_FALSE_CHOICES_INVEN = ((True, "Complete"), (False, "Incomplete"))


Test patch:
Repository: DDMAL/CantusDB

Issue:
List of Genres: alphabetization should be case-insensitive
The Lists of Feasts (this includes the list of all feasts as well as only the temporale and only the sanctorale) are ordered by alphabetic order of feast name, except that capital letters are currently taking precedence; a capital "Z" comes before a lowercase "a" and that muddles the system somewhat. Behold:

<img width="1119" alt="NewCantus temporale list" src="https://github.com/DDMAL/CantusDB/assets/83373378/2c4f7d2e-df3c-4a57-8c52-7acbfe96b099">

You can see that the alphabet thing is solid until Ad Processionem, after which the third letter becomes lowercase (Ad aquam benedictio) and the alphabet starts all over again. Maybe this is preferable, but OldCantus doesn't take capitalization into account, so I thought I'd mention it.

This is the same list on OldCantus, for reference: 

<img width="1158" alt="OldCantus temporale list" src="https://github.com/DDMAL/CantusDB/assets/83373378/598ba519-a37a-478b-884f-a1ff6b07ee34">



Discussion:
Also, as you can see from the screenshots above, NewCantus has 521 temporale feasts, whereas OldCantus has 785. Is this important?
Another update: the capitalization thing also applies to the list of Genres. "AV" comes before "Ag" in NewCantus because the V is capitalized.
nice catch - thanks!
> Also, as you can see from the screenshots above, NewCantus has 521 temporale feasts, whereas OldCantus has 785. Is this important?

I will look into this!  It seems like a whole bunch of feasts have been reclassified as sanctorale instead of temporale but I haven't yet looked to see which ones.  

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/feast.py b/django/cantusdb_project/main_app/views/feast.py
--- a/django/cantusdb_project/main_app/views/feast.py
+++ b/django/cantusdb_project/main_app/views/feast.py
@@ -1,4 +1,5 @@
 from django.views.generic import DetailView, ListView
+from django.db.models.functions import Lower
 from main_app.models import Feast, Source
 from extra_views import SearchableListMixin
 
@@ -90,13 +91,14 @@ class FeastListView(SearchableListMixin, ListView):
     context_object_name = "feasts"
     template_name = "feast_list.html"
 
-    def get_ordering(self):
+    def get_ordering(self) -> tuple:
         ordering = self.request.GET.get("sort_by")
         # feasts can be ordered by name or feast_code,
         # default to ordering by name if given anything else
         if ordering not in ["name", "feast_code"]:
             ordering = "name"
-        return ordering
+        # case insensitive ordering by name
+        return (Lower(ordering),) if ordering == "name" else (ordering,)
 
     def get_queryset(self):
         queryset = super().get_queryset()


Test patch:
Repository: DDMAL/CantusDB

Issue:
We should add "noindex" on staging
Currently, both production and staging are set up to allow them to be crawled/indexed by Google etc. (see #466, #904).

Ideally, links to the staging server don't appear among Google results. We should find a way to add `<meta name="robots" content="noindex">` back to all pages on staging (but not on production).

Marking this as low priority because many pages link to cantusdatabase.org and very few pages link to the staging IP address. Links to the staging server should thus not appear very often among Google results. Still, ideally we set this up so that they are guaranteed not to appear.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/cantusdb/settings.py b/django/cantusdb_project/cantusdb/settings.py
--- a/django/cantusdb_project/cantusdb/settings.py
+++ b/django/cantusdb_project/cantusdb/settings.py
@@ -87,6 +87,7 @@
                 "django.template.context_processors.request",
                 "django.contrib.auth.context_processors.auth",
                 "django.contrib.messages.context_processors.messages",
+                "main_app.context_processors.determine_project_environment",
             ],
         },
     },
diff --git a/django/cantusdb_project/main_app/context_processors.py b/django/cantusdb_project/main_app/context_processors.py
new file mode 100644
--- /dev/null
+++ b/django/cantusdb_project/main_app/context_processors.py
@@ -0,0 +1,18 @@
+import os
+
+
+def determine_project_environment(request):
+    project_environment = os.getenv("PROJECT_ENVIRONMENT", None)
+    if not (
+        project_environment == "DEVELOPMENT"
+        or project_environment == "STAGING"
+        or project_environment == "PRODUCTION"
+    ):
+        raise ValueError(
+            "The PROJECT_ENVIRONMENT environment variable must be either "
+            "DEVELOPMENT, STAGING, or PRODUCTION. "
+            f"Its current value is {project_environment}."
+        )
+    return {
+        "PROJECT_ENVIRONMENT": project_environment,
+    }


Test patch:
Repository: DDMAL/CantusDB

Issue:
Test Fail in SourceDetailViewTest
All of the tests are currently failing in SourceDetailViewTest

```
======================================================================
ERROR: test_context_sequences (main_app.tests.test_views.SourceDetailViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 4319, in test_context_sequences
    response = self.client.get(reverse("source-detail", args=[source.id]))
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 927, in get
    response = super().get(path, data=data, secure=secure, headers=headers, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 457, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 609, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 891, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 738, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/detail.py", line 109, in get
    context = self.get_context_data(object=self.object)
  File "/code/django/cantusdb_project/main_app/views/source.py", line 52, in get_context_data
    context["user_can_edit_source"] = user_can_edit_source(user, source)
  File "/code/django/cantusdb_project/main_app/views/source.py", line 281, in user_can_edit_source
    assigned_to_source = user.sources_user_can_edit.filter(id=source_id)
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 268, in inner
    return func(_wrapped, *args)
AttributeError: 'AnonymousUser' object has no attribute 'sources_user_can_edit'
```


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/source.py b/django/cantusdb_project/main_app/views/source.py
--- a/django/cantusdb_project/main_app/views/source.py
+++ b/django/cantusdb_project/main_app/views/source.py
@@ -277,6 +277,8 @@ def form_valid(self, form):
 
 
 def user_can_edit_source(user, source):
+    if user.is_anonymous:
+        return False
     source_id = source.id
     assigned_to_source = user.sources_user_can_edit.filter(id=source_id)
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
404 Error on NewCantus: GenreList and OfficeList views
https://cantus.uwaterloo.ca/genre

![image](https://github.com/DDMAL/CantusDB/assets/11023634/097722e9-74ee-4b97-8dc2-3173156bafb4)


https://cantusdatabase.org/genre

![image](https://github.com/DDMAL/CantusDB/assets/11023634/65db1488-2524-4447-9bef-04c2f4f169c5)

It seems that the desired information is available instead at https://cantusdatabase.org/genres (Note plural)


![image](https://github.com/DDMAL/CantusDB/assets/11023634/67d2295a-e3c9-4daa-9dda-e7be7a6b6fe8)

Note the same thing happens with `/office` vs `/offices`.



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/urls.py b/django/cantusdb_project/main_app/urls.py
--- a/django/cantusdb_project/main_app/urls.py
+++ b/django/cantusdb_project/main_app/urls.py
@@ -215,6 +215,11 @@
         GenreDetailView.as_view(),
         name="genre-detail",
     ),
+    path(
+        "genre/",
+        views.redirect_genre,
+        name="redirect-genre",
+    ),
     # indexer
     path(
         "indexers/",
@@ -238,6 +243,11 @@
         OfficeDetailView.as_view(),
         name="office-detail",
     ),
+    path(
+        "office/",
+        views.redirect_office,
+        name="redirect-office",
+    ),
     # provenance
     path(
         "provenance/<int:pk>",
@@ -383,6 +393,7 @@
         views.redirect_indexer,
         name="redirect-indexer",
     ),
+    # links to APIs that list URLs of all pages that live in the database
     path(
         "articles-list/",
         views.articles_list_export,
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -881,6 +881,32 @@ def redirect_indexer(request, pk: int) -> HttpResponse:
     raise Http404("No indexer found matching the query.")
 
 
+def redirect_office(request) -> HttpResponse:
+    """
+    Redirects from office/ (à la OldCantus) to offices/ (à la NewCantus)
+
+    Args:
+        request
+
+    Returns:
+        HttpResponse
+    """
+    return redirect("office-list")
+
+
+def redirect_genre(request) -> HttpResponse:
+    """
+    Redirects from genre/ (à la OldCantus) to genres/ (à la NewCantus)
+
+    Args:
+        request
+
+    Returns:
+        HttpResponse
+    """
+    return redirect("genre-list")
+
+
 def redirect_documents(request) -> HttpResponse:
     """Handle requests to old paths for various
     documents on OldCantus, returning an HTTP Response


Test patch:
Repository: DDMAL/CantusDB

Issue:
Source Create/Edit - RISM siglum field broken
In an email from Debra:

> For the RISM field in sources (such as this one: https://cantusdatabase.org/source/123664), I see 4100 entered instead of a RISM siglum. Is there a source for this number? (Like, is it in a table somewhere?)
> That's the place where I used to put "B-TO", and then in the Siglum field I would copy that again plus add the shelfmark. I'm wondering how I should fill out the RISM field in the future. I think we're likely changing this for New-NewCantus to be the RISM code in one field and then just the shelfmark in another, but what should we do in the meantime?

Currently, this field uses the ID of the RismSiglum object in the database - not at all useful! OldCantus has an autocomplete-style widget for this field (the user begins typing, and a list of matching values appears below the field); this may be a good place for a filterable list widget.


Discussion:
@lucasmarchd01, can you look into this when you get a chance? Assigning you because you've recently done work with filterable lists.
On the Source Create page, it's a simple dropdown. At the very least, we can use the same widget on the Source Edit page. Let me see if I can quickly get this set up before merging develop -> staging; we can implement a filterable list later.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/forms.py b/django/cantusdb_project/main_app/forms.py
--- a/django/cantusdb_project/main_app/forms.py
+++ b/django/cantusdb_project/main_app/forms.py
@@ -515,7 +515,6 @@ class Meta:
         ]
         widgets = {
             "title": TextInputWidget(),
-            "rism_siglum": TextInputWidget(),
             "siglum": TextInputWidget(),
             "provenance_notes": TextInputWidget(),
             "date": TextInputWidget(),
@@ -547,6 +546,13 @@ class Meta:
             ),
         }
 
+    rism_siglum = forms.ModelChoiceField(
+        queryset=RismSiglum.objects.all().order_by("name"), required=False
+    )
+    rism_siglum.widget.attrs.update(
+        {"class": "form-control custom-select custom-select-sm"}
+    )
+
     provenance = forms.ModelChoiceField(
         queryset=Provenance.objects.all().order_by("name"), required=False
     )


Test patch:
Repository: DDMAL/CantusDB

Issue:
in content overview, have sources by last-updated
Currently the sources are sorted by creation, but some active projects are working on fairly old source objects. I think it would be more useful to have them sorted by last update.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -853,7 +853,7 @@ def content_overview(request):
 
     objects = []
     if selected_model:
-        objects = selected_model.objects.all().order_by("-date_created")
+        objects = selected_model.objects.all().order_by("-date_updated")
 
     paginator = Paginator(objects, 100)
     page_number = request.GET.get("page")


Test patch: