internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DDMAL/CantusDB

Issue:
Add Change History to all models
At the LinkedMusic project meeting, it was suggested that we should begin keeping a history of all changes to all of our models (rather than the current approach of simply noting when the most recent edit occurred and the user who made the edit). @ahankinson noted that there are simple, off-the-shelf ways to implement this (and suggested [django-simple-history](https://django-simple-history.readthedocs.io/en/latest/), if I recall correctly).


Discussion:
Nope -- `django-reversion` https://django-reversion.readthedocs.io/en/stable/
It sounds like this will be helpful for implementing #1158. Increasing the priority from low to medium.

Resolving patch:
diff --git a/django/cantusdb_project/cantusdb/settings.py b/django/cantusdb_project/cantusdb/settings.py
--- a/django/cantusdb_project/cantusdb/settings.py
+++ b/django/cantusdb_project/cantusdb/settings.py
@@ -61,6 +61,7 @@
     "main_app",
     "articles",
     "django_quill",  # to provide rich-text field for articles
+    "reversion",  # django-reversion, for version history of objects in database
     "users",
 ]
 
@@ -73,6 +74,7 @@
     "django.contrib.messages.middleware.MessageMiddleware",
     "django.middleware.clickjacking.XFrameOptionsMiddleware",
     "django.contrib.flatpages.middleware.FlatpageFallbackMiddleware",
+    "reversion.middleware.RevisionMiddleware",
 ]
 
 ROOT_URLCONF = "cantusdb.urls"
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -1,4 +1,5 @@
 from django.contrib import admin
+from reversion.admin import VersionAdmin
 from main_app.models import *
 from main_app.forms import (
     AdminCenturyForm,
@@ -22,7 +23,7 @@
 )
 
 
-class BaseModelAdmin(admin.ModelAdmin):
+class BaseModelAdmin(VersionAdmin):
     exclude = EXCLUDE
 
     # if an object is created in the admin interface, assign the user to the created_by field


Test patch:
Repository: DDMAL/CantusDB

Issue:
typo in "Bibliography" 
Source view has "bibilography" and not bibliography:
<img width="250" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/42a71b35-3598-4e66-abd2-1830d44a9ce6">
Source edit seems to be correct though (https://cantusdatabase.org/edit-source/711225):
<img width="202" alt="image" src="https://github.com/DDMAL/CantusDB/assets/67451875/ac6d001e-c2cb-49af-9e77-a4953bc657a7">



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/admin.py b/django/cantusdb_project/main_app/admin.py
--- a/django/cantusdb_project/main_app/admin.py
+++ b/django/cantusdb_project/main_app/admin.py
@@ -16,15 +16,14 @@
 )
 
 # these fields should not be editable by all classes
-EXCLUDE = (
-    "json_info",
-)
+EXCLUDE = ("json_info",)
 
 READ_ONLY = (
     "created_by",
     "last_updated_by",
 )
 
+
 class BaseModelAdmin(VersionAdmin):
     exclude = EXCLUDE
     readonly_fields = READ_ONLY


Test patch:
Repository: DDMAL/CantusDB

Issue:
We should make sure our `ALLOWED_HOSTS` is complete
As noted by @dchiller in https://github.com/DDMAL/CantusDB/pull/1286#discussion_r1476453258

> If I understand correctly, our staging/production sites are configured to receive traffic from `www.cantusdatabase.org`, `mass.cantusdatabase.org`, `staging.cantusdatabase.org`, etc., and we open ourself up to security issues if these hosts are not all included in `ALLOWED_HOSTS`.


Discussion:
Perhaps this needs to be in our `CSRF_TRUSTED_ORIGINS`.

Trying to log in on staging:
![Screenshot 2024-02-08 at 1 31 37 PM](https://github.com/DDMAL/CantusDB/assets/58090591/c704dc37-eb0e-40c2-882b-a6bf727fdbb1)


Resolving patch:
diff --git a/django/cantusdb_project/cantusdb/settings.py b/django/cantusdb_project/cantusdb/settings.py
--- a/django/cantusdb_project/cantusdb/settings.py
+++ b/django/cantusdb_project/cantusdb/settings.py
@@ -35,11 +35,21 @@
 # SECURITY WARNING: keep the secret key used in production secret!
 SECRET_KEY = os.getenv("CANTUSDB_SECRET_KEY")
 
+PROJECT_ENVIRONMENT = os.getenv("PROJECT_ENVIRONMENT")
+
 # SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = bool(strtobool(os.getenv("CANTUSDB_DEBUG", "False")))
-# need to set this to false so that we can display the custom 404 page
+DEBUG = False  # this is switched to True below when PROJECT_ENVIRONMENT=="DEVELOPMENT"
 
-ALLOWED_HOSTS = [os.getenv("CANTUSDB_HOSTS")]
+if PROJECT_ENVIRONMENT == "DEVELOPMENT":
+    ALLOWED_HOSTS = os.getenv("CANTUSDB_HOSTS_DEVELOPMENT").split(" ")
+    CSRF_TRUSTED_ORIGINS = os.getenv("CANTUSDB_ORIGINS_DEVELOPMENT").split(" ")
+    DEBUG = True
+if PROJECT_ENVIRONMENT == "STAGING":
+    ALLOWED_HOSTS = os.getenv("CANTUSDB_HOSTS_STAGING").split(" ")
+    CSRF_TRUSTED_ORIGINS = os.getenv("CANTUSDB_ORIGINS_STAGING").split(" ")
+if PROJECT_ENVIRONMENT == "PRODUCTION":
+    ALLOWED_HOSTS = os.getenv("CANTUSDB_HOSTS_PRODUCTION").split(" ")
+    CSRF_TRUSTED_ORIGINS = os.getenv("CANTUSDB_ORIGINS_PRODUCTION").split(" ")
 
 
 # Application definition
@@ -200,8 +210,6 @@
     "127.0.0.1",
 ]
 
-CSRF_TRUSTED_ORIGINS = ["https://cantusdatabase.org", "https://www.cantusdatabase.org"]
-
 if DEBUG:
     INSTALLED_APPS.append("debug_toolbar")
     # debug toolbar must be inserted as early in the middleware as possible


Test patch:
Repository: DDMAL/CantusDB

Issue:
We should ensure header fits within bootstrap's columns
![Screen Shot 2024-02-23 at 10 40 52](https://github.com/DDMAL/CantusDB/assets/58090591/35480ddc-b241-4731-84d5-2bd9017a1a9c)

related to #1319, necessary before we can show a suggested UI update to Debra.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -1033,7 +1033,7 @@ class ChantDeleteView(LoginRequiredMixin, UserPassesTestMixin, DeleteView):
     """
 
     model = Chant
-    template_name = "chant_confirm_delete.html"
+    template_name = "chant_delete.html"
 
     def test_func(self):
         user = self.request.user
diff --git a/django/cantusdb_project/main_app/views/source.py b/django/cantusdb_project/main_app/views/source.py
--- a/django/cantusdb_project/main_app/views/source.py
+++ b/django/cantusdb_project/main_app/views/source.py
@@ -196,7 +196,7 @@ def get_queryset(self):
 
 class SourceCreateView(LoginRequiredMixin, UserPassesTestMixin, CreateView):
     model = Source
-    template_name = "source_create_form.html"
+    template_name = "source_create.html"
     form_class = SourceCreateForm
 
     def test_func(self):
@@ -232,7 +232,7 @@ class SourceDeleteView(LoginRequiredMixin, UserPassesTestMixin, DeleteView):
     """
 
     model = Source
-    template_name = "source_confirm_delete.html"
+    template_name = "source_delete.html"
 
     def test_func(self):
         user = self.request.user


Test patch:
diff --git a/django/cantusdb_project/main_app/tests/test_views.py b/django/cantusdb_project/main_app/tests/test_views.py
--- a/django/cantusdb_project/main_app/tests/test_views.py
+++ b/django/cantusdb_project/main_app/tests/test_views.py
@@ -2896,7 +2896,7 @@ def test_url_and_templates(self):
 
         response = self.client.get(reverse("chant-delete", args=[chant.id]))
         self.assertEqual(response.status_code, 200)
-        self.assertTemplateUsed(response, "chant_confirm_delete.html")
+        self.assertTemplateUsed(response, "chant_delete.html")
 
         response = self.client.get(reverse("chant-delete", args=[chant.id + 100]))
         self.assertEqual(response.status_code, 404)
@@ -4075,7 +4075,7 @@ def setUp(self):
     def test_url_and_templates(self):
         response = self.client.get(reverse("source-create"))
         self.assertEqual(response.status_code, 200)
-        self.assertTemplateUsed(response, "source_create_form.html")
+        self.assertTemplateUsed(response, "source_create.html")
 
     def test_create_source(self):
         response = self.client.post(
Repository: DDMAL/CantusDB

Issue:
views/chant.py: we have a function, `parse_json_from_ci_api`, that we don't use
We use the functions in `cantusindex.py` - the function in views.chant is no longer necessary. Removing it doesn't cause any tests to fail.


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -75,54 +75,6 @@
 )
 
 
-def parse_json_from_ci_api(path: str) -> Optional[list]:
-    """Queries a remote api from cantusindex.org that returns a json object, processes it and
-    returns a list containing its information.
-
-    We expect an array of javascript objects with a byte order mark (BOM) at the beginning
-
-    Args:
-        path (str): the path to the API. This should be just the path (not including the
-            domain), and should not include a leading slash.
-
-    Returns:
-        list, None: contents of json response in the form of a list
-    """
-    assert path[0] != "/"
-    url: str = f"https://cantusindex.org/{path}"
-    try:
-        response: Optional[Response] = requests.get(
-            url,
-            timeout=5,
-        )
-    except (
-        SSLError,
-        Timeout,
-        ConnectionError,
-    ) as exc:
-        print(  # eventually, we should log this rather than printing it to the console
-            "Encountered an error in parse_json_from_ci_api",
-            f"while making a request to {url}:",
-            exc,
-        )
-        return None
-    if response:
-        # we can't use response.json() because of the BOM at the beginning of json export
-        try:
-            return json.loads(response.text[2:])
-        except (
-            json.decoder.JSONDecodeError
-        ) as exc:  # in case of json.loads("not valid json")
-            print(  # eventually, we should log this rather than printing it to the console
-                "Encountered an error in parse_json_from_ci_api",
-                f"while parsing the response from {url}",
-                exc,
-            )
-            return None
-    else:
-        return None
-
-
 def get_feast_selector_options(
     source: Source, folios: list[str]
 ) -> list[tuple[str, Feast]]:


Test patch:
Repository: DDMAL/CantusDB

Issue:
Bad redirect for `/search`



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/urls.py b/django/cantusdb_project/main_app/urls.py
--- a/django/cantusdb_project/main_app/urls.py
+++ b/django/cantusdb_project/main_app/urls.py
@@ -370,6 +370,11 @@
         ChantSearchMSView.as_view(),
         name="chant-search-ms",
     ),
+    path(
+        "search/",
+        views.redirect_search,
+        name="redirect-search",
+    ),
     path(
         "ajax/search-bar/<str:search_term>",
         views.ajax_search_bar,
diff --git a/django/cantusdb_project/main_app/views/views.py b/django/cantusdb_project/main_app/views/views.py
--- a/django/cantusdb_project/main_app/views/views.py
+++ b/django/cantusdb_project/main_app/views/views.py
@@ -2,7 +2,7 @@
 from typing import Optional, Union
 from django.db.models.query import QuerySet
 from django.http.response import JsonResponse
-from django.http import HttpResponse, HttpResponseNotFound
+from django.http import HttpResponse, HttpResponseNotFound, HttpRequest
 from django.utils.http import urlencode
 from django.shortcuts import render, redirect
 from django.urls.base import reverse
@@ -903,6 +903,19 @@ def redirect_genre(request) -> HttpResponse:
     return redirect("genre-list")
 
 
+def redirect_search(request: HttpRequest) -> HttpResponse:
+    """
+    Redirects from search/ (à la OldCantus) to chant-search/ (à la NewCantus)
+
+    Args:
+        request
+
+    Returns:
+        HttpResponse
+    """
+    return redirect("chant-search", permanent=True)
+
+
 def redirect_documents(request) -> HttpResponse:
     """Handle requests to old paths for various
     documents on OldCantus, returning an HTTP Response


Test patch:
Repository: DDMAL/CantusDB

Issue:
Django debug toolbar is not working
I just noticed that while working in my development environment (devcontainer setup), the Django debug toolbar doesn't show. Perhaps this was as a result of #1454? 


Discussion:
I've got it to work by running `request.META["REMOTE_ADDR"]` to one of our views, which has exposed the port 172.24.0.4 as the IP host in the django container. So if I add this IP to `INTERNAL_IPS` it fixes the issue, but this solution is hardcoding the IP address which isn't really useful. Is there a way of making this dynamic in settings.py?
This works too, but not sure if it will be the same for everyone:
```
if DEBUG:
    import socket
    ip = socket.gethostbyname(socket.gethostname())
    INTERNAL_IPS += [ip[:-1] + "4"]
    INSTALLED_APPS.append("debug_toolbar")
    # debug toolbar must be inserted as early in the middleware as possible
    MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")
```
I was frustrated by this too the other day!
Was just about to post the same thing: 

```python
    import socket

    INTERNAL_IPS = [
        ".".join(socket.gethostbyname("host.docker.internal").split(".")[:-1]) + ".1"
    ]

```
What's odd to me is that this is similar to what the `debug_toolbar` already has in its callback: 

https://github.com/jazzband/django-debug-toolbar/blob/7271cac9eacb7870200d6bc965c84eb91ba36714/debug_toolbar/middleware.py#L19-L38


Interesting. I remember when setting this up seeing this warning in their documentation:
https://github.com/jazzband/django-debug-toolbar/issues/1854

But now it's gone. I think their solution assumed that the docker IP would always end with 1, but it ends in 4 for our (or my) case which is why it didn't work.
Now, while using the debug toolbar to try and see how many SQL queries are being made, I get the following error:
```
AttributeError at /
'Connection' object has no attribute 'status'
```
I think this is probably linked to https://github.com/jazzband/django-debug-toolbar/issues/1754. We have `django_debug_toolbar = "3.8.1"` in our dev dependencies and `psycopg  3.1.18` which are incompatible.

Resolving patch:
diff --git a/django/cantusdb_project/cantusdb/settings.py b/django/cantusdb_project/cantusdb/settings.py
--- a/django/cantusdb_project/cantusdb/settings.py
+++ b/django/cantusdb_project/cantusdb/settings.py
@@ -197,6 +197,9 @@
         "debug_toolbar.panels.redirects.RedirectsPanel",
         "debug_toolbar.panels.profiling.ProfilingPanel",
     },
+    "SHOW_TOOLBAR_CALLBACK": lambda request: (
+        False if request.headers.get("x-requested-with") == "XMLHttpRequest" else True
+    ),
 }
 
 INTERNAL_IPS = [


Test patch:
Repository: DDMAL/CantusDB

Issue:
"My sources" page and sidebar should be ordered by date updated
How are sources supposed to show up in "My sources"? I am an editor for some sources but they don't seem to be in "my sources", so what determines this?



Discussion:
It looks like the logic is that it includes sources that were either (a) created by you or (b) for which you are listed as a "current editor".

Could you give me an example of such a source, and then I can look at which "role" you have with the source that isn't one of these two?
wait, I found the sources I thought I was lacking after all--weirdly, buried in different places in my (very long) list of sources, even though I thought I was added to both as an editor at the same time pretty recently (so I was expecting them together and near the surface.)  So the logic is sound, it's actually a question of the order things are presented to the user (probably the most recently modified should be at the top?)

Resolving patch:
diff --git a/django/cantusdb_project/main_app/templatetags/helper_tags.py b/django/cantusdb_project/main_app/templatetags/helper_tags.py
--- a/django/cantusdb_project/main_app/templatetags/helper_tags.py
+++ b/django/cantusdb_project/main_app/templatetags/helper_tags.py
@@ -134,7 +134,7 @@ def get_user_source_pagination(context):
         Source.objects.filter(
             Q(current_editors=context["user"]) | Q(created_by=context["user"])
         )
-        .order_by("-date_created")
+        .order_by("-date_updated")
         .distinct()
     )
     paginator = Paginator(user_created_sources, 6)
diff --git a/django/cantusdb_project/main_app/views/user.py b/django/cantusdb_project/main_app/views/user.py
--- a/django/cantusdb_project/main_app/views/user.py
+++ b/django/cantusdb_project/main_app/views/user.py
@@ -104,7 +104,7 @@ def get_context_data(self, **kwargs):
 
         user_created_sources = (
             Source.objects.filter(created_by=self.request.user)
-            .order_by("-date_created")
+            .order_by("-date_updated")
             .distinct()
         )
         user_created_paginator = Paginator(user_created_sources, 6)


Test patch:
Repository: DDMAL/CantusDB

Issue:
Duplicate results show up in source search
From Debra:

> https://cantusdatabase.org/sources/?segment=4063&general=&indexing=klassen&provenance=&century=&fullSource=
> 
> I searched for the name of an indexer and I have many duplicates being returned (of the same thing). Is this finding every instance of that name in a file? Could it instead find the name but only return the source once?


Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/source.py b/django/cantusdb_project/main_app/views/source.py
--- a/django/cantusdb_project/main_app/views/source.py
+++ b/django/cantusdb_project/main_app/views/source.py
@@ -321,8 +321,12 @@ def get_queryset(self):
             )
             q_obj_filter &= indexing_search_q
 
-        return queryset.filter(q_obj_filter).prefetch_related(
-            Prefetch("century", queryset=Century.objects.all().order_by("id"))
+        return (
+            queryset.filter(q_obj_filter)
+            .distinct()
+            .prefetch_related(
+                Prefetch("century", queryset=Century.objects.all().order_by("id"))
+            )
         )
 
 


Test patch:
Repository: DDMAL/CantusDB

Issue:
Un-migrated changes in Sources
There are two minor changes in the Sources introduced here:

https://github.com/DDMAL/CantusDB/commit/1707f76cf8cf53bdeb09ad435df0f26e3b610c4f

These are showing up as not appearing in any migration, though, so the *next* migration to be added will also include these changes. 




Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/migrations/0012_alter_source_date_alter_source_title.py b/django/cantusdb_project/main_app/migrations/0012_alter_source_date_alter_source_title.py
new file mode 100644
--- /dev/null
+++ b/django/cantusdb_project/main_app/migrations/0012_alter_source_date_alter_source_title.py
@@ -0,0 +1,30 @@
+# Generated by Django 4.1.6 on 2024-06-06 12:04
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("main_app", "0011_chant_cm_melody_id_chant_incipit_of_refrain_and_more"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="source",
+            name="date",
+            field=models.CharField(
+                blank=True,
+                help_text='Date of the source (e.g. "1200s", "1300-1350", etc.)',
+                max_length=63,
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="source",
+            name="title",
+            field=models.CharField(
+                help_text="Full Source Identification (City, Archive, Shelf-mark)",
+                max_length=255,
+            ),
+        ),
+    ]


Test patch:
Repository: DDMAL/CantusDB

Issue:
Add GitHub action for running test suite



Discussion:

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -1193,7 +1193,7 @@ def get_initial(self):
         initial = super().get_initial()
         chant = self.get_object()
         has_syl_text = bool(chant.manuscript_syllabized_full_text)
-        syls_text = syllabify_text(
+        syls_text, _ = syllabify_text(
             text=chant.get_best_text_for_syllabizing(),
             clean_text=True,
             text_presyllabified=has_syl_text,


Test patch:
diff --git a/.github/workflows/django_tests.yml b/.github/workflows/django_tests.yml
new file mode 100644
--- /dev/null
+++ b/.github/workflows/django_tests.yml
@@ -0,0 +1,29 @@
+name: django-tests
+on:
+    pull_request:
+        types: [opened, synchronize]
+jobs:
+    run-tests:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v3
+            - run: mkdir -p config/envs
+            - uses: SpicyPizza/create-envfile@v2.0
+              with:
+                envkey_POSTGRES_DB: test_cantusdb
+                envkey_POSTGRES_USER: test_user
+                envkey_POSTGRES_HOST: postgres
+                envkey_POSTGRES_PORT: 5432
+                envkey_PROJECT_ENVIRONMENT: PRODUCTION
+                envkey_CANTUSDB_STATIC_ROOT: /path/to/static
+                envkey_CANTUSDB_MEDIA_ROOT: /path/to/media
+                envkey_CANTUSDB_HOST: somehost
+                envkey_CANTUSDB_SECRET_KEY: "hereisakey1234"
+                envkey_POSTGRES_PASSWORD: woahagreatpasswordabc
+                envkey_AWS_EMAIL_HOST_USER: test_user
+                envkey_AWS_EMAIL_HOST_PASSWORD: test_password
+                directory: config/envs
+                file_name: dev_env
+            - run: docker compose -f docker-compose-development.yml build
+            - run: docker compose -f docker-compose-development.yml up -d
+            - run: docker compose -f docker-compose-development.yml exec -T django python manage.py test main_app.tests
\ No newline at end of file
diff --git a/django/cantusdb_project/main_app/tests/test_views.py b/django/cantusdb_project/main_app/tests/test_views.py
--- a/django/cantusdb_project/main_app/tests/test_views.py
+++ b/django/cantusdb_project/main_app/tests/test_views.py
@@ -3644,11 +3644,11 @@ def test_edit_syllabification(self):
         self.assertEqual(chant.manuscript_syllabized_full_text, "lorem ipsum")
         response = self.client.post(
             f"/edit-syllabification/{chant.id}",
-            {"manuscript_syllabized_full_text": "lo-rem ip-sum"},
+            {"manuscript_syllabized_full_text": "lore-m i-psum"},
         )
         self.assertEqual(response.status_code, 302)  # 302 Found
         chant.refresh_from_db()
-        self.assertEqual(chant.manuscript_syllabized_full_text, "lo-rem ip-sum")
+        self.assertEqual(chant.manuscript_syllabized_full_text, "lore-m i-psum")
 
 
 class FeastListViewTest(TestCase):
Repository: analysiscenter/batchflow

Issue:
Create release gh action
Make release actions:
- build docs
- calc test coverage
- create python package and upload it to pypi.

Actions should fire at various statuses: created/edited, published, etc.


Discussion:
Remove test coverage calc from pr action
@cdtn @SergeyTsimfer Use carry forward flags for coverage (https://docs.codecov.io/docs/carryforward-flags)
@cdtn This issue should be addressed promptly.

Resolving patch:
diff --git a/check_release_version.py b/check_release_version.py
new file mode 100644
--- /dev/null
+++ b/check_release_version.py
@@ -0,0 +1,6 @@
+import re
+import sys
+
+with open('batchflow/__init__.py', 'r') as f:
+    version = "v" + re.search(r'^__version__\s*=\s*[\'"]([^\'"]*)[\'"]', f.read(), re.MULTILINE).group(1)
+sys.exit(version != sys.argv[1])


Test patch: