internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apluslms/a-plus

Issue:
Optimize the very inefficient database queries in the deviation list view
#1122 presented performance issues in the deviations list page. This new issue focuses on specific performance issues.

Using the Django Debug toolbar, we have noticed that the database queries in the deviations list page include some very inefficient parts.

* https://github.com/apluslms/a-plus/blob/8f856141f3af955668df001942a72c696f9a9722/deviations/templates/deviations/list_dl.html#L66
  * rendering the module's name causes new queries for each record because it needs to load the course instance for the module numbering settings (that set Arabic or Roman numerals). Has `select_related()` or `prefetch_related()` been used incorrectly since the code tries to prefetch the course instances?
  * https://github.com/apluslms/a-plus/blob/8f856141f3af955668df001942a72c696f9a9722/deviations/viewbase.py#L254-L265
  * does the variable in the template benefit from the prefetching in the queryset or is the queryset's cache somehow lost when the template is rendered in this case? Does the variable refer to the original queryset anymore at that stage?
* https://github.com/apluslms/a-plus/blob/8f856141f3af955668df001942a72c696f9a9722/deviations/viewbase.py#L27-L29
  * when all deviations in the course are queried, the query includes a lot of unnecessary attributes. Loading them from the database consumes a lot of resources. Especially the large JSON fields in the exercise model consume a lot of bandwidth even though those attributes are not used at all in the deviations list.
  * Unwanted attributes can be avoided with the Django queryset methods `only()` and `defer()`.
  * `exercise.exercise_info` is one of the unnecessary attributes in this case.


Discussion:

Resolving patch:
diff --git a/deviations/viewbase.py b/deviations/viewbase.py
--- a/deviations/viewbase.py
+++ b/deviations/viewbase.py
@@ -257,6 +257,12 @@ def get_deviation_groups(
             'submitter', 'submitter__user',
             'granter', 'granter__user',
             'exercise', 'exercise__course_module',
+            'exercise__course_module__course_instance',
+        )
+        .defer(
+            'exercise__exercise_info',
+            'exercise__description',
+            'exercise__course_module__course_instance__description',
         )
         # parent is prefetched because there may be multiple ancestors, and
         # they are needed for building the deviation's URL.


Test patch:
Repository: apluslms/a-plus

Issue:
Users should not be able to edit content hierarchy
Typically the A+ course content structure is configured from a JSON file generated by gitmanager. However, the teachers have possibility to edit the attributes also in the Edit course / Content view, including the parent learning object. By editing the parent selection it is possible to create a circular reference loop between two learning objects pointing to each other as a parent, that leads to excessive number of database operations, and as a result, leading the system to become unusable due to heavy database load.

Easy approach would be to just disable the possibility to modify the parent selection. Later, it might be useful to think more thoroughly how important it is allow editing the content structure in this view, while the main form of course configuration should be through the JSON configuration.


Discussion:
As discussed in the weekly meeting, we decided to keep the Parent and Course module menus still visible, but make then unchangeable for users.

Resolving patch:
diff --git a/edit_course/exercise_forms.py b/edit_course/exercise_forms.py
--- a/edit_course/exercise_forms.py
+++ b/edit_course/exercise_forms.py
@@ -55,6 +55,8 @@ def init_fields(self, **kwargs):
         self.fields["parent"].queryset = LearningObject.objects\
             .exclude(id=self.lobject.id)\
             .filter(course_module=self.lobject.course_module)
+        self.fields['parent'].widget.attrs.update(
+            {'readonly': True, 'disabled': True})
 
     @property
     def remote_service_head(self):


Test patch:
Repository: apluslms/a-plus

Issue:
Setting a non-unique CourseInstance.url in the edit course instance form crashes A+
Under the Edit course section, in the Course tab, a teacher can change the URL identifier of the `CourseInstance`. The URLs must be unique within the same `Course`. The model has the unique constraint, but A+ still crashes when you try to set a non-unique URL. The form seems to be missing validation and thus, the system tries to save an invalid value to the database, but the database `save()` operation fails due to the integrity constraints inside the relational database. Ensure that the form submission is validated correctly before saving. Invalid values should not be accepted and the user should see a validation error in the form page after submitting.

https://github.com/apluslms/a-plus/blob/dda02c73bfc009801fde61f5a144e4a159bdbd51/course/models.py#L694-L697

`CourseInstanceForm` already includes some unusual workarounds for validating the URL:

https://github.com/apluslms/a-plus/blob/dda02c73bfc009801fde61f5a144e4a159bdbd51/edit_course/course_forms.py#L151-L164

Related issues:
* #781 
* #782 


Discussion:

Resolving patch:
diff --git a/edit_course/course_forms.py b/edit_course/course_forms.py
--- a/edit_course/course_forms.py
+++ b/edit_course/course_forms.py
@@ -191,8 +191,11 @@ def clean_url(self):
         # crash due to invalid data if this method returned an invalid value.
         # Raising a ValidationError here prevents the course instance from
         # having invalid values.
-        generate_url_key_validator()(self.cleaned_data["url"])
-        return self.cleaned_data["url"]
+        generate_url_key_validator()(self.cleaned_data['url'])
+        url = self.cleaned_data['url']
+        if url != self.instance.url and CourseInstance.objects.filter(course=self.instance.course, url=url).exists():
+            raise ValidationError(_('ERROR_URL_ALREADY_TAKEN'))
+        return url
 
     def save(self, *args, **kwargs):
         self.instance.set_assistants(self.cleaned_data['assistants'])


Test patch:
Repository: apluslms/a-plus

Issue:
Setting a unique CourseInstance.url in the edit course instance form crashes A+
Under the Edit course section, in the Course tab, a teacher can change the URL identifier of the CourseInstance. The URLs must be unique within the same Course. When editing the url to a unique value and saving the changes, the page crashes as the old url is being called, which no longer exits.
<img width="856" alt="image" src="https://github.com/apluslms/a-plus/assets/150142776/dade12f2-d9a9-4f7b-aca8-764cc4ac13ae">



Discussion:

Resolving patch:
diff --git a/edit_course/views.py b/edit_course/views.py
--- a/edit_course/views.py
+++ b/edit_course/views.py
@@ -55,7 +55,7 @@ def get_success_url(self):
     def form_valid(self, form):
         self.instance = form.save()
         SecurityLog.logevent(self.request, "course-edit", "course: {}, teachers: {}, assistants: {}".format(
-            self.get_course_instance_object().course,
+            self.instance.course,
             list(map(lambda x: str(x.user), form.instance.teachers.all())),
             list(map(lambda x: str(x.user), form.instance.assistants.all()))
         ))


Test patch:
Repository: apluslms/a-plus

Issue:
Build log URL updates only after course configuring is done
Build log URL does not update immediately when creating a new course instance, instead it uses the build log URL from the previous course instance until the build is done.

It also does not update immediately when changing the Git Manager key in the Git Manager tab in A+ course settings.

This causes an error when attempting to view the build log in A+ while the course is still being built.


Discussion:

Resolving patch:
diff --git a/edit_course/views.py b/edit_course/views.py
--- a/edit_course/views.py
+++ b/edit_course/views.py
@@ -375,6 +375,10 @@ def form_valid(self, form):
             response = aplus_post(gitmanager_url, permissions=permissions, data=data)
             if response.status_code == 200:
                 new_instance.configure_url = urllib.parse.urljoin(settings.GITMANAGER_URL, f"{key}/aplus-json")
+                new_instance.build_log_url = urllib.parse.urljoin(
+                    settings.GITMANAGER_URL,
+                    f"gitmanager/{key}/build_log-json",
+                )
                 new_instance.save()
                 gitmanager_hook_url = urllib.parse.urljoin(settings.GITMANAGER_URL, f"gitmanager/{key}/hook")
                 response = aplus_post(gitmanager_hook_url, permissions=permissions, data={"key": key})
@@ -423,6 +427,10 @@ def form_valid(self, form):
             if response.status_code == 200:
                 key = form.cleaned_data['key']
                 self.instance.configure_url = urllib.parse.urljoin(settings.GITMANAGER_URL, f"{key}/aplus-json")
+                self.instance.build_log_url = urllib.parse.urljoin(
+                    settings.GITMANAGER_URL,
+                    f"gitmanager/{key}/build_log-json",
+                )
                 self.instance.save()
                 gitmanager_hook_url = urllib.parse.urljoin(settings.GITMANAGER_URL, f"gitmanager/{key}/hook")
                 response = aplus_post(gitmanager_hook_url, permissions=permissions, data={"key": key})


Test patch:
Repository: apluslms/a-plus

Issue:
Add a gitmanager link with automatic JWT auth to the A+ edit course section
Under the A+ Edit course section, there should be a link to the gitmanager so that teachers can easily open the gitmanager web site and view the course's gitmanager settings and build log. The link should include the JWT authentication tokens automatically so that the gitmanager login does not require extra work. With the link, the teachers can read the full build log in the current gitmanager web UI.

Related to #835.


Discussion:

Resolving patch:
diff --git a/edit_course/views.py b/edit_course/views.py
--- a/edit_course/views.py
+++ b/edit_course/views.py
@@ -397,6 +397,11 @@ class EditGitmanagerView(CourseInstanceMixin, BaseFormView):
     template_name = "edit_course/edit_gitmanager.html"
     form_class = GitmanagerForm
 
+    def get_common_objects(self) -> None:
+        super().get_common_objects()
+        self.gitmanager_url = settings.GITMANAGER_URL
+        self.note('gitmanager_url')
+
     def get_form_kwargs(self):
         kwargs = super().get_form_kwargs()
         kwargs["instance"] = self.instance


Test patch:
Repository: apluslms/a-plus

Issue:
Login should not take the user to the front page
Common scenario: the user is browsing a particular course module. They aren’t logged in. They decide to log in, but doing so takes them to the A+ front page, from which they have to navigate back to where they were. Inconvenient.


Discussion:
redirect to request.path - https://docs.djangoproject.com/en/5.0/topics/auth/default/#the-raw-way

Resolving patch:
diff --git a/course/templatetags/base.py b/course/templatetags/base.py
--- a/course/templatetags/base.py
+++ b/course/templatetags/base.py
@@ -2,6 +2,8 @@
 
 from django import template
 from django.conf import settings
+from django.urls import resolve
+from django.urls.exceptions import Resolver404
 from django.utils.safestring import mark_safe
 from django.utils.text import format_lazy
 from django.utils.translation import get_language, gettext_lazy as _
@@ -28,6 +30,15 @@ def get_date(cont, key):
         cont[key] = data
     return data
 
+@register.simple_tag(takes_context=True)
+def login_next(context):
+    request = context["request"]
+    try:
+        matched_url_name = resolve(request.path).url_name
+        next_path = f"?next={request.path}" if matched_url_name != 'logout' else ""
+        return next_path
+    except Resolver404:
+        return ""
 
 @register.simple_tag
 def brand_name():


Test patch:
Repository: apluslms/a-plus

Issue:
API: Add closing_time and opening_time to modules
Right now, there is no way to get the "closing_time" and "opening_time" properties for modules with any api call.

It would be useful to have it at least in the `GET /courses/<course_id>/exercises/` endpoint


Discussion:

Resolving patch:
diff --git a/course/api/views.py b/course/api/views.py
--- a/course/api/views.py
+++ b/course/api/views.py
@@ -245,6 +245,9 @@ def __module_to_dict(self, module: ModuleContent, **kwargs) -> Dict[str, Any]:
                 module.opening_time,
                 module.closing_time,
             ),
+            'reading_opening_time': module.reading_opening_time,
+            'opening_time': module.opening_time,
+            'closing_time': module.closing_time,
         }
         module_dictionary['exercises'] = self.__recurse_exercises(module, [])
         return module_dictionary


Test patch:
Repository: apluslms/a-plus

Issue:
Running unit tests produces many warnings
A lot of warnings are generated when running the unit tests using the [develop-aplus](https://github.com/apluslms/develop-aplus) repository and `pytest`.

The related code should be fixed if possible, so that the test output is cleaner.


Discussion:

Resolving patch:
diff --git a/aplus/settings.py b/aplus/settings.py
--- a/aplus/settings.py
+++ b/aplus/settings.py
@@ -391,7 +391,6 @@
 ]
 TIME_ZONE = 'EET'
 USE_I18N = True
-USE_L10N = True
 USE_TZ = True
 FORMAT_MODULE_PATH = 'aplus'
 LOCALE_PATHS = (
diff --git a/external_services/api/authentication.py b/external_services/api/authentication.py
--- a/external_services/api/authentication.py
+++ b/external_services/api/authentication.py
@@ -54,12 +54,13 @@ def verify_oauth_body_hash_and_signature(request, req_body_hash, lti_exercise=No
 
     # check the OAuth timestamp. Do not allow old requests in order to prevent replay attacks.
     try:
-        timestamp = datetime.datetime.utcfromtimestamp(int(req_oauth_params_dict.get('oauth_timestamp')))
+        timestamp = datetime.datetime.fromtimestamp(int(req_oauth_params_dict.get('oauth_timestamp')),
+                                                    datetime.timezone.utc)
         # oauth_timestamp: seconds since January 1, 1970 00:00:00 GMT
     except ValueError:
         return False, 'oauth_timestamp is missing or has an invalid format'
 
-    now = datetime.datetime.utcnow()
+    now = datetime.datetime.now(datetime.timezone.utc)
     delta = datetime.timedelta(seconds=OAuthNonceCache.CACHE_TIMEOUT_SECONDS)
     if not (now - delta < timestamp and timestamp < now + delta): # pylint: disable=chained-comparison
         return False, 'oauth_timestamp has expired'


Test patch:
diff --git a/course/sis_test.py b/course/sis_test.py
--- a/course/sis_test.py
+++ b/course/sis_test.py
@@ -29,8 +29,8 @@
         "code": "123456",
         "id": "123",
         "instance": "summer-2022",
-        "starting_time": "2022-06-01",
-        "ending_time": "2022-08-20",
+        "starting_time": "2022-06-01T00:00:00+03:00",
+        "ending_time": "2022-08-20T00:00:00+03:00",
         "teachers": [ "teacher-A", "testTeacher" ],
         "participants": [ "123TEST", "333333", "555555" ],
     },
diff --git a/lib/cache/tests.py b/lib/cache/tests.py
--- a/lib/cache/tests.py
+++ b/lib/cache/tests.py
@@ -261,6 +261,9 @@ def t2():
 
 
 class TestCached(CachedAbstract):
+
+    __test__ = False # Prevents this class from being picked up by pytest
+
     def __init__(self, func):
         self._fake_func = func
         super().__init__()
diff --git a/lib/tests.py b/lib/tests.py
--- a/lib/tests.py
+++ b/lib/tests.py
@@ -7,6 +7,7 @@
 
 
 class TestGlobal(RequestGlobal):
+    __test__ = False # Prevents this class from being picked up by pytest
     test = None
     def init(self):
         self.test = "test"
Repository: apluslms/a-plus

Issue:
API endpoint for batch fetching student points
A+ has the following API endpoints for fetching points:

`GET /courses/<course_id>/points/` - for fetching a list of students
`GET /courses/<course_id>/points/<user_id>/` - for fetching the points of an individual student

In Grades, we want to fetch points for all students of a course. The only way to do that currently is to first fetch a list of students, and then for each student, individually fetch their points. Courses can have up to hundreds of participants, so this means there will be hundreds of API calls.

An API endpoint for fetching points for all students at once would be helpful. For each student, we need:
- Their full points
- Their points for each module
- Their points by each difficulty level


Discussion:

Resolving patch:
diff --git a/exercise/api/views.py b/exercise/api/views.py
--- a/exercise/api/views.py
+++ b/exercise/api/views.py
@@ -26,7 +26,6 @@
     OnlyCourseStaffPermission,
 )
 from course.api.mixins import CourseResourceMixin
-from course.api.serializers import StudentBriefSerializer
 from exercise.async_views import _post_async_submission
 
 from ..models import (
@@ -551,7 +550,7 @@ class CoursePointsViewSet(ListSerializerMixin,
     lookup_url_kwarg = 'user_id'
     lookup_value_regex = REGEX_INT_ME
     parent_lookup_map = {}
-    listserializer_class = StudentBriefSerializer
+    listserializer_class = UserPointsSerializer
     serializer_class = UserPointsSerializer
 
     def get_queryset(self):


Test patch:
Repository: PrefectHQ/prefect

Issue:
`Flow.visualize()` should be able to display inline in a Jupyter notebook
Whether by user-provided `kwarg` or by magic, `Flow.visualize()` should be able to render inline in IPython notebooks.


Discussion:

Resolving patch:
diff --git a/src/prefect/core/flow.py b/src/prefect/core/flow.py
--- a/src/prefect/core/flow.py
+++ b/src/prefect/core/flow.py
@@ -515,6 +515,9 @@ def serialize(self) -> dict:
     # Visualization ------------------------------------------------------------
 
     def visualize(self):
+        """
+        Creates graphviz object for representing the current flow
+        """
         graph = graphviz.Digraph()
 
         for t in self.tasks:
@@ -523,6 +526,12 @@ def visualize(self):
         for e in self.edges:
             graph.edge(str(id(e.upstream_task)), str(id(e.downstream_task)), e.key)
 
+        try:
+            if get_ipython().config.get("IPKernelApp") is not None:
+                return graph
+        except NameError:
+            pass
+
         with tempfile.NamedTemporaryFile() as tmp:
             graph.render(tmp.name, view=True)
 


Test patch:
Repository: PrefectHQ/prefect

Issue:
Make GraphViz optional
Following up on #101 -- I think GraphViz should be optional. Should we have a "viz" `extras` package in `setup.py` that includes bokeh and graphviz?


Discussion:
I like that idea; and should we also make sure `dev` includes everything?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,18 +10,17 @@
     "dask >= 0.18, < 0.19",
     "distributed >= 1.21.8, < 2.0",
     "docker >= 3.4.1, < 3.5",
-    "graphviz >= 0.8.3, < 0.9",
     "mypy_extensions >= 0.3.0, < 0.4",
     "python-dateutil >= 2.7.3, < 3.0",
     "requests >= 2.19.1, < 3.0",
     "toml >= 0.9.4, < 1.0",
-    "toolz >= 0.9.0, < 1.0",
     "typing >= 3.6.4, < 4.0",
     "typing_extensions >= 3.6.4, < 4.0",
     "xxhash >= 1.2.0, < 2.0",
 ]
 
-extras = {"dev": ["pytest", "pytest-env", "pytest-xdist"]}
+viz = ["bokeh == 0.13.0", "graphviz >= 0.8.3"]
+extras = {"dev": ["pytest", "pytest-env", "pytest-xdist"] + viz, "viz": viz}
 
 setup(
     name="prefect",
diff --git a/src/prefect/core/flow.py b/src/prefect/core/flow.py
--- a/src/prefect/core/flow.py
+++ b/src/prefect/core/flow.py
@@ -20,7 +20,6 @@
     Union,
 )
 
-import graphviz
 import xxhash
 from mypy_extensions import TypedDict
 
@@ -741,8 +740,20 @@ def run(
     def visualize(self):
         """
         Creates graphviz object for representing the current flow
+
+        Raises:
+            - ImportError: if `graphviz` is not installed
         """
 
+        try:
+            import graphviz
+        except ImportError:
+            msg = (
+                "This feature requires graphviz.\n"
+                "Try re-installing prefect with `pip install prefect[viz]`"
+            )
+            raise ImportError(msg)
+
         graph = graphviz.Digraph()
 
         for t in self.tasks:
diff --git a/src/prefect/utilities/bokeh_runner.py b/src/prefect/utilities/bokeh_runner.py
--- a/src/prefect/utilities/bokeh_runner.py
+++ b/src/prefect/utilities/bokeh_runner.py
@@ -7,6 +7,16 @@
 import prefect
 
 
+try:
+    import bokeh
+except ImportError:
+    msg = (
+        "This feature requires bokeh.\n"
+        "Try re-installing prefect with `pip install prefect[viz]`"
+    )
+    raise ImportError(msg)
+
+
 class BokehRunner(prefect.engine.flow_runner.FlowRunner):
     """
     The BokehRunner class is a special [FlowRunner](../engine/flow_runner.html) subclass meant for debugging, exploring, visualizing, and


Test patch:
diff --git a/tests/core/test_flow.py b/tests/core/test_flow.py
--- a/tests/core/test_flow.py
+++ b/tests/core/test_flow.py
@@ -700,6 +700,20 @@ def test_serialize_build():
     assert isinstance(flow.serialize(build=True)["environment_key"], bytes)
 
 
+def test_visualize_raises_informative_importerror_without_graphviz(monkeypatch):
+    f = Flow()
+    f.add_task(Task())
+
+    import sys
+
+    with monkeypatch.context() as m:
+        m.setattr(sys, "path", "")
+        with pytest.raises(ImportError) as exc:
+            f.visualize()
+
+    assert "pip install prefect[viz]" in repr(exc.value)
+
+
 class TestCache:
     def test_cache_created(self):
         f = Flow()
Repository: PrefectHQ/prefect

Issue:
Add coverage report to GitHub
After `pip install pytest-cov`, can run `pytest --cov` to get a coverage report for unit tests.


Discussion:

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -15,8 +15,10 @@
 import os
 import re
 import shutil
+import subprocess
 import textwrap
 import toolz
+import warnings
 
 import prefect
 from prefect.utilities.bokeh_runner import BokehRunner
@@ -363,6 +365,22 @@ def format_subheader(obj, level=1, in_table=False):
     return call_sig
 
 
+def generate_coverage():
+    """
+    Generates a coverage report in a subprocess; if one already exists,
+    will _not_ recreate for the sake of efficiency
+    """
+    if os.path.exists(".vuepress/public/prefect-coverage"):
+        return
+
+    tests = subprocess.check_output(
+        "cd .. && pytest --cov-report html:docs/.vuepress/public/prefect-coverage --cov=src/prefect",
+        shell=True,
+    )
+    if "failed" in tests.decode():
+        warnings.warn("Some tests failed.")
+
+
 if __name__ == "__main__":
 
     assert (
@@ -379,6 +397,7 @@ def format_subheader(obj, level=1, in_table=False):
 
     shutil.rmtree("api", ignore_errors=True)
     os.makedirs("api", exist_ok=True)
+    generate_coverage()
     with open("api/README.md", "w+") as f:
         f.write("# API Documentation\n")
         f.write(
@@ -387,6 +406,10 @@ def format_subheader(obj, level=1, in_table=False):
                 short_sha=SHORT_SHA, git_sha=GIT_SHA
             )
         )
+        f.write(
+            "\n\n"
+            "*Click <a href='/prefect-coverage/index.html'>here</a> for a complete test coverage report.*"
+        )
 
         with open("../README.md", "r") as g:
             readme = g.read()


Test patch:
Repository: PrefectHQ/prefect

Issue:
Add instructions for pre-push black hook
This would be extremely useful to have in the dev instructions. Also, I'd love to know how to do it myself :)


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@
 
 install_requires = [
     "click >= 6.7, < 7.0",
-    "cloudpickle >= 0.5.3, < 0.6.0",
+    "cloudpickle == 0.5.3",
     "croniter >= 0.3.23, < 0.4",
     "cryptography >= 2.2.2, < 3.0",
     "dask >= 0.18, < 0.19",


Test patch: