internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apluslms/a-plus

Issue:
Compress HTML pages in the cache
Exercise and chapter pages are stored in the cache and only update if the backend reports that there is a change. Some pages might be large (e.g. 1M), but do compress relatively well. Memcached API supports compression on the fly, but that is not usable over django API.

Thus, we should at least compress HTML content manually. Alternatively, we can specialize `CachedAbstract` for memcached, which would also allow us to use `cas` operation.

Relevant files:
* `lib/cache/cached.py`
* `exercise/cache/exercise.py` (`content` in `_generate_data(...)` and `content()`)


Discussion:
This patch shows location, where this could be implemented (seer elated files above)

```patch
--- a/exercise/cache/exercise.py
+++ b/exercise/cache/exercise.py
@@ -28,7 +28,7 @@ class ExerciseCache(CachedAbstract):
             )
             return {
                 'head': page.head,
-                'content': page.content,
+                'content': page.content, # FIXME: add compress
                 'last_modified': page.last_modified,
                 'expires': page.expires if page.is_loaded else 0,
             }
@@ -41,6 +41,7 @@ class ExerciseCache(CachedAbstract):
         return self.data['head']
 
     def content(self):
+        # FIXME: add decompress
         return self.data['content']
 ```

Resolving patch:
diff --git a/exercise/cache/exercise.py b/exercise/cache/exercise.py
--- a/exercise/cache/exercise.py
+++ b/exercise/cache/exercise.py
@@ -1,4 +1,6 @@
+import logging
 import time
+
 from django.conf import settings
 from django.db.models.signals import post_save, post_delete
 
@@ -6,6 +8,18 @@
 from lib.remote_page import RemotePageNotModified
 from ..protocol.aplus import load_exercise_page
 
+logger = logging.getLogger('aplus.cached')
+
+try:
+    from lz4.block import compress as _compress, decompress
+    def compress(data):
+        return _compress(data, compression=1)
+except ImportError:
+    logger.warning("Unable to import lz4, using a slower zlib instead")
+    from zlib import compress as _compress, decompress
+    def compress(data):
+        return _compress(data, level=1)
+
 
 class ExerciseCache(CachedAbstract):
     """ Exercise HTML content """
@@ -26,9 +40,12 @@ def _generate_data(self, exercise, data=None):
                 *self.load_args,
                 last_modified=data['last_modified'] if data else None
             )
+
+            content = compress(page.content.encode('utf-8'))
+
             return {
                 'head': page.head,
-                'content': page.content,
+                'content': content,
                 'last_modified': page.last_modified,
                 'expires': page.expires if page.is_loaded else 0,
             }
@@ -41,7 +58,8 @@ def head(self):
         return self.data['head']
 
     def content(self):
-        return self.data['content']
+        content = decompress(self.data['content']).decode('utf-8')
+        return content
 
 
 def invalidate_instance(instance):


Test patch:
Repository: apluslms/a-plus

Issue:
Add max_submissions in the course points API endpoint
This API endpoint `/api/v2/courses/COURSEID/points/USERID/` for course points shows the user's points in exercises. It also includes some exercise info that is also provided by the exercise endpoint (`/api/v2/exercises/ID/`). The points API already contains `max_points` and `points_to_pass`, so why not also include `max_submissions` (maximum number of allowed submissions)?

Relevant code:
https://github.com/apluslms/a-plus/blob/d34f94b7c9a8e4d22cd28d08968b36787bb344c4/exercise/api/views.py#L420
https://github.com/apluslms/a-plus/blob/d34f94b7c9a8e4d22cd28d08968b36787bb344c4/exercise/api/custom_serializers.py#L76


Discussion:
@raphendyr are you against adding `max_submissions` to the course points API endpoint since the same data can already be retrieved from the exercise endpoint? The points endpoint already includes other exercise details like `max_points`, so excluding `max_submissions` feels like an oversight.
I guess it's fine. It would be more clear after creating endpoint with the exercise tree as points should be merged with that to get the full picture. max_points and max_submissions kind of belong to the exercise endpoint, but for the convenience those could be included in the points output too. Though, it should be clear that they should be the same _always_ (between points and exercises).

To my knowledge, points endpoint was created to make it possible to render "you have tso many points from stuff A, B and C", so only the information required were added.
Would it make sense to add the `max_submissions` to `ExercisePointsSerializer`  this would mean that  `/api/v2/exercises/EXERCISEID/submitter_stats/USERID/` would also contain it. 
I think it would only make sense to also have it there too, since it already contains all the same data `max_points` etc. Thoughts @raphendyr ? 
It might be a good idea. I was thinking that the `max_submission` actually does make sense as a per user value, because we have deviations, thus different users might ave different `max_submissions` (for your knowledge too @etanttila). See #424 for requests related this.

If the max_submissions was not calculated to the cache, then it should be added there first.

Resolving patch:
diff --git a/exercise/api/full_serializers.py b/exercise/api/full_serializers.py
--- a/exercise/api/full_serializers.py
+++ b/exercise/api/full_serializers.py
@@ -63,8 +63,6 @@ class Meta(ExerciseBriefSerializer.Meta):
             'course',
             'is_submittable',
             'post_url',
-            'max_points',
-            'max_submissions',
             'exercise_info',
             'templates',
             'submissions',
diff --git a/exercise/api/serializers.py b/exercise/api/serializers.py
--- a/exercise/api/serializers.py
+++ b/exercise/api/serializers.py
@@ -28,6 +28,8 @@ class Meta(AplusModelSerializer.Meta):
             'url',
             'html_url',
             'display_name',
+            'max_points',
+            'max_submissions',
         )
 
 


Test patch:
Repository: apluslms/a-plus

Issue:
Error handling for the submission parameter __aplus__ (group)
The JSON syntax errors are ignored here in the function `_check_submission_allowed` (https://github.com/apluslms/a-plus/blob/495dff2b3b7e4c81e606137ffebfa78345efa007/exercise/exercise_models.py#L581-L584) that is called before any submission is created (in `Submission.objects.create_from_post` that raises an error for JSON syntax errors). This can hide JSON errors in the request. A JSON syntax error would drop the group from the submission and then the submission may be denied due to a required group, so `create_from_post` is never called and the user gets an error about wrong group size or wrong group members, but really there was a syntax error in the JSON of the request.


Discussion:

Resolving patch:
diff --git a/exercise/exercise_models.py b/exercise/exercise_models.py
--- a/exercise/exercise_models.py
+++ b/exercise/exercise_models.py
@@ -581,7 +581,8 @@ def _check_submission_allowed(self, profile, request=None):
             try:
                 group_id = json.loads(request.POST.get('__aplus__', '{}')).get('group')
             except json.JSONDecodeError:
-                pass
+                warnings.append(_("Cannot submit exercise because of invalid JSON in POST data"))
+                return self.SUBMIT_STATUS.INVALID, warnings, students
             if not group_id:
                 group_id = request.POST.get("_aplus_group")
 


Test patch:
Repository: apluslms/a-plus

Issue:
automatic notification for user about arriving on pages of a past course
A+ knows that a course has closed since course instances have opening and closing times. Any closed course instance could show a warning easily. Adding an automatic link to the newest course instance would be easy too.

[Teacher needs spring 2020](https://wiki.aalto.fi/display/EDIT/Teacher+needs+spring+2020): nr 12

To be allocated to one of the summer interns


Discussion:

Resolving patch:
diff --git a/course/views.py b/course/views.py
--- a/course/views.py
+++ b/course/views.py
@@ -106,8 +106,22 @@ def get(self, request, *args, **kwargs):
         elif lti_msg:
             messages.info(request, html.escape(lti_msg))
 
-        return super().get(request, *args, **kwargs)
+        later_instance = None
+
+        if self.instance.is_past:
+            try:
+                later_instance = (
+                    CourseInstance.objects
+                        .get_visible(request.user)
+                        .filter(course=self.course, ending_time__gte=timezone.now())
+                        .latest('starting_time')
+                )
+            except CourseInstance.DoesNotExist:
+                pass
 
+        kwargs['later_instance'] = later_instance
+
+        return super().get(request, *args, **kwargs)
 
 class Enroll(EnrollableViewMixin, BaseRedirectMixin, BaseTemplateView):
     permission_classes = [EnrollInfoVisiblePermission]


Test patch:
Repository: apluslms/a-plus

Issue:
Course points API endpoint should contain points for all submissions
This comes from the IntelliJ project: https://github.com/Aalto-LeTech/intellij-plugin/issues/302

The `/api/v2/courses/<course-id>/points/me/` endpoint should be able to provide points for all submissions (for one student in all exercises of one course). We need to still consider if all points are always included in the output or only when some parameter is given in the request GET query parameters. All points should already be available in the points cache: https://github.com/apluslms/a-plus/blob/9e595a0a902d19bcadeeaff8b3160873b0265f43/exercise/cache/points.py#L98

Let's not modify the existing submissions URL list in order to preserve backwards-compatibility. A new key shall be added to the output.

Example snippet for the output (the existing submissions list only contains the URLs of the submissions):

```
"submissions_and_points": [
  {
    "url": "https://plus.cs.aalto.fi/api/v2/submissions/123/",
    "points": 10
  },
  {
    "url": "https://plus.cs.aalto.fi/api/v2/submissions/456/",
    "points": 5
  }
]
```

Jaakko says that it could be best to add the grade field to the existing `SubmissionBriefSerializer` class. It could be more uniform with the rest of the API.



Discussion:

Resolving patch:
diff --git a/exercise/api/custom_serializers.py b/exercise/api/custom_serializers.py
--- a/exercise/api/custom_serializers.py
+++ b/exercise/api/custom_serializers.py
@@ -49,10 +49,21 @@ def submission_url(submission_id):
                 'submission_id': submission_id
             }, request=request)
 
+        def submission_obj(submission_cached):
+            id_ = submission_cached['id']
+            return {
+                'id': id_,
+                'url': submission_url(id_),
+                'submission_time': submission_cached['date'],
+                'grade': submission_cached['points'],
+            }
+
+        submissions = [submission_obj(s) for s in entry['submissions']]
         exercise_data = {
             'url': exercise_url(entry['id']),
             'best_submission': submission_url(entry['best_submission']),
-            'submissions': [submission_url(s['id']) for s in entry['submissions']],
+            'submissions': [s['url'] for s in submissions],
+            'submissions_with_points': submissions,
         }
         for key in [
             # exercise
diff --git a/exercise/api/serializers.py b/exercise/api/serializers.py
--- a/exercise/api/serializers.py
+++ b/exercise/api/serializers.py
@@ -40,6 +40,7 @@ class Meta(AplusModelSerializer.Meta):
         model = Submission
         fields = (
             'submission_time',
+            'grade',
         )
         extra_kwargs = {
             'url': {


Test patch:
Repository: apluslms/a-plus

Issue:
A+ frontpage shows hidden courses to students
The A+ frontpage was heavily changed in the summer 2020. Unfortunately, it now shows hidden courses to students at least if the student has previously enrolled in the course. If the `CourseInstance` field `visible_to_students` is `False`, then the course must not be shown to students, not even the title of the course in the frontpage.


Discussion:

Resolving patch:
diff --git a/course/cache/menu.py b/course/cache/menu.py
--- a/course/cache/menu.py
+++ b/course/cache/menu.py
@@ -36,7 +36,8 @@ def divider_entry():
 
         enrolled = []
         for instance in profile.enrolled.all():
-            enrolled.append(course_entry(instance))
+            if instance.visible_to_students:
+               enrolled.append(course_entry(instance))
 
         teaching = []
         for course in profile.teaching_courses.all():
diff --git a/course/views.py b/course/views.py
--- a/course/views.py
+++ b/course/views.py
@@ -57,10 +57,13 @@ def get_common_objects(self):
                 if instance not in my_instances:
                     my_instances.append(instance)
             
-            for instance in user.userprofile.enrolled.all().filter(ending_time__gte=end_threshold):
+            for instance in user.userprofile.enrolled.all().filter(
+                    ending_time__gte=end_threshold,
+                    visible_to_students=True,
+                ):
                 if instance not in my_instances:
                     my_instances.append(instance)
-        
+
         all_instances = CourseInstance.objects.get_visible(user).filter(ending_time__gte=end_threshold)
         all_instances = [c for c in all_instances if c not in my_instances]
         


Test patch:
Repository: apluslms/a-plus

Issue:
HTML Plugin admin interface does not show relevant information
Some times we have to copy plugins from previous course instances to the current instance. However, it is difficult to know which plugin belongs to the course we want.

**Current view**
![image](https://user-images.githubusercontent.com/10437475/105320693-25795a80-5bcf-11eb-974a-0bc3fa5ae45c.png)

**Proposed view**
![image](https://user-images.githubusercontent.com/10437475/105320818-5194db80-5bcf-11eb-81b9-73d869a12e26.png)

HTML Plugin admin interface does not show relevant information
Some times we have to copy plugins from previous course instances to the current instance. However, it is difficult to know which plugin belongs to the course we want.

**Current view**
![image](https://user-images.githubusercontent.com/10437475/105320693-25795a80-5bcf-11eb-974a-0bc3fa5ae45c.png)

**Proposed view**
![image](https://user-images.githubusercontent.com/10437475/105320818-5194db80-5bcf-11eb-81b9-73d869a12e26.png)



Discussion:

Resolving patch:
diff --git a/apps/admin.py b/apps/admin.py
--- a/apps/admin.py
+++ b/apps/admin.py
@@ -11,6 +11,12 @@
     ExternalIFramePlugin,
 )
 
+class HTMLPluginAdmin(admin.ModelAdmin):
+    list_display_links = ["title"]
+    list_display = ["title", "course_instance_id", "container_type", "views"]
+
+    def course_instance_id(self, obj):
+        return obj.container_pk
 
 admin.site.register(BaseTab)
 admin.site.register(HTMLTab)
@@ -18,5 +24,5 @@
 admin.site.register(ExternalIFrameTab)
 admin.site.register(BasePlugin)
 admin.site.register(RSSPlugin)
-admin.site.register(HTMLPlugin)
+admin.site.register(HTMLPlugin, HTMLPluginAdmin)
 admin.site.register(ExternalIFramePlugin)


Test patch:
Repository: apluslms/a-plus

Issue:
Edit course menu has outdated link to Glyphicons
A+ edit course - menu - add menu item. The menu item has an icon. The hint in the icon field has outdated link to the Bootstrap documentation. You can not find the right icons in that link anymore. The link must be fixed. That is very easy to do.

https://github.com/apluslms/a-plus/blob/57031587030092ac4c482fde53ff9d9577da0f0f/external_services/models.py#L212

We still use Bootstrap 3. The right link is this: 
https://getbootstrap.com/docs/3.4/components/#glyphicons

The old link lands in the latest Bootstrap version.


Discussion:

Resolving patch:
diff --git a/external_services/models.py b/external_services/models.py
--- a/external_services/models.py
+++ b/external_services/models.py
@@ -209,7 +209,7 @@ class MenuItem(UrlMixin, models.Model):
         max_length=32,
         null=True,
         blank=True,
-        help_text=_("Menu icon style name (else service default), e.g. star see http://getbootstrap.com/components/#glyphicons-glyphs")
+        help_text=_("Menu icon style name (else service default), e.g. star see https://getbootstrap.com/docs/3.4/components/#glyphicons")
     )
     menu_weight = models.IntegerField(
         default=0,


Test patch:
Repository: apluslms/a-plus

Issue:
Add separate button for Aalto login (aside Haka login)
In a recent change (pull request #804 ) Aalto (or other local organization) login button was replaced by general Haka login, that directs user to organization selector, to allow login also using other Haka organization accounts than Aalto. This was an intermediate step (due to difficulties in shibboleth configuration), and a separate button for local organization login should now be added back, as majority of students would be using it, and usually some additional guidance may need to be added, e.g. for open university students to use local organization account instead of other organization.


Discussion:

Resolving patch:
diff --git a/aplus/urls.py b/aplus/urls.py
--- a/aplus/urls.py
+++ b/aplus/urls.py
@@ -31,7 +31,7 @@
 #  Pay attention to the order the URL patterns will be matched!
 urlpatterns = [
     url(r'^admin/', admin.site.urls),
-    url(r'^shibboleth/', include(shibboleth_login.urls)),
+    url(r'^', include(shibboleth_login.urls)),
     url('', include(social_django.urls, namespace='social')),
     url(r'^api/v(?P<version>(2))/', include(api.urls_v2)), # why version in url? doc/api_versioning.md
     url(r'^accounts/', include(userprofile.urls)),
diff --git a/shibboleth_login/urls.py b/shibboleth_login/urls.py
--- a/shibboleth_login/urls.py
+++ b/shibboleth_login/urls.py
@@ -5,7 +5,8 @@
 
 
 urlpatterns = [
-    url(r'^login/$', views.login, name="shibboleth-login"),
+    url(r'^shibboleth/login/$', views.login, name="shibboleth-login"),
+    url(r'^Shibboleth.sso/haka_login$', views.login, name="haka-login"),
 ]
 
 if settings.DEBUG:


Test patch:
Repository: apluslms/a-plus

Issue:
separate shibboleth and haka logins?
https://github.com/apluslms/a-plus/blob/6d6baff429ae543f976e3f292bdf6535541bc475/userprofile/templates/userprofile/login.html#L22

If shibboleth login is enabled, haka-login is also included.  Could these be separate items?  A 'haka_login' flag in settings?



Discussion:

Resolving patch:
diff --git a/aplus/local_settings.example.py b/aplus/local_settings.example.py
--- a/aplus/local_settings.example.py
+++ b/aplus/local_settings.example.py
@@ -68,6 +68,9 @@
 #    # ..more options in aplus/settings.py
 #}
 
+## Haka
+#HAKA_LOGIN = True
+
 ## Database
 #DATABASES = {
 #    'default': {
diff --git a/aplus/settings.py b/aplus/settings.py
--- a/aplus/settings.py
+++ b/aplus/settings.py
@@ -168,6 +168,9 @@
 #    'STUDENT_FILTERS': {3: 'int', 2: 'studentID'},
 #}
 
+## Haka
+#HAKA_LOGIN = True
+
 
 ## Google OAuth2 settings
 
diff --git a/userprofile/views.py b/userprofile/views.py
--- a/userprofile/views.py
+++ b/userprofile/views.py
@@ -35,9 +35,12 @@ class CustomLoginView(LoginView):
     # unless images are hosted on a different domain from the Django app.
     extra_context = {
         'shibboleth_login': 'shibboleth_login' in settings.INSTALLED_APPS,
+        'haka_login': getattr(settings, 'HAKA_LOGIN', False),
         'mooc_login': 'social_django' in settings.INSTALLED_APPS,
         'brand_name': settings_text('BRAND_NAME'),
     }
+    if extra_context['haka_login'] and not extra_context['shibboleth_login']:
+        logger.warning("Shibboleth login not enabled, but Haka login flag set as true.")
 
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)


Test patch:
Repository: apluslms/a-plus

Issue:
Exercise messages are not displayed after the first submission
(From #880, delayed feedback)

After the user makes their first submission to an exercise, any messages the exercise had are never shown again. These include the "Staff can submit assignments without enrolling" message and the "Only your last submission will be graded" message that was added in the delayed feedback update.

This is caused by chapter.js. In `loadLastSubmission`, the entire exercise container is replaced, including the message area:

https://github.com/apluslms/a-plus/blob/8a852f917e815876892ffa82eeca3e08d82d4882/exercise/static/exercise/chapter.js#L796-L799

`update`, which is called after making a submission, has a similar issue:

https://github.com/apluslms/a-plus/blob/8a852f917e815876892ffa82eeca3e08d82d4882/exercise/static/exercise/chapter.js#L325-L327


Discussion:

Resolving patch:
diff --git a/exercise/views.py b/exercise/views.py
--- a/exercise/views.py
+++ b/exercise/views.py
@@ -87,8 +87,6 @@ def get(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponse:
                 SUBMIT_STATUS.NOT_ENROLLED,
             ]
             should_enroll = submission_status == SUBMIT_STATUS.NOT_ENROLLED
-            if self.exercise.grading_mode == BaseExercise.GRADING_MODE.LAST:
-                messages.warning(request, _('ONLY_YOUR_LAST_SUBMISSION_WILL_BE_GRADED'))
 
         if (self.exercise.status == LearningObject.STATUS.MAINTENANCE
               or self.module.status == CourseModule.STATUS.MAINTENANCE):
@@ -210,6 +208,9 @@ def post(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponse:
             page=page, students=students, submission=new_submission))
 
     def submission_check(self, error=False, request=None):
+        if self.exercise.grading_mode == BaseExercise.GRADING_MODE.LAST:
+            # Add warning about the grading mode.
+            messages.warning(self.request, _('ONLY_YOUR_LAST_SUBMISSION_WILL_BE_GRADED'))
         if not self.profile:
             issue = _('SUBMISSION_MUST_SIGN_IN_AND_ENROLL_TO_SUBMIT_EXERCISES')
             messages.error(self.request, issue)


Test patch:
Repository: apluslms/a-plus

Issue:
Model and template files are decoded using wrong character encoding when viewed in A+
For example, when clicking the "Show model answer" button on A+, some characters in the model answer are misrepresented or garbled because of incorrect character encoding being used.

This issue is also present when right-clicking the "Show model answer" button and opening it in a new tab.

The same issue is present with template files.

Correct character encoding is used when inspecting student submissions, so this is only a problem when viewing template and model files specifically.

Example:
![ima](https://user-images.githubusercontent.com/38466145/144580817-62305433-d16f-4a39-9064-533a0209ea1d.png)


Discussion:

Resolving patch:
diff --git a/exercise/views.py b/exercise/views.py
--- a/exercise/views.py
+++ b/exercise/views.py
@@ -368,6 +368,7 @@ def get_common_objects(self):
         for url,name in self.exercise.get_models():
             try:
                 response = request_for_response(url)
+                response.encoding = "UTF-8"
             except RemotePageNotFound:
                 self.models.append({'name': name})
             else:
@@ -391,6 +392,7 @@ def get_common_objects(self):
         for url,name in self.exercise.get_templates():
             try:
                 response = request_for_response(url)
+                response.encoding = "UTF-8"
             except RemotePageNotFound as error:
                 self.templates.append({'name': name})
             else:


Test patch: