Repository: apluslms/a-plus

Issue:
Refactoring the duplication of CachedPoints
(From #880, delayed feedback)

As a result of the delayed feedback update, the `CachedPoints` cache is now duplicated for each student: there is one instance that respects reveal rules and hides points where necessary, and one instance that does not hide points (the latter is used when course staff is viewing the points of a student). The reasoning for this is explained below. In this issue, research if it would be possible to remove this duplication, and instead store two sets of points in the cache (real points, and points where unrevealed exercises are not included) and use a template tag to automatically pick the correct points based on whether the user is staff. Would this complicate the code that uses the cache, or increase the likelihood of accidentally leaking unrevealed points? If the refactoring seems to be worth it, implement it.

---

**Background discussion from #880:**

> Why is it necessary to store two versions of `CachedPoints` for the same student (`is_staff` True/False)? There are other approaches too:
> 
> - store separate fields in the same cache, `real_points` and `display_points` or something like that. The user of the cache needs to choose which one to use. (`real_points` could be called just `points` as before.)
> - just store the real points in the `points` field and require the user of the cache to hide it in some cases.
> 
> You already store `feedback_revealed` and `feedback_reveal_time` fields in the cache, and they would also be useful in the above approaches.
> 
> Why did you choose the approach with the `is_staff` parameter?

> Originally, my changes to `CachedPoints` were much more rudimentary. I basically only stored the `feedback_revealed` and `feedback_reveal_time` values for exercises, and nothing else. This was a problematic approach because there were multiple places in the UI that displayed point sums (e.g. the module headings, the `points_progress` elements). Before delayed feedback, these elements could simply get the `points` value of the container (module/category/whatever) and display that. But now we couldn't do that anymore, because some exercises within the container might use delayed feedback. Originally I had made a new template tag that traversed the cache and calculated the displayed points, but that added unnecessary repetitive computations and made it more complicated to use the cache. So, I decided to move the point calculation logic inside the cache.
> 
> But now that I think about it, I could've moved the calculation code inside the cache without duplicating the cache. I think my idea was to make it easier to use the cache: templates that use it don't need to worry whether to use the `real_points` or `display_points` value, instead you could just use `points` and that would be correct as long as you had initialized the `CachedPoints` instance correctly. In a way, the idea was to reduce the chance for developers to accidentally leak unrevealed points by using the wrong cache field. Although I guess I could've just created a new template tag to automatically pick the correct points to be displayed in the template. I'd be open to refactoring this like that.


Discussion:

Resolving patch:
diff --git a/exercise/cache/points.py b/exercise/cache/points.py
--- a/exercise/cache/points.py
+++ b/exercise/cache/points.py
@@ -1,12 +1,19 @@
 import datetime
 from copy import deepcopy
-from typing import Any, Dict, Iterable, List, Optional, Type, Tuple, Union
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    Iterable,
+    List,
+    Optional,
+    Type,
+    Tuple,
+    Union,
+)
 
 from django.contrib.auth.models import User
 from django.db.models.base import Model
-from django.db.models.aggregates import Max
-from django.db.models.expressions import Exists, OuterRef
-from django.db.models.query_utils import Q
 from django.db.models.signals import post_save, post_delete, m2m_changed
 from django.utils import timezone
 
@@ -22,28 +29,34 @@
 from .hierarchy import ContentMixin
 
 
-def has_more_points(submission: Submission, current_entry: Dict[str, Any]) -> bool:
-    if submission.grade == current_entry['points']:
-        return is_newer(submission, current_entry)
-    return submission.grade >= current_entry['points']
+def has_more_points(submission: Submission, current_best_submission: Optional[Submission]) -> bool:
+    if current_best_submission is None:
+        return True
+    if submission.grade == current_best_submission.grade:
+        return is_newer(submission, current_best_submission)
+    return submission.grade >= current_best_submission.grade
 
 
-def is_newer(submission: Submission, current_entry: Dict[str, Any]) -> bool:
+def is_newer(submission: Submission, current_best_submission: Optional[Submission]) -> bool:
     return (
-        current_entry['submission_date'] is None
-        or submission.submission_time >= current_entry['submission_date']
+        current_best_submission is None
+        or submission.submission_time >= current_best_submission.submission_time
     )
 
 
 class CachedPoints(ContentMixin, CachedAbstract):
+    """
+    Extends `CachedContent` to include data about a user's submissions and
+    points in the course's exercises.
+
+    Note that the `data` returned by this is dependent on the `is_staff`
+    parameter. When `is_staff` is `False`, reveal rules are respected and
+    exercise results are hidden when the reveal rule does not evaluate to true.
+    When `is_staff` is `True`, reveal rules are ignored and the results are
+    always revealed.
+    """
     KEY_PREFIX = 'points'
 
-    @classmethod
-    def invalidate(cls, course_instance: CourseInstance, user: User) -> None:
-        # Invalidate both the staff cache and the non-staff cache
-        super().invalidate(course_instance, user, modifiers=[str(False)])
-        super().invalidate(course_instance, user, modifiers=[str(True)])
-
     def __init__(
             self,
             course_instance: CourseInstance,
@@ -54,8 +67,8 @@ def __init__(
         self.content = content
         self.instance = course_instance
         self.user = user
-        self.is_staff = is_staff
-        super().__init__(course_instance, user, modifiers=[str(is_staff)])
+        super().__init__(course_instance, user)
+        self._extract_tuples(self.data, 0 if is_staff else 1)
 
     def _needs_generation(self, data: Dict[str, Any]) -> bool:
         return (
@@ -73,6 +86,63 @@ def _generate_data(
             user: User,
             data: Optional[Dict[str, Any]] = None,
             ) -> Dict[str, Any]:
+        # Perform all database queries before generating the cache.
+        if user.is_authenticated:
+            submissions = list(
+                user.userprofile.submissions.exclude_errors()
+                .filter(exercise__course_module__course_instance=instance)
+                .select_related()
+                .prefetch_related('exercise__parent', 'exercise__submission_feedback_reveal_rule', 'notifications')
+                .only('id', 'exercise', 'submission_time', 'status', 'grade', 'force_exercise_points')
+                .order_by('exercise', '-submission_time')
+            )
+            exercises = BaseExercise.objects.filter(course_module__course_instance=instance)
+            deadline_deviations = list(
+                DeadlineRuleDeviation.objects
+                .get_max_deviations(user.userprofile, exercises)
+            )
+            submission_deviations = list(
+                MaxSubmissionsRuleDeviation.objects
+                .get_max_deviations(user.userprofile, exercises)
+            )
+        else:
+            submissions = []
+            deadline_deviations = []
+            submission_deviations = []
+
+        # Generate the staff and student version of the cache, and merge them.
+        generate_args = (user.is_authenticated, submissions, deadline_deviations, submission_deviations)
+        staff_data = self._generate_data_internal(True, *generate_args)
+        student_data = self._generate_data_internal(False, *generate_args)
+        self._pack_tuples(staff_data, student_data) # Now staff_data is the final, combined data.
+
+        # Pick the lowest invalidate_time if it is duplicated.
+        invalidate_time = staff_data['invalidate_time']
+        if isinstance(invalidate_time, tuple):
+            if invalidate_time[0] is not None:
+                if invalidate_time[1] is not None:
+                    staff_data['invalidate_time'] = min(invalidate_time)
+                else:
+                    staff_data['invalidate_time'] = invalidate_time[0]
+            else:
+                staff_data['invalidate_time'] = invalidate_time[1]
+
+        staff_data['points_created'] = timezone.now()
+        return staff_data
+
+    def _generate_data_internal(
+            self,
+            is_staff: bool,
+            is_authenticated: bool,
+            submissions: Iterable[Submission],
+            deadline_deviations: Iterable[DeadlineRuleDeviation],
+            submission_deviations: Iterable[MaxSubmissionsRuleDeviation],
+            ) -> Dict[str, Any]:
+        """
+        Handles the generation of one version of the cache (staff or student).
+        All source data is prefetched by `_generate_data` and provided as
+        arguments to this method.
+        """
         data = deepcopy(self.content.data)
         module_index = data['module_index']
         exercise_index = data['exercise_index']
@@ -81,8 +151,6 @@ def _generate_data(
         total = data['total']
         data['invalidate_time'] = None
 
-        exercises = BaseExercise.objects.filter(course_module__course_instance=instance)
-
         # Augment submission parameters.
         def r_augment(children: List[Dict[str, Any]]) -> None:
             for entry in children:
@@ -91,7 +159,6 @@ def r_augment(children: List[Dict[str, Any]]) -> None:
                         'submission_count': 0,
                         'submissions': [],
                         'best_submission': None,
-                        'submission_date': None,
                         'points': 0,
                         'formatted_points': '0',
                         'passed': entry['points_to_pass'] == 0,
@@ -133,21 +200,103 @@ def r_augment(children: List[Dict[str, Any]]) -> None:
             'unconfirmed_points_by_difficulty': {},
         })
 
-        if user.is_authenticated:
-            # Augment submission data.
-            submissions = (
-                user.userprofile.submissions.exclude_errors()
-                .filter(exercise__course_module__course_instance=instance)
-                .prefetch_related('exercise', 'notifications')
-                .only('id', 'exercise', 'submission_time', 'status', 'grade', 'force_exercise_points')
-            )
-            for submission in submissions:
+        if is_authenticated:
+            # Augment deviation data.
+            for deviation in deadline_deviations:
+                try:
+                    tree = self._by_idx(modules, exercise_index[deviation.exercise.id])
+                except KeyError:
+                    self.dirty = True
+                    continue
+                entry = tree[-1]
+                entry['personal_deadline'] = (
+                    entry['closing_time'] + datetime.timedelta(minutes=deviation.extra_minutes)
+                )
+                entry['personal_deadline_has_penalty'] = not deviation.without_late_penalty
+
+            for deviation in submission_deviations:
                 try:
-                    tree = self._by_idx(modules, exercise_index[submission.exercise.id])
+                    tree = self._by_idx(modules, exercise_index[deviation.exercise.id])
                 except KeyError:
                     self.dirty = True
                     continue
                 entry = tree[-1]
+                entry['personal_max_submissions'] = (
+                    entry['max_submissions'] + deviation.extra_submissions
+                )
+
+            # Initialize variables for the submission loop.
+            exercise = None
+            entry = None
+            is_better_than = None
+            final_submission = None
+            last_submission = None
+
+            def check_reveal_rule() -> None:
+                """
+                Evaluate the reveal rule of the current exercise and ensure
+                that feedback is hidden appropriately.
+                """
+                rule = exercise.active_submission_feedback_reveal_rule
+                state = ExerciseRevealState(entry)
+                is_revealed = rule.is_revealed(state)
+                reveal_time = rule.get_reveal_time(state)
+
+                entry.update({
+                    'best_submission': entry['best_submission'] if is_revealed else last_submission.id,
+                    'points': entry['points'] if is_revealed else 0,
+                    'formatted_points': format_points(entry['points'], is_revealed, False),
+                    'passed': entry['passed'] if is_revealed else False,
+                    'feedback_revealed': is_revealed,
+                    'feedback_reveal_time': reveal_time,
+                })
+
+                for submission in entry['submissions']:
+                    submission.update({
+                        'points': submission['points'] if is_revealed else 0,
+                        'formatted_points': format_points(submission['points'], is_revealed, False),
+                        'passed': submission['passed'] if is_revealed else False,
+                        'feedback_revealed': is_revealed,
+                        'feedback_reveal_time': reveal_time,
+                    })
+
+                # If the reveal rule depends on time, update the cache's
+                # invalidation time.
+                if (
+                    reveal_time is not None
+                    and reveal_time > timezone.now()
+                    and (
+                        data['invalidate_time'] is None
+                        or reveal_time < data['invalidate_time']
+                    )
+                ):
+                    data['invalidate_time'] = reveal_time
+
+            # Augment submission data.
+            for submission in submissions:
+                # The submissions are ordered by exercise. Check here if the
+                # exercise has changed.
+                if exercise is None or submission.exercise.id != exercise.id:
+                    if exercise is not None and not is_staff:
+                        # Check the reveal rule of the last exercise now that
+                        # all of its submissions have been iterated.
+                        check_reveal_rule()
+                    final_submission = None
+                    last_submission = None
+                    # These variables stay constant throughout the exercise.
+                    exercise = submission.exercise
+                    try:
+                        tree = self._by_idx(modules, exercise_index[exercise.id])
+                    except KeyError:
+                        self.dirty = True
+                        continue
+                    entry = tree[-1]
+                    if exercise.grading_mode == BaseExercise.GRADING_MODE.BEST:
+                        is_better_than = has_more_points
+                    elif exercise.grading_mode == BaseExercise.GRADING_MODE.LAST:
+                        is_better_than = is_newer
+                    else:
+                        is_better_than = has_more_points
                 ready = submission.status == Submission.STATUS.READY
                 unofficial = submission.status == Submission.STATUS.UNOFFICIAL
                 if ready or submission.status in (Submission.STATUS.WAITING, Submission.STATUS.INITIALIZED):
@@ -167,28 +316,23 @@ def r_augment(children: List[Dict[str, Any]]) -> None:
                     'unofficial': unofficial,
                     'date': submission.submission_time,
                     'url': submission.get_url('submission-plain'),
+                    'feedback_revealed': True,
+                    'feedback_reveal_time': None,
                 })
-                # TODO: implement way to select algorithm for the best
-                if submission.exercise.grading_mode == BaseExercise.GRADING_MODE.BEST:
-                    is_better_than = has_more_points
-                elif submission.exercise.grading_mode == BaseExercise.GRADING_MODE.LAST:
-                    is_better_than = is_newer
-                else:
-                    is_better_than = has_more_points
-                # Update best submission if one of these is true
+                # Update best submission if exercise points are not forced, and
+                # one of these is true:
                 # 1) current submission in ready (thus is not unofficial) AND
                 #    a) current best is an unofficial OR
-                #    b) current submission has better grade
+                #    b) current submission is better depending on grading mode
                 # 2) All of:
                 #    - current submission is unofficial AND
                 #    - current best is unofficial
-                #    - current submission has better grade
+                #    - current submission is better depending on grading mode
                 if submission.force_exercise_points:
-                    # This submission is chosen as the best submission and no
+                    # This submission is chosen as the final submission and no
                     # further submissions are considered.
                     entry.update({
                         'best_submission': submission.id,
-                        'submission_date': submission.submission_time,
                         'points': submission.grade,
                         'formatted_points': format_points(submission.grade, True, False),
                         'passed': (ready and submission.grade >= entry['points_to_pass']),
@@ -196,99 +340,43 @@ def r_augment(children: List[Dict[str, Any]]) -> None:
                         'unofficial': False,
                         'forced_points': True,
                     })
+                    final_submission = submission
                 if not entry.get('forced_points', False):
                     if (
                         ready and (
                             entry['unofficial'] or
-                            is_better_than(submission, entry)
+                            is_better_than(submission, final_submission)
                         )
                     ) or (
                         unofficial and
                         not entry['graded'] and # NOTE: == entry['unofficial'], but before any submissions entry['unofficial'] is False
-                        is_better_than(submission, entry)
+                        is_better_than(submission, final_submission)
                     ):
                         entry.update({
                             'best_submission': submission.id,
-                            'submission_date': submission.submission_time,
                             'points': submission.grade,
                             'formatted_points': format_points(submission.grade, True, False),
                             'passed': (ready and submission.grade >= entry['points_to_pass']),
                             'graded': ready, # != unofficial
                             'unofficial': unofficial,
                         })
+                        final_submission = submission
+                # Update last_submission to be the last submission, or the last
+                # official submission if there are any official submissions.
+                # Note that the submissions are ordered by descendng time.
+                if last_submission is None or (
+                    last_submission.status == Submission.STATUS.UNOFFICIAL
+                    and not unofficial
+                ):
+                    last_submission = submission
                 if submission.notifications.count() > 0:
                     entry['notified'] = True
                     if submission.notifications.filter(seen=False).count() > 0:
                         entry['unseen'] = True
-
-            # Augment deviation data.
-            for deviation in DeadlineRuleDeviation.objects.get_max_deviations(user.userprofile, exercises):
-                try:
-                    tree = self._by_idx(modules, exercise_index[deviation.exercise.id])
-                except KeyError:
-                    self.dirty = True
-                    continue
-                entry = tree[-1]
-                entry['personal_deadline'] = (
-                    entry['closing_time'] + datetime.timedelta(minutes=deviation.extra_minutes)
-                )
-                entry['personal_deadline_has_penalty'] = not deviation.without_late_penalty
-
-            for deviation in MaxSubmissionsRuleDeviation.objects.get_max_deviations(user.userprofile, exercises):
-                try:
-                    tree = self._by_idx(modules, exercise_index[deviation.exercise.id])
-                except KeyError:
-                    self.dirty = True
-                    continue
-                entry = tree[-1]
-                entry['personal_max_submissions'] = (
-                    entry['max_submissions'] + deviation.extra_submissions
-                )
-
-            # Augment exercise reveal rules.
-            if not self.is_staff:
-                for exercise in (
-                    exercises
-                    .prefetch_related('submission_feedback_reveal_rule')
-                    .only('id', 'submission_feedback_reveal_rule')
-                ):
-                    try:
-                        tree = self._by_idx(modules, exercise_index[exercise.id])
-                    except KeyError:
-                        self.dirty = True
-                        continue
-                    entry = tree[-1]
-                    rule = exercise.active_submission_feedback_reveal_rule
-                    state = ExerciseRevealState(entry)
-                    is_revealed = rule.is_revealed(state)
-                    reveal_time = rule.get_reveal_time(state)
-
-                    entry.update({
-                        'points': entry['points'] if is_revealed else 0,
-                        'formatted_points': format_points(entry['points'], is_revealed, False),
-                        'passed': entry['passed'] if is_revealed else False,
-                        'feedback_revealed': is_revealed,
-                        'feedback_reveal_time': reveal_time,
-                    })
-
-                    for submission in entry['submissions']:
-                        submission.update({
-                            'points': submission['points'] if is_revealed else 0,
-                            'formatted_points': format_points(submission['points'], is_revealed, False),
-                            'passed': submission['passed'] if is_revealed else False,
-                            'feedback_revealed': is_revealed,
-                            'feedback_reveal_time': reveal_time,
-                        })
-
-                    if (
-                        reveal_time is not None
-                        and reveal_time > timezone.now()
-                        and (
-                            data['invalidate_time'] is None
-                            or reveal_time < data['invalidate_time']
-                        )
-                    ):
-                        data['invalidate_time'] = reveal_time
+            # All submissions of all exercises have been iterated. Check the
+            # reveal rule of the last exercise (if there was one).
+            if exercise is not None and not is_staff:
+                check_reveal_rule()
 
         # Confirm points.
         def r_check(parent: Dict[str, Any], children: List[Dict[str, Any]]) -> None:
@@ -382,7 +470,6 @@ def r_collect(
                 category['points'] >= category['points_to_pass']
             )
 
-        data['points_created'] = timezone.now()
         return data
 
     def created(self) -> Tuple[datetime.datetime, datetime.datetime]:
@@ -415,6 +502,44 @@ def submission_ids(
                 submissions.extend(s['id'] for s in entry.get('submissions', []))
         return submissions
 
+    def _pack_tuples(self, value1, value2, parent_container=None, parent_key=None):
+        """
+        Compare two data structures, and when conflicting values are found,
+        pack them into a tuple. `value1` is modified in this operation.
+
+        Example: when called with `value1={"key1": "a", "key2": "b"}` and
+        `value2={"key1": "a", "key2": "c"}`, `value1` will become
+        `{"key1": "a", "key2": ("b", "c")}`.
+        """
+        if isinstance(value1, dict):
+            for key, inner_value1 in value1.items():
+                inner_value2 = value2[key]
+                self._pack_tuples(inner_value1, inner_value2, value1, key)
+        elif isinstance(value1, list):
+            for index, inner_value1 in enumerate(value1):
+                inner_value2 = value2[index]
+                self._pack_tuples(inner_value1, inner_value2, value1, index)
+        else:
+            if value1 != value2:
+                parent_container[parent_key] = (value1, value2)
+
+    def _extract_tuples(self, value, tuple_index, parent_container=None, parent_key=None):
+        """
+        Find tuples within a data structure, and replace them with the value
+        at `tuple_index` in the tuple. `value` is modified in this operation.
+
+        Example: when called with `value={"key1": "a", "key2": ("b", "c")}` and
+        `tuple_index=0`, `value` will become `{"key1": "a", "key2": "b"}`.
+        """
+        if isinstance(value, dict):
+            for key, inner_value in value.items():
+                self._extract_tuples(inner_value, tuple_index, value, key)
+        elif isinstance(value, list):
+            for index, inner_value in enumerate(value):
+                self._extract_tuples(inner_value, tuple_index, value, index)
+        elif isinstance(value, tuple):
+            parent_container[parent_key] = value[tuple_index]
+
 
 def invalidate_content(sender: Type[Model], instance: Submission, **kwargs: Any) -> None:
     course = instance.exercise.course_instance
diff --git a/exercise/staff_views.py b/exercise/staff_views.py
--- a/exercise/staff_views.py
+++ b/exercise/staff_views.py
@@ -134,7 +134,7 @@ def get(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponse:
         )
 
         # Find the submitter's best submission using the cache.
-        cache = CachedPoints(self.instance, user, self.content)
+        cache = CachedPoints(self.instance, user, self.content, True)
         ids = cache.submission_ids(exercise_id=self.exercise.id, best=True)
         if not ids:
             raise Http404()
@@ -306,7 +306,7 @@ def get(self, request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponse:
             return self.redirect(self.exercise.get_submission_list_url())
 
         # Find the submitter's best submission using the cache.
-        cache = CachedPoints(self.instance, submitter.user, self.content)
+        cache = CachedPoints(self.instance, submitter.user, self.content, True)
         ids = cache.submission_ids(exercise_id=self.exercise.id, best=True)
         if not ids:
             raise Http404()
diff --git a/exercise/templatetags/exercise.py b/exercise/templatetags/exercise.py
--- a/exercise/templatetags/exercise.py
+++ b/exercise/templatetags/exercise.py
@@ -145,12 +145,17 @@ def _points_data(
         user: User,
         classes: Optional[str] = None,
         is_staff: bool = False,
+        known_revealed: Optional[bool] = None,
         ) -> Dict[str, Any]:
     reveal_time = None
-    is_revealed = is_staff
+    is_revealed = None
+    if known_revealed is not None:
+        is_revealed = known_revealed
+    elif is_staff:
+        is_revealed = True
     if isinstance(obj, UserExerciseSummary):
         exercise = obj.exercise
-        if not is_staff:
+        if is_revealed is None:
             is_revealed, reveal_time = _reveal_rule(exercise, user)
         data = {
             'points': obj.get_points() if is_revealed else 0,
@@ -170,7 +175,7 @@ def _points_data(
         }
     elif isinstance(obj, Submission):
         exercise = obj.exercise
-        if not is_staff:
+        if is_revealed is None:
             is_revealed, reveal_time = _reveal_rule(exercise, user)
         data = {
             'points': obj.grade if is_revealed else 0,
@@ -246,8 +251,9 @@ def _points_data(
 def points_progress(
         context: Context,
         obj: Union[UserExerciseSummary, Submission, Dict[str, Any]],
+        is_revealed: Optional[bool] = None,
         ) -> Dict[str, Any]:
-    return _points_data(obj, context['request'].user, None, context['is_course_staff'])
+    return _points_data(obj, context['request'].user, None, context['is_course_staff'], is_revealed)
 
 
 @register.inclusion_tag("exercise/_points_badge.html", takes_context=True)
@@ -255,8 +261,9 @@ def points_badge(
         context: Context,
         obj: Union[UserExerciseSummary, Submission, Dict[str, Any]],
         classes: Optional[str] = None,
+        is_revealed: Optional[bool] = None,
         ) -> Dict[str, Any]:
-    return _points_data(obj, context['request'].user, classes, context['is_course_staff'])
+    return _points_data(obj, context['request'].user, classes, context['is_course_staff'], is_revealed)
 
 
 @register.simple_tag


Test patch: