Repository: apluslms/a-plus

Issue:
Inspect submitter button leads to 404 "not found" if the student's all submissions have status "error" or "rejected"
If all submissions from a student are in the status "error" or "rejected", then the inspect button in the "list submitters" page leads to 404 not found error.

To access the submitter list, press the "Group by submitter" button in the assignment's submission list.

Reported by @ahrenberg 

`InspectSubmitterView` uses `CachedPoints.submission_ids()` to find the best submission from the student. The inspect submitter button is supposed to open the inspect submission page of the best submission. However, the submission list stored in `CachedPoints` excludes submissions in the "error" or "rejected" statuses.

If `CachedPoints` were changed to include all submissions with any status, it would affect some other parts of the system. We need to consider carefully if we can change `CachedPoints` or if `InspectSubmitterView` should separately check for the existence of error/rejected submissions. As far as I remember, at least the user interface uses `CachedPoints`, but I don't remember this now. Check how the `CachedPoints` changes could affect the system overall.

* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/staff_views.py#L143
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104//exercise/cache/points.py#L480-L508
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/cache/points.py#L91
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/submission_models.py#L265-L269

Inspect submitter button leads to 404 "not found" if the student's all submissions have status "error" or "rejected"
If all submissions from a student are in the status "error" or "rejected", then the inspect button in the "list submitters" page leads to 404 not found error.

To access the submitter list, press the "Group by submitter" button in the assignment's submission list.

Reported by @ahrenberg 

`InspectSubmitterView` uses `CachedPoints.submission_ids()` to find the best submission from the student. The inspect submitter button is supposed to open the inspect submission page of the best submission. However, the submission list stored in `CachedPoints` excludes submissions in the "error" or "rejected" statuses.

If `CachedPoints` were changed to include all submissions with any status, it would affect some other parts of the system. We need to consider carefully if we can change `CachedPoints` or if `InspectSubmitterView` should separately check for the existence of error/rejected submissions. As far as I remember, at least the user interface uses `CachedPoints`, but I don't remember this now. Check how the `CachedPoints` changes could affect the system overall.

* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/staff_views.py#L143
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104//exercise/cache/points.py#L480-L508
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/cache/points.py#L91
* https://github.com/apluslms/a-plus/blob/f14b0de94b53e79d7ab463af58dd2095ebe63104/exercise/submission_models.py#L265-L269



Discussion:
Adding rejected and error submissions to the points cache has the following effects:
1. The submissions show up in some APIs. Right now they are omitted.
2. The submissions show up in the course instance view in the submission dropdown next to each exercise. Strangely enough the submissions in the submission dropdown in the exercise pages comes from a different place which does include ERROR and REJECTED submissions already.

I think we should include ERROR and REJECTED in the cache. This will make the submission dropdowns match. It also makes sense to me that the invalid submissions also show up in the API. We just need to make sure that the grade is 0 for all these submissions.

Of note is that the "best submission" is calculated in (at least) three separate places. 
1. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/exercise_summary.py#L20-L78 this is the one that provides the submissions to the exercise pages
2. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/cache/points.py#L331-L364 this is the one that provides the submissions to the course instance page and some API views
3. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/api/csv/submission_sheet.py#L10-L42

I think these should be consolidated. In fact, I think `UserExerciseSummary` should be consolidated into `CachedPoints`. As far as I can tell `CachedPoints` does everything that `UserExerciseSummary` is used for but just the data inside a dict instead of a class.
It is important to emphasize that there are parts of the system that do not include ERROR or REJECTED submissions to the submission count. Thus, those submissions do not affect the "max submissions" limit (number of submission attempts) when submitting to assignments. How would the `CachedPoints` changes affect this?

> I think these should be consolidated. In fact, I think `UserExerciseSummary` should be consolidated into `CachedPoints`. As far as I can tell `CachedPoints` does everything that `UserExerciseSummary` is used for but just the data inside a dict instead of a class.

About the points calculation, there have been different approaches over the years and some of the code should be refactored/consolidated indeed. There is an issue for that with more details: #888.
> It is important to emphasize that there are parts of the system that do not include ERROR or REJECTED submissions to the submission count. Thus, those submissions do not affect the "max submissions" limit (number of submission attempts) when submitting to assignments. How would the `CachedPoints` changes affect this?

`CachedPoints` already does a special check so that it only counts READY, WAITING, and INITIALIZED in the submission count. Adding ERROR and REJECTED thus wouldn't affect it.


Adding rejected and error submissions to the points cache has the following effects:
1. The submissions show up in some APIs. Right now they are omitted.
2. The submissions show up in the course instance view in the submission dropdown next to each exercise. Strangely enough the submissions in the submission dropdown in the exercise pages comes from a different place which does include ERROR and REJECTED submissions already.

I think we should include ERROR and REJECTED in the cache. This will make the submission dropdowns match. It also makes sense to me that the invalid submissions also show up in the API. We just need to make sure that the grade is 0 for all these submissions.

Of note is that the "best submission" is calculated in (at least) three separate places. 
1. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/exercise_summary.py#L20-L78 this is the one that provides the submissions to the exercise pages
2. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/cache/points.py#L331-L364 this is the one that provides the submissions to the course instance page and some API views
3. https://github.com/apluslms/a-plus/blob/7f21d5ad5b20d7dc4d02c2bb7601720527d07c77/exercise/api/csv/submission_sheet.py#L10-L42

I think these should be consolidated. In fact, I think `UserExerciseSummary` should be consolidated into `CachedPoints`. As far as I can tell `CachedPoints` does everything that `UserExerciseSummary` is used for but just the data inside a dict instead of a class.
It is important to emphasize that there are parts of the system that do not include ERROR or REJECTED submissions to the submission count. Thus, those submissions do not affect the "max submissions" limit (number of submission attempts) when submitting to assignments. How would the `CachedPoints` changes affect this?

> I think these should be consolidated. In fact, I think `UserExerciseSummary` should be consolidated into `CachedPoints`. As far as I can tell `CachedPoints` does everything that `UserExerciseSummary` is used for but just the data inside a dict instead of a class.

About the points calculation, there have been different approaches over the years and some of the code should be refactored/consolidated indeed. There is an issue for that with more details: #888.
> It is important to emphasize that there are parts of the system that do not include ERROR or REJECTED submissions to the submission count. Thus, those submissions do not affect the "max submissions" limit (number of submission attempts) when submitting to assignments. How would the `CachedPoints` changes affect this?

`CachedPoints` already does a special check so that it only counts READY, WAITING, and INITIALIZED in the submission count. Adding ERROR and REJECTED thus wouldn't affect it.



Resolving patch:
diff --git a/exercise/cache/points.py b/exercise/cache/points.py
--- a/exercise/cache/points.py
+++ b/exercise/cache/points.py
@@ -88,7 +88,7 @@ def _generate_data( # pylint: disable=arguments-differ
         # Perform all database queries before generating the cache.
         if user.is_authenticated:
             submissions = list(
-                user.userprofile.submissions.exclude_errors()
+                user.userprofile.submissions
                 .filter(exercise__course_module__course_instance=instance)
                 .select_related()
                 .prefetch_related('exercise__parent', 'exercise__submission_feedback_reveal_rule', 'notifications')
@@ -488,6 +488,23 @@ def submission_ids( # pylint: disable=too-many-arguments
             fallback_to_last: bool = False,
             raise_404=True,
             ) -> List[int]:
+        """Return submission ids for the exercises matching the search terms
+
+        :param best: whether to get the best submission for each exercise or all submissions
+        :param fallback_to_last: whether to fallback to the latest submission when best == True.
+        Prioritizes INITIALIZED and WAITING submissions over REJECTED and ERROR.
+        """
+        def latest_submission(submissions: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
+            # Find latest non REJECTED or ERROR submission
+            for entry in submissions:
+                if entry["submission_status"] not in (Submission.STATUS.REJECTED, Submission.STATUS.ERROR):
+                    return entry
+
+            if submissions:
+                return submissions[0] # Last submission is first in the cache
+
+            return None
+
         exercises = self.search_exercises(
             number=number,
             category_id=category_id,
@@ -503,9 +520,9 @@ def submission_ids( # pylint: disable=too-many-arguments
                 if sid is not None:
                     submissions.append(sid)
                 elif fallback_to_last:
-                    entry_submissions = entry.get('submissions', [])
-                    if entry_submissions:
-                        submissions.append(entry_submissions[0]['id']) # Last submission is first in the cache
+                    latest = latest_submission(entry.get('submissions', []))
+                    if latest is not None:
+                        submissions.append(latest["id"])
         else:
             for entry in exercises:
                 submissions.extend(s['id'] for s in entry.get('submissions', []))


Test patch: