Repository: learningequality/kolibri

Issue:
Question/quiz validation error
_Originally posted by @radinamatic in https://github.com/learningequality/kolibri/issues/12111#issuecomment-2101535743_
            
> That quiz with 107 questions kept throwing error when the learner inputs the answer on the last questions and clicks the ``Next`` button (or presses ``Enter`` when navigating by keyboard).

   <img width="842" alt="last-question" src="https://github.com/learningequality/kolibri/assets/1457929/bce248b5-4184-42b1-a4be-831961316a2f">

This seems like it may be either a validation problem or an error handling problem at some step along the quiz or assessment process. This needs some further exploration to determine if the quiz had an invalid question (and how to prevent such edge cases) or, if there is something wrong with advancing through the quiz on the learner side. Or possibly something else. 


Discussion:
That quiz with 107 questions can be considered "corrupted data". When saving the quiz during creation, the total number of questions for each section was added together.

The first section was the default value, a number `10` -- but because you'd changed the number of questions by typing in the number 7, the value on the second section was the text `"7"`. 

The base Exam data structure has a "num_questions" property so the quiz creator adds that same value for each section together to get the Exam's total sum of questions from all sections.

So when that quiz was saved, I added the number `10` to the text `"7"` which resulted in Javascript saving the text "107" in that exam's `num_questions` field. 

I've fixed this problem by forcing the values to be numbers when adding them together.

---

That all said, this is a problem that _could_ be avoided by summing the questions when the user takes the quiz rather than relying on the vestigal total quiz questions count. That means -- when the Learner begins a quiz, code on their computer will count the questions. Basically, there isn't really any _need_ for the `num_questions` value to be saved on the base exam.

Or, perhaps, the value of `num_questions` could be dynamically set in the relevant serializer -- basically the API can choose to ignore the saved `num_questions` property and count the sections' questions themselves. Or perhaps a method on `Exam` would be helpful.

---

In any case -- there are definitely some improvements that we can apply here, but I think this is relatively low priority overall since it is only a bug in the case of corrupted data.
I think dynamically summing the questions from all sections seems like the most sensible resolution here.
This is a problem when we have a section but we dont have questions resources on that section.

https://github.com/learningequality/kolibri/assets/51239030/9ab3647c-a3b1-41f7-8c58-26a827fd7a8a

It keeps adding the 10 questions in the section even if there are not resources in it, and thats why learners cant complete the quiz. 

What should we do with the sections that do not have resources? I would think we should filter them out and not show them to students.
> I think dynamically summing the questions from all sections seems like the most sensible resolution here.

Should we do it when saving the quiz? Or when reading it? I think it would be better to do it on save, and have `exam.question_count` have the actual value of the actual questions (with exercise resources) that exist (i.e. accumulate section.questions.length instead of section.question_count)
We currently are doing this when we save the quiz.

The idea is that in Learn, when the user goes to take the exam, that is when we just count all of the questions in `Exam.question_sources` (aka the quiz's sections) rather than relying on the `Exam.question_count` property to be accurate.

I think this is fine - but I wonder if perhaps it'd be better to implement an on-save validator on the Exam class that basically serves as the "One validation to rule them all" which can return an error when the Exam.question_count value doesn't match the number of questions in the sections total.

Overall I think it'd be better to utilize the Exam model and handle this logic in one place. It feels to me that if we just say "we'll code it client-side to derive the value of `Exam.question_count`" is kind of resigning the `Exam.question_count` property to being functionally pointless... like if we can't trust that number then why is it even on the model anymore - why not just drop the column?

Resolving patch:
diff --git a/kolibri/core/exams/serializers.py b/kolibri/core/exams/serializers.py
--- a/kolibri/core/exams/serializers.py
+++ b/kolibri/core/exams/serializers.py
@@ -60,7 +60,7 @@ class ExamSerializer(ModelSerializer):
     creator = PrimaryKeyRelatedField(
         read_only=False, queryset=FacilityUser.objects.all()
     )
-    question_count = IntegerField()
+    question_count = IntegerField(allow_null=True)
     date_archived = DateTimeTzField(allow_null=True)
     date_activated = DateTimeTzField(allow_null=True)
 
@@ -83,7 +83,7 @@ class Meta:
             "learners_see_fixed_order",
             "learner_ids",
         )
-        read_only_fields = ("data_model_version",)
+        read_only_fields = ("data_model_version", "question_count")
 
     def validate(self, attrs):
         title = attrs.get("title")
@@ -137,6 +137,12 @@ def to_internal_value(self, data):
             else:
                 # Otherwise we are just updating the exam, so allow a partial update
                 self.partial = True
+
+        question_sources = data.get("question_sources", [])
+        data["question_count"] = sum(
+            len(source.get("questions", [])) for source in question_sources
+        )
+
         return super(ExamSerializer, self).to_internal_value(data)
 
     def create(self, validated_data):


Test patch:
diff --git a/kolibri/core/exams/test/test_exam_api.py b/kolibri/core/exams/test/test_exam_api.py
--- a/kolibri/core/exams/test/test_exam_api.py
+++ b/kolibri/core/exams/test/test_exam_api.py
@@ -42,10 +42,8 @@ def setUpTestData(cls):
 
     def make_basic_exam(self):
         sections = self.make_basic_sections(1)
-        total_questions = sum(section["question_count"] for section in sections)
         return {
             "title": "Exam",
-            "question_count": total_questions,
             "active": True,
             "collection": self.classroom.id,
             "learners_see_fixed_order": False,
@@ -551,3 +549,15 @@ def test_exam_model_get_questions_v2_v1(self):
 
         self.exam.save()
         self.assertEqual(len(self.exam.get_questions()), 1)
+
+    def test_exam_question_count_calculation(self):
+        self.login_as_admin()
+        exam = self.make_basic_exam()
+        question_count = sum(
+            len(source["questions"]) for source in exam["question_sources"]
+        )
+        response = self.post_new_exam(exam)
+        exam_id = response.data["id"]
+        self.assertEqual(response.status_code, 201)
+        exam_model_instance = models.Exam.objects.get(id=exam_id)
+        self.assertEqual(exam_model_instance.question_count, question_count)