Repository: learningequality/kolibri

Issue:
users should not be able to get 1000% on an exam, unfortunately

### Observed behavior

reported by @jtamiace re: @radinamatic's apparent good luck:

![image](https://user-images.githubusercontent.com/2367265/52095361-812c2280-2577-11e9-86d6-8f6cff4e0f5b.png)

### Expected behavior

exams are scored between 0 and 100

### User-facing consequences

????

### Errors and logs

unknown

### Steps to reproduce

see http://kolibribeta.learningequality.org/coach/#/fa4cbfeda32c0c0fbf1832fc1ddd10c3/reports/learners

### Context

k 0.12.0 alpha 7



Discussion:
And average quiz score of 550% here
http://kolibribeta.learningequality.org/coach/#/fa4cbfeda32c0c0fbf1832fc1ddd10c3/reports/quizzes

<img width="962" alt="screen shot 2019-01-31 at 4 51 35 pm" src="https://user-images.githubusercontent.com/6668144/52095607-8178ed80-2578-11e9-8af0-9d61193acd4a.png">

I can take a look at this -- at least as far as diagnosing what's going on behind the scenes in the DB/API, to start with.
It seems the backend is returning "num_correct", and the frontend is interpreting it as a percentage correct.

The second entry here is for Radina's account:

![image](https://user-images.githubusercontent.com/618416/52152155-37504480-262a-11e9-837d-a0e6ee13f970.png)

A parallel anomaly in the data is that there are actually 11 correct attemptlogs for her for that exam, but two have the same `item`:

```
In [27]: el.attemptlogs.filter(correct=True).values_list('item').distinct().count()
Out[27]: 10

In [28]: el.attemptlogs.filter(correct=True).values_list('item').count()
Out[28]: 11
```
```
In [47]: [(a.item, a.completion_timestamp.strftime("%H:%M:%S")) for a in el.attemptlogs.filter(correct=True).order_by('completion_timestamp')]
Out[47]: 
[(u'a5f508eb2ba05d429812dc43b577ef03', '13:43:01'),
 (u'8358fbbd0a285e9d99da558094eabd4c', '13:43:13'),
 (u'84f8e275aec652bf8a418d133e422f7f', '13:43:25'),
 (u'fc60ecb9f83f505fa31e734e517e6523', '13:43:58'),
 (u'4dd2526065ee572998a06b1fdae9cb4b', '13:44:04'),
 (u'50065107c7a0581e9bb30911ba6cc379', '13:44:10'),
 (u'642e2e18170055d2b44b0e408038176c', '13:44:29'),
 (u'6ef7996754de54ad92b660d06436976c', '13:44:41'),
 (u'9e54ffbf7138534982c617e52d5907bb', '13:44:46'),
 (u'cce3659096295edca9aaeff623fa4a50', '13:44:51'),
 (u'cce3659096295edca9aaeff623fa4a50', '13:44:56')]
```
(so the two attemptlogs for the same question happened at the end, 5 seconds apart)
From Slack discussion, we determined that the client-side needs to use `examMap[status.exam_id].question_count` instead of `examMap[status.exam_id].question_sources.length` at https://github.com/learningequality/kolibri/blob/develop/kolibri/plugins/coach/assets/src/modules/classSummary/index.js#L198
Regarding the second issue (duplicate logs), there seem to be 3 cases of it in the database:
```
In [56]: ExamAttemptLog.objects.values_list("examlog_id", "item").count()
Out[56]: 249

In [57]: ExamAttemptLog.objects.values_list("examlog_id", "item").distinct().count()
Out[57]: 246
```
Is it possible that item ids could be duplicated across exercises? Could you perhaps double check this with `content_id` also?
Seems unlikely, but would like to be sure!
It's all from Radina's interactions with that one exam:

```
[(u'a5f508eb2ba05d429812dc43b577ef03', '13:43:01'),
 (u'8358fbbd0a285e9d99da558094eabd4c', '13:43:13'),
 (u'8358fbbd0a285e9d99da558094eabd4c', '13:43:13'),
 (u'84f8e275aec652bf8a418d133e422f7f', '13:43:25'),
 (u'84f8e275aec652bf8a418d133e422f7f', '13:43:25'),
 (u'06bb1d1bdd6250c4ad4912c558801139', '13:43:46'),
 (u'fc60ecb9f83f505fa31e734e517e6523', '13:43:58'),
 (u'4dd2526065ee572998a06b1fdae9cb4b', '13:44:04'),
 (u'50065107c7a0581e9bb30911ba6cc379', '13:44:10'),
 (u'beb5eae9491c564fb6bc5b9c1421d085', '13:44:19'),
 (u'642e2e18170055d2b44b0e408038176c', '13:44:29'),
 (u'6ef7996754de54ad92b660d06436976c', '13:44:41'),
 (u'9e54ffbf7138534982c617e52d5907bb', '13:44:46'),
 (u'cce3659096295edca9aaeff623fa4a50', '13:44:51'),
 (u'cce3659096295edca9aaeff623fa4a50', '13:44:56')]
```
Ah, yeah, will check
Another issue, unless I'm missing something, is that we seem to be anchoring to the PK (`node_id`) of an exercise, rather than the `content_id`. I remember discussing how this could be brittle, especially with cheffed channels.

```
In [67]: e.question_sources
Out[67]: 
[{u'exercise_id': u'b2097b29acbe4349924ff4510c5f8777',
  u'number_of_questions': 12,
  u'title': u'Count with small numbers'}]

In [71]: ContentNode.objects.get(content_id="b2097b29acbe4349924ff4510c5f8777")
DoesNotExist: ContentNode matching query does not exist.

In [72]: ContentNode.objects.get(id="b2097b29acbe4349924ff4510c5f8777")
Out[72]: <ContentNode: Count with small numbers>
```
To @rtibbles' question above: they're all from the same exercise. The exercise has 20 assessments, all with unique ids.
Yeah, thats a misleading field name on the ExamAttemptLog. In the case of exams in particular, where the assessment content is abstracted from the metadata `content_id` _could_ work, but I am hesistant to suggest we make that shift, when, e.g. the same exercise in different languages in KA is currently published with the same content_id - I can imagine that being very confusing for a user taking an exam.
Since there's only one case of it, is it possible @radinamatic was doing something like interacting with the same exam in multiple tabs?
> interacting with the same exam in multiple tabs

Can't recall offhand, but I am a tab junkie, so cannot entirely exclude the possibility 😊 
I gather that's something I should steer away in future testing?
> I gather that's something I should steer away in future testing?

never steer away from anything in testing :)
Gotta love a job where it actually pays out to be "naughty" and step out of line now and then... 😎 
![image](https://user-images.githubusercontent.com/618416/52156665-712b4600-263e-11e9-8be3-fccf9a4f02d5.png)

for posterity:

We identified that the issue was because older exams are represented differently, and the logic was assuming the newer representation when calculating progress.

see https://github.com/learningequality/kolibri/pull/4667

Resolving patch:
diff --git a/kolibri/plugins/coach/class_summary_api.py b/kolibri/plugins/coach/class_summary_api.py
--- a/kolibri/plugins/coach/class_summary_api.py
+++ b/kolibri/plugins/coach/class_summary_api.py
@@ -188,7 +188,7 @@ class ExamSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = Exam
-        fields = ("id", "title", "active", "question_sources", "groups", "data_model_version")
+        fields = ("id", "title", "active", "question_sources", "groups", "data_model_version", "question_count")
 
 
 class ContentSerializer(serializers.ModelSerializer):


Test patch:
diff --git a/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleServerResponse.js b/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleServerResponse.js
--- a/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleServerResponse.js
+++ b/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleServerResponse.js
@@ -77,6 +77,8 @@ export default {
         { exercise_id: '2a722a9e57575148bc55deed7550ed62', question_id: '3' },
       ],
       groups: ['c4625c3fef6b7d918e9417d92e482e6f'],
+      data_version_model: 1,
+      question_count: 3,
     },
     {
       id: 'd7033a1cb888493763dc9b5f3ab2505b',
@@ -87,6 +89,8 @@ export default {
         { exercise_id: 'eadec7f803994b6eb8f401237ec0f777', question_id: 'B' },
       ],
       groups: ['8d2e8c66c05004657d676155dd0b305d'],
+      data_version_model: 1,
+      question_count: 2,
     },
     {
       id: '4018bcea43cee3d05811b641fca0b152',
@@ -98,6 +102,8 @@ export default {
         { exercise_id: '3a655a4b8adb5114a571dfd0c75cbc19', question_id: '12' },
       ],
       groups: ['7c20f664b6a5c43d64b0cdd3161be513'],
+      data_version_model: 1,
+      question_count: 3,
     },
     {
       id: '97316f077d470b45e912096edb534076',
@@ -109,6 +115,8 @@ export default {
         { exercise_id: '9baf781e43b0514085cc205176b0ee71', question_id: 'z' },
       ],
       groups: [],
+      data_version_model: 1,
+      question_count: 3,
     },
   ],
   exam_learner_status: [
diff --git a/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleState.js b/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleState.js
--- a/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleState.js
+++ b/kolibri/plugins/coach/assets/src/modules/classSummary/__test__/sampleState.js
@@ -153,6 +153,8 @@ export default {
         { exercise_id: '2a722a9e57575148bc55deed7550ed62', question_id: '3' },
       ],
       groups: ['c4625c3fef6b7d918e9417d92e482e6f'],
+      data_version_model: 1,
+      question_count: 3,
     },
     d7033a1cb888493763dc9b5f3ab2505b: {
       id: 'd7033a1cb888493763dc9b5f3ab2505b',
@@ -163,6 +165,8 @@ export default {
         { exercise_id: 'eadec7f803994b6eb8f401237ec0f777', question_id: 'B' },
       ],
       groups: ['8d2e8c66c05004657d676155dd0b305d'],
+      data_version_model: 1,
+      question_count: 2,
     },
     '4018bcea43cee3d05811b641fca0b152': {
       id: '4018bcea43cee3d05811b641fca0b152',
@@ -174,6 +178,8 @@ export default {
         { exercise_id: '3a655a4b8adb5114a571dfd0c75cbc19', question_id: '12' },
       ],
       groups: ['7c20f664b6a5c43d64b0cdd3161be513'],
+      data_version_model: 1,
+      question_count: 3,
     },
     '97316f077d470b45e912096edb534076': {
       id: '97316f077d470b45e912096edb534076',
@@ -185,6 +191,8 @@ export default {
         { exercise_id: '9baf781e43b0514085cc205176b0ee71', question_id: 'z' },
       ],
       groups: [],
+      data_version_model: 1,
+      question_count: 3,
     },
   },
   examLearnerStatusMap: {