Repository: oppia/oppia

Issue:
Randomize the pretest questions.
Currently, the first 3 pretest questions sorted by their keys will be returned for a fixed set of prerequisite skill ids for an exploration. This needs to be changed to make it a random set of 3 questions.
Randomize the pretest questions.
Currently, the first 3 pretest questions sorted by their keys will be returned for a fixed set of prerequisite skill ids for an exploration. This needs to be changed to make it a random set of 3 questions.


Discussion:
Hi @aks681, can I work on this issue?
Yup! Feel free to start working on this and ping me if you have any issues!
@aks681 it seems the current logic is in this line https://github.com/oppia/oppia/blob/8aa2e3b96b9621f9c80f546b863a7e0dadcff6c4/core/storage/question/gae_models.py#L223

If we don't want to sort all the relevant questions by the key, how do we want it to be? Do we want to shuffle them before calling fetch_page(question_count), or we just rely on the order returned by the query?
I think we might need to solve a more general problem, actually. Let's say we want to retrieve a random subset of elements of a given query result from the datastore. How do we structure our datastore so that we can do that quickly?

There are a number of suggestions on StackOverflow that may be helpful. One is to generate a random key and find the first element >= that key (which can be done using a single inequality query). Another is  to store a randomly-generated field together with the question and do the same thing with that field. There may be others but this is something that might need a little research/testing.
@seanlip I was thinking about this and we only need the random generation of questions when a learner is requesting it, right? When the questions are displayed in the topic editor (in questions tab), this should not be done, right? So, maybe another argument is needed for this function called `randomize_questions` or similar which would enable the randomization.
Yes, that is correct. There are two use cases: (a) retrieve all questions in a paginated view (perhaps you could clarify if this is for topic/skill?), and (b) randomly select a subset of questions given a set of skills. I think these are different code paths and don't have much in common.
I think I need to un-assign myself from this story as I'll be working on other issues. This story is now open for anyone else to work on.
Hi @aks681, can I work on this issue?
Yup! Feel free to start working on this and ping me if you have any issues!
@aks681 it seems the current logic is in this line https://github.com/oppia/oppia/blob/8aa2e3b96b9621f9c80f546b863a7e0dadcff6c4/core/storage/question/gae_models.py#L223

If we don't want to sort all the relevant questions by the key, how do we want it to be? Do we want to shuffle them before calling fetch_page(question_count), or we just rely on the order returned by the query?
I think we might need to solve a more general problem, actually. Let's say we want to retrieve a random subset of elements of a given query result from the datastore. How do we structure our datastore so that we can do that quickly?

There are a number of suggestions on StackOverflow that may be helpful. One is to generate a random key and find the first element >= that key (which can be done using a single inequality query). Another is  to store a randomly-generated field together with the question and do the same thing with that field. There may be others but this is something that might need a little research/testing.
@seanlip I was thinking about this and we only need the random generation of questions when a learner is requesting it, right? When the questions are displayed in the topic editor (in questions tab), this should not be done, right? So, maybe another argument is needed for this function called `randomize_questions` or similar which would enable the randomization.
Yes, that is correct. There are two use cases: (a) retrieve all questions in a paginated view (perhaps you could clarify if this is for topic/skill?), and (b) randomly select a subset of questions given a set of skills. I think these are different code paths and don't have much in common.
I think I need to un-assign myself from this story as I'll be working on other issues. This story is now open for anyone else to work on.

Resolving patch:
diff --git a/core/controllers/reader.py b/core/controllers/reader.py
--- a/core/controllers/reader.py
+++ b/core/controllers/reader.py
@@ -34,7 +34,6 @@
 from core.domain import interaction_registry
 from core.domain import learner_progress_services
 from core.domain import moderator_services
-from core.domain import question_fetchers
 from core.domain import question_services
 from core.domain import rating_services
 from core.domain import recommendations_services
@@ -216,24 +215,22 @@ class PretestHandler(base.BaseHandler):
     @acl_decorators.can_play_exploration
     def get(self, exploration_id):
         """Handles GET request."""
-        start_cursor = self.request.get('cursor')
         story_id = self.request.get('story_id')
         story = story_fetchers.get_story_by_id(story_id, strict=False)
         if story is None:
             raise self.InvalidInputException
         if not story.has_exploration(exploration_id):
             raise self.InvalidInputException
-        pretest_questions, _, next_start_cursor = (
-            question_fetchers.get_questions_and_skill_descriptions_by_skill_ids(
+        pretest_questions = (
+            question_services.get_questions_by_skill_ids(
                 feconf.NUM_PRETEST_QUESTIONS,
                 story.get_prerequisite_skill_ids_for_exp_id(exploration_id),
-                start_cursor)
+                True)
         )
         question_dicts = [question.to_dict() for question in pretest_questions]
 
         self.values.update({
             'pretest_question_dicts': question_dicts,
-            'next_start_cursor': next_start_cursor
         })
         self.render_json(self.values)
 
diff --git a/core/domain/question_services.py b/core/domain/question_services.py
--- a/core/domain/question_services.py
+++ b/core/domain/question_services.py
@@ -202,15 +202,15 @@ def delete_question_skill_link(user_id, question_id, skill_id):
 
 
 def get_questions_by_skill_ids(
-        total_question_count, skill_ids, fetch_by_difficulty):
+        total_question_count, skill_ids, require_medium_difficulty):
     """Returns constant number of questions linked to each given skill id.
 
     Args:
         total_question_count: int. The total number of questions to return.
         skill_ids: list(str). The IDs of the skills to which the questions
             should be linked.
-        fetch_by_difficulty: bool. Indicates whether the returned questions
-            should be fetched by skill difficulty.
+        require_medium_difficulty: bool. Indicates whether the returned
+            questions should be of medium difficulty.
 
     Returns:
         list(Question). The list containing an expected number of
@@ -220,9 +220,9 @@ def get_questions_by_skill_ids(
             divisible. If not enough questions for one skill, simply return
             all questions linked to it. The order of questions will follow the
             order of given skill ids, and the order of questions for the same
-            skill is random when fetch_by_difficulty is false, otherwise the
-            order is sorted by absolute value of the difference between skill
-            difficulty and the medium difficulty.
+            skill is random when require_medium_difficulty is false, otherwise
+            the order is sorted by absolute value of the difference between
+            skill difficulty and the medium difficulty.
     """
 
     if total_question_count > feconf.MAX_QUESTIONS_FETCHABLE_AT_ONE_TIME:
@@ -230,7 +230,7 @@ def get_questions_by_skill_ids(
             'Question count is too high, please limit the question count to '
             '%d.' % feconf.MAX_QUESTIONS_FETCHABLE_AT_ONE_TIME)
 
-    if fetch_by_difficulty:
+    if require_medium_difficulty:
         question_skill_link_models = (
             question_models.QuestionSkillLinkModel.get_question_skill_links_based_on_difficulty_equidistributed_by_skill( #pylint: disable=line-too-long
                 total_question_count, skill_ids,
diff --git a/core/storage/question/gae_models.py b/core/storage/question/gae_models.py
--- a/core/storage/question/gae_models.py
+++ b/core/storage/question/gae_models.py
@@ -18,6 +18,7 @@
 from __future__ import unicode_literals  # pylint: disable=import-only-modules
 
 import math
+import random
 
 from constants import constants
 from core.platform import models
@@ -326,7 +327,7 @@ def get_question_skill_links_by_skill_ids(
     @classmethod
     def get_question_skill_links_based_on_difficulty_equidistributed_by_skill(
             cls, total_question_count, skill_ids, difficulty_requested):
-        """Fetches the list of constant number of QuestionSkillLinkModels
+        """Fetches the list of constant number of random QuestionSkillLinkModels
         linked to the skills, sorted by the absolute value of the difference
         between skill difficulty and the requested difficulty.
 
@@ -338,15 +339,12 @@ def get_question_skill_links_based_on_difficulty_equidistributed_by_skill(
                 requested to be fetched.
 
         Returns:
-            list(QuestionSkillLinkModel). A list of QuestionSkillLinkModels
-                corresponding to given skill_ids, with
+            list(QuestionSkillLinkModel). A list of random
+                QuestionSkillLinkModels corresponding to given skill_ids, with
                 total_question_count/len(skill_ids) number of questions for
                 each skill. If not evenly divisible, it will be rounded up.
                 If not enough questions for a skill, just return all questions
-                it links to. The order of questions will follow the order of
-                given skill ids, and the order of questions for the same skill
-                follows the absolute value of the difference between skill
-                difficulty and the requested difficulty.
+                it links to.
         """
         if len(skill_ids) > feconf.MAX_NUMBER_OF_SKILL_IDS:
             raise Exception('Please keep the number of skill IDs below 20.')
@@ -357,59 +355,98 @@ def get_question_skill_links_based_on_difficulty_equidistributed_by_skill(
 
         question_skill_link_mapping = {}
 
+        # For fetching the questions randomly we have used a random offset.
+        # But this is a temporary solution since this method scales linearly.
+        # Other alternative methods were:
+        # 1) Using a random id in question id filter
+        # 2) Adding an additional column that can be filtered upon.
+        # But these methods are not viable because google datastore limits
+        # each query to have at most one inequality filter. So we can't filter
+        # on both question_id and difficulty. Please see
+        # https://github.com/oppia/oppia/pull/9061#issuecomment-629765809
+        # for more details.
+
+        def get_offset(query):
+            """Helper function to get the offset."""
+            question_count = query.count()
+            if question_count > 2 * question_count_per_skill:
+                return utils.get_random_int(
+                    question_count - (question_count_per_skill * 2))
+            return 0
+
         for skill_id in skill_ids:
             query = cls.query(cls.skill_id == skill_id)
 
             equal_questions_query = query.filter(
                 cls.skill_difficulty == difficulty_requested)
+
             # We fetch more questions here in order to try and ensure that the
             # eventual number of returned questions is sufficient to meet the
             # number requested, even after deduplication.
-            new_question_skill_link_models = (
-                equal_questions_query.fetch(question_count_per_skill * 2))
+            new_question_skill_link_models = equal_questions_query.fetch(
+                limit=question_count_per_skill * 2,
+                offset=get_offset(equal_questions_query))
             for model in new_question_skill_link_models:
                 if model.question_id in question_skill_link_mapping:
                     new_question_skill_link_models.remove(model)
 
-            if len(new_question_skill_link_models) < question_count_per_skill:
+            if len(new_question_skill_link_models) >= question_count_per_skill:
+                new_question_skill_link_models = random.sample(
+                    new_question_skill_link_models, question_count_per_skill)
+            else:
                 # Fetch QuestionSkillLinkModels with difficulty smaller than
-                # requested difficulty and sort them by decreasing difficulty.
+                # requested difficulty.
                 easier_questions_query = query.filter(
                     cls.skill_difficulty < difficulty_requested)
-                easier_questions_query = easier_questions_query.order(
-                    -cls.skill_difficulty)
                 easier_question_skill_link_models = (
-                    easier_questions_query.fetch(question_count_per_skill))
+                    easier_questions_query.fetch(
+                        limit=question_count_per_skill * 2,
+                        offset=get_offset(easier_questions_query)))
                 for model in easier_question_skill_link_models:
                     if model.question_id in question_skill_link_mapping:
                         easier_question_skill_link_models.remove(model)
-                new_question_skill_link_models.extend(
-                    easier_question_skill_link_models)
-
-                if (len(new_question_skill_link_models) <
-                        question_count_per_skill):
+                question_extra_count = (
+                    len(new_question_skill_link_models) +
+                    len(easier_question_skill_link_models) -
+                    question_count_per_skill)
+                if question_extra_count >= 0:
+                    easier_question_skill_link_models = random.sample(
+                        easier_question_skill_link_models,
+                        question_count_per_skill -
+                        len(new_question_skill_link_models)
+                    )
+                    new_question_skill_link_models.extend(
+                        easier_question_skill_link_models)
+                else:
                     # Fetch QuestionSkillLinkModels with difficulty larger than
-                    # requested difficulty and sort them by increasing
-                    # difficulty.
+                    # requested difficulty.
+                    new_question_skill_link_models.extend(
+                        easier_question_skill_link_models)
                     harder_questions_query = query.filter(
                         cls.skill_difficulty > difficulty_requested)
-                    harder_questions_query = harder_questions_query.order(
-                        cls.skill_difficulty)
                     harder_question_skill_link_models = (
-                        harder_questions_query.fetch(question_count_per_skill))
+                        harder_questions_query.fetch(
+                            limit=question_count_per_skill * 2,
+                            offset=get_offset(harder_questions_query)))
+                    harder_question_skill_link_models = (
+                        harder_questions_query.fetch())
                     for model in harder_question_skill_link_models:
                         if model.question_id in question_skill_link_mapping:
                             harder_question_skill_link_models.remove(model)
+                    question_extra_count = (
+                        len(new_question_skill_link_models) +
+                        len(harder_question_skill_link_models) -
+                        question_count_per_skill)
+                    if question_extra_count >= 0:
+                        harder_question_skill_link_models = (
+                            random.sample(
+                                harder_question_skill_link_models,
+                                question_count_per_skill -
+                                len(new_question_skill_link_models)
+                            ))
                     new_question_skill_link_models.extend(
                         harder_question_skill_link_models)
 
-                # Sort QuestionSkillLinkModels by the difference between their
-                # difficulty and requested difficulty.
-                new_question_skill_link_models = sorted(
-                    new_question_skill_link_models,
-                    key=lambda model: abs(
-                        model.skill_difficulty - difficulty_requested)
-                )
             new_question_skill_link_models = (
                 new_question_skill_link_models[:question_count_per_skill])
 
@@ -422,8 +459,8 @@ def get_question_skill_links_based_on_difficulty_equidistributed_by_skill(
     @classmethod
     def get_question_skill_links_equidistributed_by_skill(
             cls, total_question_count, skill_ids):
-        """Fetches the list of constant number of QuestionSkillLinkModels
-        linked to the skills.
+        """Fetches the list of constant number of random
+        QuestionSkillLinkModels linked to the skills.
 
         Args:
             total_question_count: int. The number of questions expected.
@@ -431,14 +468,12 @@ def get_question_skill_links_equidistributed_by_skill(
                 question ids are to be retrieved.
 
         Returns:
-            list(QuestionSkillLinkModel). A list of QuestionSkillLinkModels
-                corresponding to given skill_ids, with
+            list(QuestionSkillLinkModel). A list of random
+                QuestionSkillLinkModelscorresponding to given skill_ids, with
                 total_question_count/len(skill_ids) number of questions for
                 each skill. If not evenly divisible, it will be rounded up.
                 If not enough questions for a skill, just return all questions
-                it links to. The order of questions will follow the order of
-                given skill ids, but the order of questions for the same skill
-                is random.
+                it links to.
         """
         if len(skill_ids) > feconf.MAX_NUMBER_OF_SKILL_IDS:
             raise Exception('Please keep the number of skill IDs below 20.')
@@ -450,23 +485,44 @@ def get_question_skill_links_equidistributed_by_skill(
         question_skill_link_models = []
         existing_question_ids = []
 
+        def get_offset(query):
+            """Helper function to get the offset."""
+            question_count = query.count()
+            if question_count > 2 * question_count_per_skill:
+                return utils.get_random_int(
+                    question_count - (question_count_per_skill * 2))
+            return 0
+
         for skill_id in skill_ids:
             query = cls.query(cls.skill_id == skill_id)
+
             # We fetch more questions here in order to try and ensure that the
             # eventual number of returned questions is sufficient to meet the
             # number requested, even after deduplication.
             new_question_skill_link_models = query.fetch(
-                question_count_per_skill * 2)
+                limit=question_count_per_skill * 2,
+                offset=get_offset(query))
 
             # Deduplicate if the same question is linked to multiple skills.
             for model in new_question_skill_link_models:
                 if model.question_id in existing_question_ids:
                     new_question_skill_link_models.remove(model)
+            if len(new_question_skill_link_models) > question_count_per_skill:
+                sampled_question_skill_link_models = random.sample(
+                    new_question_skill_link_models,
+                    question_count_per_skill
+                )
+            else:
+                sampled_question_skill_link_models = (
+                    new_question_skill_link_models)
+
 
             question_skill_link_models.extend(
-                new_question_skill_link_models[:question_count_per_skill])
-            existing_question_ids.extend(
-                [model.question_id for model in new_question_skill_link_models])
+                sampled_question_skill_link_models)
+            existing_question_ids.extend([
+                model.question_id for model in (
+                    sampled_question_skill_link_models)
+            ])
 
         return question_skill_link_models
 


Test patch:
diff --git a/core/controllers/reader_test.py b/core/controllers/reader_test.py
--- a/core/controllers/reader_test.py
+++ b/core/controllers/reader_test.py
@@ -318,19 +318,10 @@ def test_get_exploration_pretests(self):
         # Call the handler.
         with self.swap(feconf, 'NUM_PRETEST_QUESTIONS', 1):
             json_response_1 = self.get_json(
-                '%s/%s?story_id=%s&cursor=' % (
+                '%s/%s?story_id=%s' % (
                     feconf.EXPLORATION_PRETESTS_URL_PREFIX, exp_id, story_id))
-            next_cursor = json_response_1['next_start_cursor']
-
-            self.assertEqual(len(json_response_1['pretest_question_dicts']), 1)
-            json_response_2 = self.get_json(
-                '%s/%s?story_id=%s&cursor=%s' % (
-                    feconf.EXPLORATION_PRETESTS_URL_PREFIX, exp_id, story_id,
-                    next_cursor))
-            self.assertEqual(len(json_response_2['pretest_question_dicts']), 1)
-            self.assertNotEqual(
-                json_response_1['pretest_question_dicts'][0]['id'],
-                json_response_2['pretest_question_dicts'][0]['id'])
+        self.assertTrue(json_response_1['pretest_question_dicts'][0]['id'] in
+                        [question_id, question_id_2])
 
         self.get_json(
             '%s/%s?story_id=%s' % (
diff --git a/core/storage/question/gae_models_test.py b/core/storage/question/gae_models_test.py
--- a/core/storage/question/gae_models_test.py
+++ b/core/storage/question/gae_models_test.py
@@ -18,6 +18,7 @@
 from __future__ import unicode_literals  # pylint: disable=import-only-modules
 
 import datetime
+import random
 import types
 
 from constants import constants
@@ -26,6 +27,7 @@
 from core.platform import models
 from core.tests import test_utils
 import python_utils
+import utils
 
 (base_models, question_models) = models.Registry.import_models(
     [models.NAMES.base_model, models.NAMES.question])
@@ -388,6 +390,68 @@ def test_get_question_skill_links_based_on_difficulty(self):
         self.assertTrue(questionskilllink_model2 in question_skill_links)
         self.assertTrue(questionskilllink_model4 in question_skill_links)
 
+    def test_get_random_question_skill_links_based_on_difficulty(self):
+        questionskilllink_model1 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id1', 'skill_id1', 0.6)
+            )
+        questionskilllink_model2 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id2', 'skill_id1', 0.6)
+            )
+        questionskilllink_model3 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id3', 'skill_id1', 0.6)
+            )
+        questionskilllink_model4 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id4', 'skill_id1', 0.6)
+            )
+        questionskilllink_model5 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id5', 'skill_id1', 0.6)
+            )
+        questionskilllink_model6 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id6', 'skill_id1', 0.6)
+            )
+        questionskilllink_model7 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id7', 'skill_id1', 0.6)
+            )
+        questionskilllink_model8 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id8', 'skill_id1', 0.6)
+            )
+        question_models.QuestionSkillLinkModel.put_multi_question_skill_links(
+            [questionskilllink_model1, questionskilllink_model2,
+             questionskilllink_model3, questionskilllink_model4,
+             questionskilllink_model5, questionskilllink_model6,
+             questionskilllink_model7, questionskilllink_model8])
+        def mock_random_sample(alist, num):
+            if num >= len(alist):
+                return alist
+            alist.sort(key=lambda x: x.question_id)
+            return alist[:num]
+
+        sample_swap = self.swap(random, 'sample', mock_random_sample)
+
+        def mock_random_int(upper_bound):
+            return 1 if upper_bound > 1 else 0
+        random_int_swap = self.swap(utils, 'get_random_int', mock_random_int)
+        with sample_swap, random_int_swap:
+            question_skill_links_1 = (
+                question_models.QuestionSkillLinkModel.
+                get_question_skill_links_based_on_difficulty_equidistributed_by_skill( # pylint: disable=line-too-long
+                    3, ['skill_id1'], 0.6
+                )
+            )
+        self.assertEqual(len(question_skill_links_1), 3)
+        self.assertEqual(
+            question_skill_links_1,
+            [questionskilllink_model2, questionskilllink_model3,
+             questionskilllink_model4])
+
     def test_request_too_many_skills_raises_error_when_fetch_by_difficulty(
             self):
         skill_ids = ['skill_id%s' % number for number in python_utils.RANGE(25)]
@@ -492,6 +556,67 @@ def test_get_question_skill_links_equidistributed_by_skill(
         question_ids = [link.question_id for link in question_skill_links]
         self.assertEqual(question_ids.count('question_id2'), 1)
 
+    def test_get_random_question_skill_links_equidistributed_by_skill(self):
+        questionskilllink_model1 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id1', 'skill_id1', 0.1)
+            )
+        questionskilllink_model2 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id2', 'skill_id1', 0.5)
+            )
+        questionskilllink_model3 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id3', 'skill_id1', 0.8)
+            )
+        questionskilllink_model4 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id4', 'skill_id1', 0.9)
+            )
+        questionskilllink_model5 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id5', 'skill_id1', 0.6)
+            )
+        questionskilllink_model6 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id6', 'skill_id1', 0.6)
+            )
+        questionskilllink_model7 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id7', 'skill_id1', 0.6)
+            )
+        questionskilllink_model8 = (
+            question_models.QuestionSkillLinkModel.create(
+                'question_id8', 'skill_id1', 0.6)
+            )
+        question_models.QuestionSkillLinkModel.put_multi_question_skill_links(
+            [questionskilllink_model1, questionskilllink_model2,
+             questionskilllink_model3, questionskilllink_model4,
+             questionskilllink_model5, questionskilllink_model6,
+             questionskilllink_model7, questionskilllink_model8])
+        def mock_random_sample(alist, num):
+            if num >= len(alist):
+                return alist
+            alist.sort(key=lambda x: x.question_id)
+            return alist[:num]
+
+        sample_swap = self.swap(random, 'sample', mock_random_sample)
+        def mock_random_int(upper_bound):
+            return 1 if upper_bound > 1 else 0
+        random_int_swap = self.swap(utils, 'get_random_int', mock_random_int)
+        with sample_swap, random_int_swap:
+            question_skill_links_1 = (
+                question_models.QuestionSkillLinkModel.
+                get_question_skill_links_equidistributed_by_skill(
+                    3, ['skill_id1']
+                )
+            )
+        self.assertEqual(len(question_skill_links_1), 3)
+        self.assertEqual(
+            question_skill_links_1,
+            [questionskilllink_model2, questionskilllink_model3,
+             questionskilllink_model4])
+
     def test_request_too_many_skills_raises_error(self):
         skill_ids = ['skill_id%s' % number for number in python_utils.RANGE(25)]
         with self.assertRaisesRegexp(