Repository: oppia/oppia

Issue:
At the end of an exploration that's part of a collection, the next exploration in the collection should be clearly highlighted.
When playing through the ["Programming with Carla" collection](https://www.oppia.org/collection/inDXV0w8-p1C) on Oppia.org, I noticed these recommendations at the end of the collection:

![bad-recs](https://cloud.githubusercontent.com/assets/10575562/17647916/606db22c-61b9-11e6-8b74-83723fdfc92a.png)

These seem mostly random. I think that, at the end of an exploration that's being played in the context of a collection, only the next exploration(s) in the collection should be highlighted as recommendations, and nothing else. The design for this needs to be worked out, too -- perhaps it is better to do it as part of the "interaction" section of the card.



Discussion:
I recall this was the behavior previously. Is this a regression?

Yes. Checked with @BenHenning and he believes it's a regression too. (I couldn't recall what the previous behaviour was.)

previous behavior was to show 1 exploration which was the next one in the collection.

@edallison, would you be up for looking into this? 

(Note that instructions for accessing the collection editor are [here](https://github.com/oppia/oppia/issues/1587#issuecomment-196313127).)

@seanlip you can assign it to me please.
I spent some time yesterday looking at this.  It looks like since the recommended modules object thats being passed in doesn't include a 'collectionId' property for explorations the issue may lie in the RecommendationsHandler in the backend.  Does this sound like a reasonable place to start?

Assigned, thanks!

Hm, I'm not sure -- I think this needs more investigation. We don't necessarily need to use the recommendations handler for this, actually; the main thing is that the frontend has enough data about the "next" exploration in the collection, which is a different thing. A quick glance suggests that collection_services.get_next_exploration_ids_to_complete_by_user() might be relevant; you could perhaps make an async call to the backend to get this info when the learner reaches the end of their exploration.

@BenHenning, do you have any advice? In any case, @edallison -- it's probably worth outlining your approach here first, without coding anything, just so that we get agreement on the design approach. If you haven't found anything by tonight, I could dig into a bit more then.

I don't have a lot of time to work on this today and apologize that this has been thought out pretty hastily but the approach I'm looking at right now would very generally be:
- In  ExplorationRecommendationsService.js make a call to get_next_exploration_ids_to_complete_by_user().  
- Have ExplorationRecommendationsService.js return another function (named getNextExplorationIdsInCollection?) that passes these ids to ConversationSkinDirective.js as an array or object (named $scope.nextExplorationIdsInCollection?).
- In conversation_skin_directive.html line 198 where recommendedExplorationsSummaries is being looped/filtered,  add filtering that will prioritize explorations that match the tentatively named nextExplorationIdsInCollection array/object.

I am open to any and all suggestions.

Hi @edallison, I took a look! It seems to me that the frontend logic is sound, but I did find a bug in the backend logic that _might_ be the cause of this issue. In particular, in core.controllers.reader.RecommendationsHandler, the logic might need to be changed.

Here's what I suspect the issue is. If I'm reading correctly, I'm noticing that in that handler, if the user is not logged in, then the 'if' condition in the first branch does not trigger, and the generic recommendations get included instead. The function in the 'if' condition seems to require a user_id, though, so I wonder if we could do something like collection.get_next_exploration_ids([current_exploration_id]) instead...

Also, @BenHenning, a question for you: if the user is not signed in, what should be the intended behaviour for recommendations at the end of the collection? I'm guessing we just show the "next" one. Is that reasonable?

The problem is, though, that in this case collection.get_next_exploration_ids() is going to recommend, e.g., the very first exploration in the sequence, since it doesn't know that the user has already done it (since the user isn't logged in). So, assuming we don't do any local storage stuff yet, maybe we have to add a function like collection.get_next_exploration_id_in_sequence(); this function checks whether the backend collection is linear and, if so, returns the next exploration id after the one that's passed in (which is the exploration that the user is currently on). If the collection is not linear, it errors. What do you think?

@seanlip @BenHenning Locally I've added the logic that will suggest other explorations in the collection to logged out users.  As @seanlip pointed out, there is still the issue of previous explorations in the collection being suggested.  I'm happy to move forward with the collection.get_next_exploration_id_in_sequence() approach if everyone is in agreement.

> Also, @BenHenning, a question for you: if the user is not signed in, what should be the intended behaviour for recommendations at the end of the collection? I'm guessing we just show the "next" one. Is that reasonable?
> 
> @seanlip @BenHenning Locally I've added the logic that will suggest other explorations in the collection to logged out users. As @seanlip pointed out, there is still the issue of previous explorations in the collection being suggested. I'm happy to move forward with the collection.get_next_exploration_id_in_sequence() approach if everyone is in agreement.

My concern is that collections are not linear in the backend, so a notion of "next in sequence" is confusing. The original bug reported in this issue is definitely a regression for logged in users, since recommendations should not be displayed at all. It's possible for more than one exploration to be suggested for a collection, since collections are non-linear. We simply cannot create those collections yet in the current collection editor.

For logged out users, I see two possible solutions:
1. We store a temporary state recording their progress and use this with the current `get_next_exploration_ids` to get real suggestions
2. We show N explorations which must come after completing the given exploration (this isn't quite the same thing as next-in-sequence since collections are non-linear, but if exploration E is completed, we can assume all explorations leading to it have also been completed). Some sort of edge case logic needs to be applied if the last exploration of the collection is the one that's finished (ie, maybe loop back around?)

I think the original bug happened when I was logged out, so we probably just need to consider that case.

How about approach 2? I want to avoid solutions which involve localstorage for now; we should define and design the usage of localStorage more comprehensively in a later pass. So, here's a possible algorithm:
- call the current exploration E; say its acquired skills are A
- deduce recursively which explorations must have been completed, based on E's prerequisite skills, and let P be the set of all skills the user must have acquired by the time they've completed E
- return only unseen explorations in the collection whose prerequisites are contained in P, and whose prerequisites contain at least one member of A (since this means that such explorations have only just become 'unlocked')
- if the preceding list is empty, return instead up to N unseen explorations in the collection whose prerequisites are contained in P
- if the preceding step gives no results, just return any N explorations in general (this refers to the "last exploration of the collection" case)

@seanlip I'm having a little trouble understanding

> call the current exploration E; say its acquired skills are A

By acquired skills do you mean explorations that come before E in the collection or is there a property of explorations that I am unaware of?

Each "exploration node" in a collection is associated with prerequisite skills and acquired skils. In this case I'm referring just to the "acquired skills" property of only exploration E.

I've spent some time looking at this today and should be able to continue tomorrow.  One issue I'm facing is being able to recursively determine which explorations have been completed based on their prerequisite skills.  The problem is that prerequisite skills are stored as an array of exploration titles instead of ids. This makes checking the prerequisite skills of a prerequisite skill difficult because collection.get_node() requires an id as the parameter.

I considered using exp_services.get_exploration_titles_and_categories(collection_node_ids) to map the collection node's to their titles but didn't know if this was a good idea. @seanlip Thoughts?

I think the skills are not exploration titles, actually -- they're actually just meant to be arbitrary strings that are not related to the ids or titles of the collections. In the current 'linear' version of collections, the exploration titles might be being used as the collection skill strings by default.

So, I think, to do this calculation, you might indeed want to get a list of all the prereq/acquired skills for each exploration node in the collection. Then you should have all the data that you need to implement the recursive algorithm. You shouldn't need to get the titles of the individual explorations, though.

/cc @BenHenning @amgowano -- please correct me if anything I've said above is wrong. Thanks!

@seanlip I have this in a functioning state but have a question.  

I'm not sure what you mean by "unseen explorations".  It doesn't look like collection nodes have this property and I haven't seen it elsewhere.  The reason I ask is that the third case you've said to return is never reached. In the situation that the current exploration is the last exploration in the collection, the second return case (ignoring the "unseen explorations" stipulation) will happen since there would be several explorations in the collection whose prerequisites are contained in P.

Let me know if I can clarify further.  I can also push a branch of what a currently have up if that would be helpful.

By "unseen" I meant that the student hasn't touched the exploration, but I just realized this was unclear since we don't have a logged-in student. I meant "unseen explorations" to be any exploration that is not in the set that was collated in the second bullet point. Does this clarify?

Also, feel free to push a branch! Happy to take a look.

Also, for the "last exploration" case, I think this means that all explorations will (presumably) be in the set you compiled in the second bullet point, so that would result in a fall-through to the third case.


Resolving patch:
diff --git a/core/controllers/reader.py b/core/controllers/reader.py
--- a/core/controllers/reader.py
+++ b/core/controllers/reader.py
@@ -593,16 +593,28 @@ def get(self, exploration_id):
             auto_recommended_exp_ids = list(
                 set(next_exp_ids_in_collection) -
                 set(author_recommended_exp_ids))
-        elif include_system_recommendations:
-            system_chosen_exp_ids = (
-                recommendations_services.get_exploration_recommendations(
-                    exploration_id))
-            filtered_exp_ids = list(
-                set(system_chosen_exp_ids) -
-                set(author_recommended_exp_ids))
-            auto_recommended_exp_ids = random.sample(
-                filtered_exp_ids,
-                min(MAX_SYSTEM_RECOMMENDATIONS, len(filtered_exp_ids)))
+        else:
+            next_exp_ids_in_collection = []
+            if collection_id:
+                collection = collection_services.get_collection_by_id(
+                    collection_id)
+                next_exp_ids_in_collection = (
+                    collection.get_next_exploration_ids_in_sequence(
+                        exploration_id))
+            if next_exp_ids_in_collection:
+                auto_recommended_exp_ids = list(
+                    set(next_exp_ids_in_collection) -
+                    set(author_recommended_exp_ids))
+            elif include_system_recommendations:
+                system_chosen_exp_ids = (
+                    recommendations_services.get_exploration_recommendations(
+                        exploration_id))
+                filtered_exp_ids = list(
+                    set(system_chosen_exp_ids) -
+                    set(author_recommended_exp_ids))
+                auto_recommended_exp_ids = random.sample(
+                    filtered_exp_ids,
+                    min(MAX_SYSTEM_RECOMMENDATIONS, len(filtered_exp_ids)))
 
         self.values.update({
             'summaries': (
diff --git a/core/domain/collection_domain.py b/core/domain/collection_domain.py
--- a/core/domain/collection_domain.py
+++ b/core/domain/collection_domain.py
@@ -435,6 +435,59 @@ def get_next_exploration_ids(self, completed_exploration_ids):
                 next_exp_ids.append(node.exploration_id)
         return next_exp_ids
 
+    def get_next_exploration_ids_in_sequence(self, current_exploration):
+        """Returns a list of exploration IDs that a logged-out user should
+        complete next based on the prerequisite skills they must have attained
+        by the time they completed the current exploration.  This recursively
+        compiles a list of 'learned skills' then, depending on the
+        'learned skills' and the current exploration's acquired skills,
+        returns either a list of exploration ids that have either just
+        unlocked or the user is qualified to explore.  If neither of these
+        lists can be generated a blank list is returned instead."""
+        skills_learned_by_exp_id = {}
+
+        def _recursively_find_learned_skills(node):
+            """Given a node, returns the skills that the user must have
+            acquired by the time they've completed it."""
+            if node.exploration_id in skills_learned_by_exp_id:
+                return skills_learned_by_exp_id[node.exploration_id]
+
+            skills_learned = set(node.acquired_skills)
+            for other_node in self.nodes:
+                if other_node.exploration_id not in skills_learned_by_exp_id:
+                    for skill in node.prerequisite_skills:
+                        if skill in other_node.acquired_skills:
+                            skills_learned = skills_learned.union(
+                                _recursively_find_learned_skills(other_node))
+
+            skills_learned_by_exp_id[node.exploration_id] = skills_learned
+            return skills_learned
+
+        explorations_just_unlocked = []
+        explorations_qualified_for = []
+
+        collection_node = self.get_node(current_exploration)
+        collected_skills = _recursively_find_learned_skills(collection_node)
+
+        for node in self.nodes:
+            if node.exploration_id in skills_learned_by_exp_id:
+                continue
+
+            if set(node.prerequisite_skills).issubset(set(collected_skills)):
+                if (any([
+                        skill in collection_node.acquired_skills
+                        for skill in node.prerequisite_skills])):
+                    explorations_just_unlocked.append(node.exploration_id)
+                else:
+                    explorations_qualified_for.append(node.exploration_id)
+
+        if explorations_just_unlocked:
+            return explorations_just_unlocked
+        elif explorations_qualified_for:
+            return explorations_qualified_for
+        else:
+            return []
+
     @classmethod
     def is_demo_collection_id(cls, collection_id):
         """Whether the collection id is that of a demo collection."""
diff --git a/core/jobs_registry.py b/core/jobs_registry.py
--- a/core/jobs_registry.py
+++ b/core/jobs_registry.py
@@ -23,6 +23,7 @@
 from core.domain import user_jobs_continuous
 from core.domain import user_jobs_one_off
 from core.domain import email_jobs_one_off
+from core.domain import recommendations_jobs_one_off
 
 # List of all manager classes for one-off batch jobs for which to show controls
 # on the admin dashboard.
@@ -41,7 +42,8 @@
     email_jobs_one_off.EmailHashRegenerationOneOffJob,
     user_jobs_one_off.UserProfilePictureOneOffJob,
     exp_jobs_one_off.ItemSelectionInteractionOneOffJob,
-    user_jobs_one_off.UserLastExplorationActivityOneOffJob]
+    user_jobs_one_off.UserLastExplorationActivityOneOffJob,
+    recommendations_jobs_one_off.ExplorationRecommendationsOneOffJob]
 
 # List of all ContinuousComputation managers to show controls for on the
 # admin dashboard.


Test patch:
diff --git a/core/domain/collection_domain_test.py b/core/domain/collection_domain_test.py
--- a/core/domain/collection_domain_test.py
+++ b/core/domain/collection_domain_test.py
@@ -460,6 +460,68 @@ def test_next_explorations(self):
             collection.get_next_exploration_ids(
                 ['exp_id_0', 'exp_id_1', 'exp_id_2', 'exp_id_3']), [])
 
+    def test_next_explorations_in_sequence(self):
+
+        collection = collection_domain.Collection.create_default_collection(
+            'collection_id')
+        exploration_id = 'exp_id_0'
+        collection.add_node(exploration_id)
+
+        # Completing the only exploration of the collection should lead to no
+        # available explorations thereafter.
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence(exploration_id), [])
+
+        # If the current exploration has no acquired skills, a list of all
+        # explorations with no prerequisite skills should be returned.
+        collection.add_node('exp_id_1')
+        collection.add_node('exp_id_2')
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence(exploration_id),
+            ['exp_id_1', 'exp_id_2'])
+
+        # If only one exploration in the collection has a prerequisite skill
+        # that is included in the user's learned skills, only that exploration
+        # should be returned.
+        collection_node0 = collection.get_node('exp_id_0')
+        collection_node1 = collection.get_node('exp_id_1')
+
+        collection_node0.update_acquired_skills(['skill1a'])
+        collection_node1.update_prerequisite_skills(['skill1a'])
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence(exploration_id),
+            ['exp_id_1'])
+
+        # Given a chain of explorations in a collections where each
+        # exploration's acquired skills are the following exploration's
+        # prerequisite skills, each exploration should return the following
+        # exploration as a recommendation.  The last exploration should
+        # return an empty list.
+        collection.add_node('exp_id_3')
+
+        collection_node2 = collection.get_node('exp_id_2')
+        collection_node3 = collection.get_node('exp_id_3')
+
+        collection_node1.update_acquired_skills(['skill2a'])
+        collection_node2.update_acquired_skills(['skill3a'])
+
+        collection_node0.update_prerequisite_skills([])
+        collection_node2.update_prerequisite_skills(['skill2a'])
+        collection_node3.update_prerequisite_skills(['skill3a'])
+
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence('exp_id_0'),
+            ['exp_id_1'])
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence('exp_id_1'),
+            ['exp_id_2'])
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence('exp_id_2'),
+            ['exp_id_3'])
+        self.assertEqual(
+            collection.get_next_exploration_ids_in_sequence('exp_id_3'),
+            [])
+
     def test_next_explorations_with_invalid_exploration_ids(self):
         collection = collection_domain.Collection.create_default_collection(
             'collection_id')