Repository: oppia/oppia

Issue:
Topics and Skills not showing in contributors dashboard to suggest questions.
<!--
  - Thanks for taking the time to report a bug in the Oppia project.
  - Before filing a new issue, please do a quick search to check that it hasn't
  - already been filed on the [issue tracker](https://github.com/oppia/oppia/issues)._
  -->

**Describe the bug**
Topics and skills are added on the server but on the contributors dashboard it is showing no opportunities available under submit question section.

**To Reproduce**
Steps to reproduce the behavior:
 1. Get permission to suggest questions through contributors dashboard on Oppia test server.
 2. Go to the contributors dashboard and click on submit a question. There you'll see no opportunity available but published topics are present on Oppia server.

**Observed behavior**
Topics and skills are added on the server but on the contributors dashboard it is showing no opportunities available under submit question section.

**Expected behavior**
Opportunities should be shown in submit question section of contributors dashboard.

**Screenshots**
![image](https://user-images.githubusercontent.com/56637531/129400059-323a44ec-0603-45fc-98a7-ef1f15e5ba35.png)

**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: Mac
 - Browser Chrome
 - Browser-version 92




Discussion:
@DubeySandeep could you please look into this contributor dashboard issue or reassign it?
Reassigning to @sagangwee as lead of the contributor dashboard team. @sagangwee could you please prioritize fixing this blocking bug?
@atpug22 Are the topics part of a classroom? As of https://github.com/oppia/oppia/pull/12982, we only show opportunities for classroom topics.
> @atpug22 Are the topics part of a classroom? As of #12982, we only show opportunities for classroom topics.

@sagangwee I am not sure of this as only admins can access the topics pages. @kevintab95 Can you please confirm that the topics on the test server are part of a classroom or not?
@atpug22 You can find the full list of classroom topics by going to the classroom page: https://www.oppiatestserver.org/learn/math
> @atpug22 You can find the full list of classroom topics by going to the classroom page: https://www.oppiatestserver.org/learn/math

Oh got it. Sorry for the confusion. So are the opportunities shown in the questions for all the topics or the skills? I am a bit confused regarding this? @sagangwee 
A question opportunity is created for each skill, but is only shown in the dashboard if the opportunity's skill is linked to a classroom topic.
Here's what I see in the Fractions topic editor view:

![image](https://user-images.githubusercontent.com/11008603/130290103-79541410-32c4-4e39-821b-98ccc2af7cd5.png)


Looks like this topic is attached to the "math" classroom (it shows up in `/learn/math`) and it has a total of 12 questions attached to it.
Checking the Network tab in Chrome dev tools, doesn't look like the server handler returned any opportunities, so this isn't an issue with filtering on the frontend, but rather something server-side.
Also still seems to be working locally.
@sagangwee since it's working locally, I see two possible explanations:

1. This could be because of some problem with the data on the server
2. A recent PR could have fixed this. You could try checking out the release branch locally to test this?
@U8NWXD Which is the correct release branch?
`release-3.1.4`
I'm not able to start a local server on `release-3.1.4` due to:

```
Traceback (most recent call last):
  File "/Users/sagangwee/.pyenv/versions/3.7.10/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/sagangwee/.pyenv/versions/3.7.10/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/sagangwee/opensource3/oppia/scripts/start.py", line 28, in <module>
    from . import install_third_party_libs
  File "/Users/sagangwee/opensource3/oppia/scripts/install_third_party_libs.py", line 48, in <module>
    if 'can\'t combine user with prefix' in output_stderr:
TypeError: a bytes-like object is required, not 'str'
```
The release 3.1.4 branch was cut before the Python 3 changes were merged. So you'll need to use Python 2 to start the server. 
And to double-check, the topic ID of the topic in https://github.com/oppia/oppia/issues/13657#issuecomment-902940420 is listed in the Admin config for classroom topics?
Yup. "GJ2rLXRKD5hw" is present in the admin config for classroom. 
Hmm seems to be working locally on the release 3.1.4 branch as well.
@kevintab95 Are there SkillOpportunityModels in storage on the test server? Wondering if they were deleted somehow, or if they exist and they're just not being returned on fetch.
Yup there seem to be `SkillOpportunityModel`s in the test server.
Ok thanks. Then, I'm not sure how to debug this further on my end as I can't reproduce it locally and I'm not able to inspect the test server state.
Hi @sagangwee, I see the following options to debug this further:

1. Check if the opportunity models created in release-3.1.3 on your local machine persist and show up in the CD when you checkout release-3.1.4 while the server is up and running.
2. Add relevant logging in the fetch flow for opportunities. We can then deploy these logs on the test server and hopefully get some useful data to understand this better and come up with a fix.
Added relevant logging in https://github.com/oppia/oppia/pull/13744
Thanks @sagangwee. I've deployed #13744 on the test server and got some logs. I've emailed the results/logs to you & U8NWXD. 
From offline debugging: "I think what might be happening is that we only fetch 10 opportunities from the datastore at once and then filter these 10 depending on whether they correspond to Classroom topics in the controller. If none of them do, we'll just return no opportunities and the user isn't able to fetch more. Also realizing the current logic might miss some opportunities as well."

Resolving patch:
diff --git a/core/controllers/contributor_dashboard.py b/core/controllers/contributor_dashboard.py
--- a/core/controllers/contributor_dashboard.py
+++ b/core/controllers/contributor_dashboard.py
@@ -119,7 +119,7 @@ def get(self, opportunity_type):
 
     def _get_skill_opportunities_with_corresponding_topic_name(self, cursor):
         """Returns a list of skill opportunities available for questions with
-        topic information.
+        a corresponding topic name.
 
         Args:
             cursor: str or None. If provided, the list of returned entities
@@ -143,24 +143,38 @@ def _get_skill_opportunities_with_corresponding_topic_name(self, cursor):
         for classroom_dict in config_domain.CLASSROOM_PAGES_DATA.value:
             classroom_topic_ids.extend(classroom_dict['topic_ids'])
         classroom_topics = topic_fetchers.get_topics_by_ids(classroom_topic_ids)
-        classroom_topics_with_skills = [
-            topic for topic in classroom_topics
-            if topic and topic.get_all_skill_ids()
-        ]
+        # Associate each skill with one classroom topic name.
+        # TODO(#8912): Associate each skill/skill opportunity with  all linked
+        # topics.
+        classroom_topic_skill_id_to_topic_name = {}
+        for topic in classroom_topics:
+            if topic is None:
+                continue
+            for skill_id in topic.get_all_skill_ids():
+                classroom_topic_skill_id_to_topic_name[skill_id] = topic.name
+
         skill_opportunities, cursor, more = (
             opportunity_services.get_skill_opportunities(cursor))
-        id_to_skill_opportunity_dict = {
-            opp.id: opp.to_dict() for opp in skill_opportunities}
         opportunities = []
-        for topic in classroom_topics_with_skills:
-            for skill_id in topic.get_all_skill_ids():
-                if len(opportunities) == constants.OPPORTUNITIES_PAGE_SIZE:
-                    break
-                if skill_id in id_to_skill_opportunity_dict:
-                    skill_opportunity_dict = (
-                        id_to_skill_opportunity_dict[skill_id])
-                    skill_opportunity_dict['topic_name'] = topic.name
+        # Fetch opportunities until we have at least a page's worth that
+        # correspond to a classroom or there are no more opportunities.
+        while len(opportunities) < constants.OPPORTUNITIES_PAGE_SIZE:
+            for skill_opportunity in skill_opportunities:
+                if (
+                        skill_opportunity.id
+                        in classroom_topic_skill_id_to_topic_name):
+                    skill_opportunity_dict = skill_opportunity.to_dict()
+                    skill_opportunity_dict['topic_name'] = (
+                        classroom_topic_skill_id_to_topic_name[
+                            skill_opportunity.id])
                     opportunities.append(skill_opportunity_dict)
+            if (
+                    not more or
+                    len(opportunities) >= constants.OPPORTUNITIES_PAGE_SIZE):
+                break
+            skill_opportunities, cursor, more = (
+                opportunity_services.get_skill_opportunities(cursor))
+
         return opportunities, cursor, more
 
     def _get_translation_opportunity_dicts(self, language_code, search_cursor):
diff --git a/core/domain/opportunity_services.py b/core/domain/opportunity_services.py
--- a/core/domain/opportunity_services.py
+++ b/core/domain/opportunity_services.py
@@ -560,10 +560,9 @@ def get_skill_opportunities(cursor):
                 this batch. If False, there are no further results after
                 this batch.
     """
-    page_size = constants.OPPORTUNITIES_PAGE_SIZE
     skill_opportunity_models, cursor, more = (
         opportunity_models.SkillOpportunityModel
-        .get_skill_opportunities(page_size, cursor))
+        .get_skill_opportunities(constants.OPPORTUNITIES_PAGE_SIZE, cursor))
     opportunities = []
     for skill_opportunity_model in skill_opportunity_models:
         skill_opportunity = (
diff --git a/core/storage/opportunity/gae_models.py b/core/storage/opportunity/gae_models.py
--- a/core/storage/opportunity/gae_models.py
+++ b/core/storage/opportunity/gae_models.py
@@ -298,7 +298,8 @@ def get_skill_opportunities(
         # TODO(#13462): Refactor this so that we don't do the lookup.
         # Do a forward lookup so that we can know if there are more values.
         plus_one_query_models, _, _ = (
-            created_on_query.fetch_page(page_size + 1, start_cursor=cursor))
+            created_on_query.fetch_page(
+                page_size + 1, start_cursor=start_cursor))
         more_results = len(plus_one_query_models) == page_size + 1
         # The urlsafe returns bytes and we need to decode them to string.
         return (


Test patch:
diff --git a/core/controllers/contributor_dashboard_test.py b/core/controllers/contributor_dashboard_test.py
--- a/core/controllers/contributor_dashboard_test.py
+++ b/core/controllers/contributor_dashboard_test.py
@@ -24,6 +24,8 @@
 from core.domain import exp_services
 from core.domain import story_domain
 from core.domain import story_services
+from core.domain import subtopic_page_domain
+from core.domain import subtopic_page_services
 from core.domain import suggestion_services
 from core.domain import topic_domain
 from core.domain import topic_services
@@ -78,35 +80,19 @@ def setUp(self):
         for exp in explorations:
             self.publish_exploration(self.owner_id, exp.id)
 
-        topic_id = '0'
+        self.topic_id = '0'
         topic = topic_domain.Topic.create_default_topic(
-            topic_id, 'topic', 'abbrev', 'description')
-        topic.thumbnail_filename = 'thumbnail.svg'
-        topic.thumbnail_bg_color = '#C6DCDA'
-        topic.subtopics = [
-            topic_domain.Subtopic(
-                1, 'Title', ['skill_id_3'], 'image.svg',
-                constants.ALLOWED_THUMBNAIL_BG_COLORS['subtopic'][0], 21131,
-                'dummy-subtopic-three')]
-        topic.next_subtopic_id = 2
-        topic_services.save_new_topic(self.owner_id, topic)
-        topic_services.publish_topic(topic_id, self.admin_id)
-
+            self.topic_id, 'topic', 'abbrev', 'description')
         self.skill_id_0 = 'skill_id_0'
         self.skill_id_1 = 'skill_id_1'
-        self.skill_ids = [self.skill_id_0, self.skill_id_1]
-        for skill_id in self.skill_ids:
-            self.save_new_skill(
-                skill_id, self.admin_id, description='skill_description')
-            topic_services.add_uncategorized_skill(
-                self.admin_id, topic_id, skill_id)
+        self._publish_valid_topic(topic, [self.skill_id_0, self.skill_id_1])
 
         # Add skill opportunity topic to a classroom.
         config_services.set_property(
             self.admin_id, 'classroom_pages_data', [{
                 'name': 'math',
                 'url_fragment': 'math-one',
-                'topic_ids': [topic_id],
+                'topic_ids': [self.topic_id],
                 'course_details': '',
                 'topic_list_intro': ''
             }])
@@ -220,6 +206,19 @@ def test_get_skill_opportunity_data_does_not_return_non_classroom_topics(
         self.assertTrue(
             isinstance(response['next_cursor'], python_utils.BASESTRING))
 
+    def test_get_skill_opportunity_data_does_not_throw_for_deleted_topics(self):
+        topic_services.delete_topic(self.admin_id, self.topic_id)
+
+        response = self.get_json(
+            '%s/skill' % feconf.CONTRIBUTOR_OPPORTUNITIES_DATA_URL,
+            params={})
+
+        self.assertEqual(
+            response['opportunities'], [])
+        self.assertFalse(response['more'])
+        self.assertTrue(
+            isinstance(response['next_cursor'], python_utils.BASESTRING))
+
     def test_get_translation_opportunity_data(self):
         response = self.get_json(
             '%s/translation' % feconf.CONTRIBUTOR_OPPORTUNITIES_DATA_URL,
@@ -256,7 +255,7 @@ def test_get_skill_opportunity_data_pagination(self):
             self.assertEqual(
                 response['opportunities'],
                 [self.expected_skill_opportunity_dict_0])
-            self.assertFalse(response['more'])
+            self.assertTrue(response['more'])
             self.assertTrue(
                 isinstance(response['next_cursor'], python_utils.BASESTRING))
 
@@ -274,6 +273,74 @@ def test_get_skill_opportunity_data_pagination(self):
                 isinstance(
                     next_response['next_cursor'], python_utils.BASESTRING))
 
+    def test_get_skill_opportunity_data_pagination_multiple_fetches(self):
+        # Unassign topic 0 from the classroom.
+        config_services.revert_property(
+            self.admin_id, 'classroom_pages_data')
+
+        # Create a new topic.
+        topic_id = '1'
+        topic_name = 'topic1'
+        topic = topic_domain.Topic.create_default_topic(
+            topic_id, topic_name, 'url-fragment', 'description')
+        skill_id_2 = 'skill_id_2'
+        skill_id_3 = 'skill_id_3'
+        skill_id_4 = 'skill_id_4'
+        skill_id_5 = 'skill_id_5'
+        self._publish_valid_topic(
+            topic, [skill_id_2, skill_id_3, skill_id_4, skill_id_5])
+
+        # Add new topic to a classroom.
+        config_services.set_property(
+            self.admin_id, 'classroom_pages_data', [{
+                'name': 'math',
+                'url_fragment': 'math-one',
+                'topic_ids': [topic_id],
+                'course_details': '',
+                'topic_list_intro': ''
+            }])
+
+        # Opportunities with IDs skill_id_0, skill_id_1, skill_id_2 will be
+        # fetched first. Since skill_id_0, skill_id_1 are not linked to a
+        # classroom, another fetch will be made to retrieve skill_id_3,
+        # skill_id_4, skill_id_5 to fulfill the page size.
+        with self.swap(constants, 'OPPORTUNITIES_PAGE_SIZE', 3):
+            response = self.get_json(
+                '%s/skill' % feconf.CONTRIBUTOR_OPPORTUNITIES_DATA_URL,
+                params={})
+            self.assertEqual(len(response['opportunities']), 4)
+            self.assertEqual(
+                response['opportunities'],
+                [
+                    {
+                        'id': skill_id_2,
+                        'skill_description': 'skill_description',
+                        'question_count': 0,
+                        'topic_name': topic_name
+                    },
+                    {
+                        'id': skill_id_3,
+                        'skill_description': 'skill_description',
+                        'question_count': 0,
+                        'topic_name': topic_name
+                    },
+                    {
+                        'id': skill_id_4,
+                        'skill_description': 'skill_description',
+                        'question_count': 0,
+                        'topic_name': topic_name
+                    },
+                    {
+                        'id': skill_id_5,
+                        'skill_description': 'skill_description',
+                        'question_count': 0,
+                        'topic_name': topic_name
+                    }
+                ])
+            self.assertFalse(response['more'])
+            self.assertTrue(
+                isinstance(response['next_cursor'], python_utils.BASESTRING))
+
     def test_get_translation_opportunity_data_pagination(self):
         with self.swap(constants, 'OPPORTUNITIES_PAGE_SIZE', 1):
             response = self.get_json(
@@ -360,6 +427,44 @@ def test_get_opportunity_for_invalid_opportunity_type(self):
                     feconf.CONTRIBUTOR_OPPORTUNITIES_DATA_URL),
                 expected_status_int=404)
 
+    def _publish_valid_topic(self, topic, uncategorized_skill_ids):
+        """Saves and publishes a valid topic with linked skills and subtopic.
+
+        Args:
+            topic: Topic. The topic to be saved and published.
+            uncategorized_skill_ids: list(str). List of uncategorized skills IDs
+                to add to the supplied topic.
+        """
+        topic.thumbnail_filename = 'thumbnail.svg'
+        topic.thumbnail_bg_color = '#C6DCDA'
+        subtopic_id = 1
+        subtopic_skill_id = 'subtopic_skill_id' + topic.id
+        topic.subtopics = [
+            topic_domain.Subtopic(
+                subtopic_id, 'Title', [subtopic_skill_id], 'image.svg',
+                constants.ALLOWED_THUMBNAIL_BG_COLORS['subtopic'][0], 21131,
+                'dummy-subtopic')]
+        topic.next_subtopic_id = 2
+        subtopic_page = (
+            subtopic_page_domain.SubtopicPage.create_default_subtopic_page(
+                subtopic_id, topic.id))
+        subtopic_page_services.save_subtopic_page(
+            self.owner_id, subtopic_page, 'Added subtopic',
+            [topic_domain.TopicChange({
+                'cmd': topic_domain.CMD_ADD_SUBTOPIC,
+                'subtopic_id': 1,
+                'title': 'Sample'
+            })]
+        )
+        topic_services.save_new_topic(self.owner_id, topic)
+        topic_services.publish_topic(topic.id, self.admin_id)
+
+        for skill_id in uncategorized_skill_ids:
+            self.save_new_skill(
+                skill_id, self.admin_id, description='skill_description')
+            topic_services.add_uncategorized_skill(
+                self.admin_id, topic.id, skill_id)
+
 
 class TranslatableTextHandlerTest(test_utils.GenericTestBase):
     """Unit test for the ContributionOpportunitiesHandler."""
diff --git a/core/storage/opportunity/gae_models_test.py b/core/storage/opportunity/gae_models_test.py
--- a/core/storage/opportunity/gae_models_test.py
+++ b/core/storage/opportunity/gae_models_test.py
@@ -224,7 +224,7 @@ def test_get_skill_opportunities_pagination(self) -> None:
         assert results is not None
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0].id, 'opportunity_id1')
-        self.assertFalse(more)
+        self.assertTrue(more)
         self.assertTrue(isinstance(cursor, python_utils.BASESTRING))
 
         results, cursor, more = (
@@ -244,7 +244,7 @@ def test_delete_all_skill_opportunities(self) -> None:
         # Ruling out the possibility of None for mypy type checking.
         assert results is not None
         self.assertEqual(len(results), 1)
-        self.assertFalse(more)
+        self.assertTrue(more)
 
         opportunity_models.SkillOpportunityModel.delete_all()