Repository: oppia/oppia

Issue:
[Feature Request]: Fetch translation suggestion target IDs using GAE Projection Queries 
### Is your feature request related to a problem? Please describe.

This feature request's problem is related to issue #15948. In short, fetching reviewable translation opportunities can take a significant amount of time, which blocks translation reviewers from reviewing translations until they are fetched.

### Describe the solution (or solutions) you'd like

According to the [GAE docs](https://cloud.google.com/appengine/docs/legacy/standard/python/ndb/projectionqueries), projection queries allows the fetching of specific datastore fields instead of fetching entire models, which could save a significant portion of server response time. I would like the app to run a projection query to fetch only translation suggestions' target IDs, which is an important step towards fetching reviewable translation opportunities.

### Describe alternatives you've considered and rejected

Initially, I created a new field for `TopicSummaryModel` with the hopes of running a projection query to fetch it. However, I later found out that certain property types like `JSONProperty` could not be fetched using projection queries. I ended up rejecting the projection query idea for this new field. But I'm still trying to merge PR #18988 that introduces the new field, since being able to skip fetching of stories entirely means significantly less datastore queries are made. I don't want to introduce this feature into that PR because it has been opened for months and I would like to get the PR merged ASAP. 

### Additional context

_No response_
[Feature Request]: Fetch translation suggestion target IDs using GAE Projection Queries 
### Is your feature request related to a problem? Please describe.

This feature request's problem is related to issue #15948. In short, fetching reviewable translation opportunities can take a significant amount of time, which blocks translation reviewers from reviewing translations until they are fetched.

### Describe the solution (or solutions) you'd like

According to the [GAE docs](https://cloud.google.com/appengine/docs/legacy/standard/python/ndb/projectionqueries), projection queries allows the fetching of specific datastore fields instead of fetching entire models, which could save a significant portion of server response time. I would like the app to run a projection query to fetch only translation suggestions' target IDs, which is an important step towards fetching reviewable translation opportunities.

### Describe alternatives you've considered and rejected

Initially, I created a new field for `TopicSummaryModel` with the hopes of running a projection query to fetch it. However, I later found out that certain property types like `JSONProperty` could not be fetched using projection queries. I ended up rejecting the projection query idea for this new field. But I'm still trying to merge PR #18988 that introduces the new field, since being able to skip fetching of stories entirely means significantly less datastore queries are made. I don't want to introduce this feature into that PR because it has been opened for months and I would like to get the PR merged ASAP. 

### Additional context

_No response_


Discussion:
@seanlip When re-reading the projection query limitations, it says "Properties referenced in an equality (`=`) or membership (`IN`) filter cannot be projected." at https://cloud.google.com/appengine/docs/legacy/standard/python/ndb/projectionqueries#Limitations_on_projections

So if we're using projection queries to fetch all reviewable translation suggestions' `target_id`, trying to run the datastore query

```
SELECT target_id FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes AND
  target_id IN exp_ids
);
```

is impossible because of the `target_id IN exp_ids` condition in the `WHERE clause`. The next best query that is possible is to run 

```
SELECT target_id FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes
);
```

to fetch the `target_id` of every in-review translation suggestion for the given languages, and then within a backend service function, filter each `target_id` by whether its in a list of given `exp_ids`. Do you think this will be faster than running the following query (which is the current way of fetching reviewable translation suggestion's target_ids)?

```
SELECT * FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes AND
  target_id IN exp_ids
);
```
@seanlip When re-reading the projection query limitations, it says "Properties referenced in an equality (`=`) or membership (`IN`) filter cannot be projected." at https://cloud.google.com/appengine/docs/legacy/standard/python/ndb/projectionqueries#Limitations_on_projections

So if we're using projection queries to fetch all reviewable translation suggestions' `target_id`, trying to run the datastore query

```
SELECT target_id FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes AND
  target_id IN exp_ids
);
```

is impossible because of the `target_id IN exp_ids` condition in the `WHERE clause`. The next best query that is possible is to run 

```
SELECT target_id FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes
);
```

to fetch the `target_id` of every in-review translation suggestion for the given languages, and then within a backend service function, filter each `target_id` by whether its in a list of given `exp_ids`. Do you think this will be faster than running the following query (which is the current way of fetching reviewable translation suggestion's target_ids)?

```
SELECT * FROM GeneralSuggestionModel WHERE (
  status = 'in_review' AND
  suggestion_type = 'translation' AND
  language_code IN language_codes AND
  target_id IN exp_ids
);
```

Resolving patch:
diff --git a/core/controllers/contributor_dashboard.py b/core/controllers/contributor_dashboard.py
--- a/core/controllers/contributor_dashboard.py
+++ b/core/controllers/contributor_dashboard.py
@@ -378,34 +378,34 @@ def _get_reviewable_exploration_opportunity_summaries(
         pinned_opportunity_summary = None
         if topic_name is None:
             topic_exp_ids = (
-                topic_services.get_all_published_story_exploration_ids())
+                topic_services.get_all_published_story_exploration_ids()
+            )
         else:
             topic = topic_fetchers.get_topic_by_name(topic_name)
             if topic is None:
                 raise self.InvalidInputException(
-                    'The supplied input topic: %s is not valid' % topic_name)
-            topic_exp_ids = (
-                topic_services.get_all_published_story_exploration_ids(
-                    topic_id=topic.id))
+                    'The supplied input topic: %s is not valid' % topic_name
+                )
             if language and self.user_id:
                 pinned_opportunity_summary = (
                     opportunity_services.get_pinned_lesson(
                         self.user_id,
                         language,
                         topic.id
-                    ))
-
-        # TODO (#19664): Implement fetching target IDs using GAE projection
-        # queries.
-        in_review_suggestions, _ = (
+                    )
+                )
+            topic_exp_ids = (
+                topic_services.get_all_published_story_exploration_ids(
+                    topic_id=topic.id
+                )
+            )
+        in_review_suggestion_target_ids = (
             suggestion_services
-            .get_reviewable_translation_suggestions_by_offset(
-                user_id, topic_exp_ids, None, 0, None, language))
-        # This is defined as a set as we only care about the unique IDs.
-        in_review_suggestion_target_ids = {
-            suggestion.target_id for suggestion in in_review_suggestions
-        }
-        exp_ids = [
+            .get_reviewable_translation_suggestion_target_ids(
+                user_id, language
+            )
+        )
+        topic_exp_ids_targeted_by_in_review_suggestions = [
             exp_id
             for exp_id in topic_exp_ids
             if exp_id in in_review_suggestion_target_ids
@@ -413,7 +413,7 @@ def _get_reviewable_exploration_opportunity_summaries(
 
         exp_opp_summaries = (
             opportunity_services.get_exploration_opportunity_summaries_by_ids(
-                exp_ids
+                topic_exp_ids_targeted_by_in_review_suggestions
             )
         )
 
diff --git a/core/domain/suggestion_services.py b/core/domain/suggestion_services.py
--- a/core/domain/suggestion_services.py
+++ b/core/domain/suggestion_services.py
@@ -1103,6 +1103,50 @@ def get_reviewable_translation_suggestions_by_offset(
     return translation_suggestions, next_offset
 
 
+def get_reviewable_translation_suggestion_target_ids(
+    user_id: str,
+    language_code: Optional[str] = None
+) -> List[str]:
+    """Returns a list of translation suggestions matching the
+    passed opportunity IDs which the user can review.
+
+    Args:
+        user_id: str. The ID of the user.
+        language_code: str|None. ISO 639-1 language code for which to filter.
+            If it is None, all available languages will be returned.
+
+    Returns:
+        list(str). A list of translation suggestion target ids
+        which the supplied user is permitted to review.
+    """
+    contribution_rights = user_services.get_user_contribution_rights(
+        user_id
+    )
+    allowed_language_codes_for_review = (
+        contribution_rights.can_review_translation_for_language_codes
+    )
+
+    filtering_by_language_code = language_code is not None
+    language_codes = (
+        allowed_language_codes_for_review if not filtering_by_language_code
+        else [language_code]
+        if language_code in allowed_language_codes_for_review
+        else []
+    )
+
+    user_can_review_translations = len(language_codes) != 0
+    if not user_can_review_translations:
+        return []
+
+    return (
+        suggestion_models.GeneralSuggestionModel
+        .get_in_review_translation_suggestion_target_ids(
+            user_id,
+            language_codes
+        )
+    )
+
+
 def get_reviewable_translation_suggestions_for_single_exp(
     user_id: str,
     opportunity_summary_exp_id: str,
diff --git a/core/storage/suggestion/gae_models.py b/core/storage/suggestion/gae_models.py
--- a/core/storage/suggestion/gae_models.py
+++ b/core/storage/suggestion/gae_models.py
@@ -699,6 +699,40 @@ def get_in_review_translation_suggestions_by_offset(
             next_offset
         )
 
+    @classmethod
+    def get_in_review_translation_suggestion_target_ids(
+        cls,
+        user_id: str,
+        language_codes: List[str]
+    ) -> List[str]:
+        """Fetches all target ids of translation suggestion that are in-review
+        where the author_id != user_id and language_code matches one of the
+        supplied language_codes.
+
+        Args:
+            user_id: str. The id of the user trying to make this query. As a
+                user cannot review their own suggestions, suggestions authored
+                by the user will be excluded.
+            language_codes: list(str). List of language codes that the
+                suggestions should match.
+
+        Returns:
+            list(str). A list of target ids of the matching suggestions.
+        """
+        fetched_models: Sequence[
+            GeneralSuggestionModel
+        ] = cls.get_all().filter(datastore_services.all_of(
+            cls.status == STATUS_IN_REVIEW,
+            cls.suggestion_type == feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            cls.author_id != user_id,
+            cls.language_code.IN(language_codes)
+        )).fetch(
+            projection=[cls.target_id]
+        )
+
+        # We start with a set as we only care about unique target IDs.
+        return list({model.target_id for model in fetched_models})
+
     @classmethod
     def get_reviewable_translation_suggestions(
         cls,


Test patch:
diff --git a/core/domain/suggestion_services_test.py b/core/domain/suggestion_services_test.py
--- a/core/domain/suggestion_services_test.py
+++ b/core/domain/suggestion_services_test.py
@@ -1852,6 +1852,72 @@ def test_get_reviewable_translation_suggestions_with_language_filter(
         expected_language_code_list: List[str] = []
         self.assertEqual(actual_language_code_list, expected_language_code_list)
 
+    def test_get_target_ids_of_reviewable_translation_suggestions_for_user(
+        self
+    ) -> None:
+        language_code = 'hi'
+        fetched_target_id_1, fetched_target_id_2 = ('exp1', 'exp2')
+        self._create_translation_suggestion(language_code, fetched_target_id_1)
+        self._create_translation_suggestion(language_code, fetched_target_id_2)
+        self._create_translation_suggestion(language_code, fetched_target_id_2)
+        self._create_translation_suggestion('bn', 'exp3')
+        user_services.allow_user_to_review_translation_in_language(
+            self.reviewer_id_1, 'hi'
+        )
+
+        target_ids = (
+            suggestion_services.
+            get_reviewable_translation_suggestion_target_ids(
+                self.reviewer_id_1, language_code
+            )
+        )
+
+        self.assertCountEqual(
+            target_ids, [fetched_target_id_1, fetched_target_id_2]
+        )
+
+    def test_get_target_ids_of_translations_in_user_reviewable_languages_when_not_filtering_by_language( # pylint: disable=line-too-long
+        self
+    ) -> None:
+        fetched_target_id_1, fetched_target_id_2 = ('exp1', 'exp2')
+        self._create_translation_suggestion('hi', fetched_target_id_1)
+        self._create_translation_suggestion('fr', fetched_target_id_2)
+        user_services.allow_user_to_review_translation_in_language(
+            self.reviewer_id_1, 'hi'
+        )
+        user_services.allow_user_to_review_translation_in_language(
+            self.reviewer_id_1, 'fr'
+        )
+        language_code = None
+
+        target_ids = (
+            suggestion_services.
+            get_reviewable_translation_suggestion_target_ids(
+                self.reviewer_id_1, language_code
+            )
+        )
+
+        self.assertCountEqual(
+            target_ids, [fetched_target_id_1, fetched_target_id_2]
+        )
+
+    def test_get_no_translation_target_ids_when_user_cannot_review_in_given_language( # pylint: disable=line-too-long
+        self
+    ) -> None:
+        language_code = 'cs'
+        user_services.allow_user_to_review_translation_in_language(
+            self.reviewer_id_1, 'hi'
+        )
+
+        target_ids = (
+            suggestion_services.
+            get_reviewable_translation_suggestion_target_ids(
+                self.reviewer_id_1, language_code
+            )
+        )
+
+        self.assertCountEqual(target_ids, [])
+
     def test_get_reviewable_question_suggestions(self) -> None:
         # Add a few translation suggestions in different languages.
         self._create_translation_suggestion_with_language_code('hi')
diff --git a/core/storage/suggestion/gae_models_test.py b/core/storage/suggestion/gae_models_test.py
--- a/core/storage/suggestion/gae_models_test.py
+++ b/core/storage/suggestion/gae_models_test.py
@@ -869,6 +869,71 @@ def test_get_in_review_translation_suggestions_by_offset_sorted(
         self.assertEqual(sorted_results[1].id, suggestion_1_id)
         self.assertEqual(offset_4, 3)
 
+    def test_get_target_ids_of_translation_suggestions_in_review(self) -> None:
+        user_id = 'author1'
+        language_codes = [self.translation_language_code, 'fr']
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp1', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, 'author_2',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'matched_en_1', self.translation_language_code)
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp2', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, 'author_2',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'matched_en_2', self.translation_language_code)
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp2', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, 'author_3',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'matched_en_2_duplicate', self.translation_language_code)
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp3', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, 'author_3',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'matched_fr_1', 'fr')
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp4', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, 'author_3',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'not_matched_since_language_not_in_codes', 'na')
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp5', self.target_version_at_submission,
+            suggestion_models.STATUS_ACCEPTED, 'author_4',
+            'reviewer_2', self.change_cmd, self.score_category,
+            'not_matched_since_not_in_review',
+            self.translation_language_code)
+        suggestion_models.GeneralSuggestionModel.create(
+            feconf.SUGGESTION_TYPE_TRANSLATE_CONTENT,
+            feconf.ENTITY_TYPE_EXPLORATION,
+            'exp6', self.target_version_at_submission,
+            suggestion_models.STATUS_IN_REVIEW, user_id,
+            'reviewer_2', self.change_cmd, self.score_category,
+            'not_matched_since_reviewer_is_author',
+            self.translation_language_code)
+
+        target_ids = (
+            suggestion_models.GeneralSuggestionModel
+            .get_in_review_translation_suggestion_target_ids(
+                user_id,
+                language_codes
+            )
+        )
+
+        self.assertCountEqual(target_ids, ['exp1', 'exp2', 'exp3'])
+
     def test_get_in_review_question_suggestions_by_offset(self) -> None:
         suggestion_1_id = 'skill1.thread1'
         suggestion_2_id = 'skill1.thread2'