Repository: oppia/oppia

Issue:
Error on opening existing explorations on the test server
**Describe the bug**
On opening existing explorations on the test server, an error is obtained which says 
![image](https://user-images.githubusercontent.com/11898234/65528126-744aad80-df11-11e9-99a6-680c48a6ef7d.png)

(The exploration is [here](https://oppiatestserver.appspot.com/create/c0xv8d3Gt1A1#/improvements))

/cc @brianrodri looks serious? I tried on multiple existing explorations and it failed.

**To Reproduce**
Steps to reproduce the behavior:
 1. Go to [this exploration](https://oppiatestserver.appspot.com/create/c0xv8d3Gt1A1#/improvements)
 2. See the error displayed

**Expected behavior**
There should be no error messages.

**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: OS X
 - Browser Chrome
 - Version 77

Error on opening existing explorations on the test server
**Describe the bug**
On opening existing explorations on the test server, an error is obtained which says 
![image](https://user-images.githubusercontent.com/11898234/65528126-744aad80-df11-11e9-99a6-680c48a6ef7d.png)

(The exploration is [here](https://oppiatestserver.appspot.com/create/c0xv8d3Gt1A1#/improvements))

/cc @brianrodri looks serious? I tried on multiple existing explorations and it failed.

**To Reproduce**
Steps to reproduce the behavior:
 1. Go to [this exploration](https://oppiatestserver.appspot.com/create/c0xv8d3Gt1A1#/improvements)
 2. See the error displayed

**Expected behavior**
There should be no error messages.

**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: OS X
 - Browser Chrome
 - Version 77



Discussion:
@brianrodri could you PTAL at this? This is blocking the release of the improvements tab.
I tried reproducing this and I saw another error in the console due to missing require statement for capitalize filter. I am not sure if this issue is related to that but I have issued #7757 to fix the capitalize filter.

Here is a screenshot of the error I get:

<img width="1643" alt="Screenshot 2019-10-09 at 11 11 58 PM" src="https://user-images.githubusercontent.com/15226041/66506471-05fa1380-eaeb-11e9-820c-334fd4246df7.png">

Thanks!
I added some brief analysis in the doc [here](https://docs.google.com/document/d/1VarxAJNuzQA2FEKtvstbHT2mXLt2l7ngnT45Gfc5j-4/edit?usp=sharing).

@nithusha21 @DubeySandeep can you suggest on what can we do next? Do we need to add more logging info to debug this issue since the source of error is not clear?
Hi @kevintab95, can you please look into [this](https://docs.google.com/document/d/1VarxAJNuzQA2FEKtvstbHT2mXLt2l7ngnT45Gfc5j-4/edit?usp=sharing) analysis once if you are free and provide some inputs? Thanks!
@DubeySandeep further looked into this and checked the ExpIssuesModel on test server. The model is missing for some of the exp versions of old explorations. We will need to:

- Find a reason for how this happened and improve the saving pipeline for the models
- Fix this issue for the release by finding a way to regenerate these models.

@nithusha21 can you help us with the regeneration process since you have worked on similar pipeline earlier?

Thanks!
I looked into this issue and found the error is happening around this line:
https://github.com/oppia/oppia/blob/0eb964ebfe2f9446d24a2a60e76dcea11c2264c5/core/controllers/editor.py#L534-L538

**Reason:**
On further investigation, I found that the ExplorationIssueModel is missing for the updated version of exploration. I'm not sure how and why the model is missing but I saw a pattern that can cause this issue. In _svae_exploration method, we updated the exploration and connected model but the process is bit buggy. We follow the following pattern:
1. Update the exploration model instance.
2. Commit exploration model instance into datastore.
3. Update connected model instance (like stats model summary mode etc.)
4. Save connected models into datastore one at a time!

So even if by any chance the stats model updation failed the exploration will be in the updated stage. Although I'm not sure whether this is the cause of the issue but I think we should fix it.

**Long term solution:**
We should do the model update in the following steps: 1 --> 3 --> (2, 4) in one call.

**Quick solution to escalating the release process:**

I'm not sure what should we do as we don't know the actual cause of this issue, I think finding the reals cause will help us find the quick solution.

@nithusha21, it would be great if you can help us find the cause of this issue, asking you as you have worked on the stats related model recently and maybe you know what can cause this.   
   

Hi, I took a look at this and I am actually quite confused. So there are many exploration versions for which ExplorationIssuesModel is missing, for example, the exploration https://oppiatestserver.appspot.com/create/Kr_LjJHJ3Fu2 which Ankita mentions, has versions upto 7 but has issue models only for versions 5 and 6. Is it necessary for every exploration version to contain an issue model? If so what is missing in the complete flow of issue model creation which causes such a failure. 

I can confirm that the error is raised at line 538 which Sandeep mentions above (which is hit due to the missing model). @brianrodri @pranavsid98 could you PTAL and let us know what the expected behavior is?
Hi @brianrodri @pranavsid98 can you please help us with this issue since solving this will unblock the release. Thanks!
@brianrodri could you PTAL at this? This is blocking the release of the improvements tab.
I tried reproducing this and I saw another error in the console due to missing require statement for capitalize filter. I am not sure if this issue is related to that but I have issued #7757 to fix the capitalize filter.

Here is a screenshot of the error I get:

<img width="1643" alt="Screenshot 2019-10-09 at 11 11 58 PM" src="https://user-images.githubusercontent.com/15226041/66506471-05fa1380-eaeb-11e9-820c-334fd4246df7.png">

Thanks!
I added some brief analysis in the doc [here](https://docs.google.com/document/d/1VarxAJNuzQA2FEKtvstbHT2mXLt2l7ngnT45Gfc5j-4/edit?usp=sharing).

@nithusha21 @DubeySandeep can you suggest on what can we do next? Do we need to add more logging info to debug this issue since the source of error is not clear?
Hi @kevintab95, can you please look into [this](https://docs.google.com/document/d/1VarxAJNuzQA2FEKtvstbHT2mXLt2l7ngnT45Gfc5j-4/edit?usp=sharing) analysis once if you are free and provide some inputs? Thanks!
@DubeySandeep further looked into this and checked the ExpIssuesModel on test server. The model is missing for some of the exp versions of old explorations. We will need to:

- Find a reason for how this happened and improve the saving pipeline for the models
- Fix this issue for the release by finding a way to regenerate these models.

@nithusha21 can you help us with the regeneration process since you have worked on similar pipeline earlier?

Thanks!
I looked into this issue and found the error is happening around this line:
https://github.com/oppia/oppia/blob/0eb964ebfe2f9446d24a2a60e76dcea11c2264c5/core/controllers/editor.py#L534-L538

**Reason:**
On further investigation, I found that the ExplorationIssueModel is missing for the updated version of exploration. I'm not sure how and why the model is missing but I saw a pattern that can cause this issue. In _svae_exploration method, we updated the exploration and connected model but the process is bit buggy. We follow the following pattern:
1. Update the exploration model instance.
2. Commit exploration model instance into datastore.
3. Update connected model instance (like stats model summary mode etc.)
4. Save connected models into datastore one at a time!

So even if by any chance the stats model updation failed the exploration will be in the updated stage. Although I'm not sure whether this is the cause of the issue but I think we should fix it.

**Long term solution:**
We should do the model update in the following steps: 1 --> 3 --> (2, 4) in one call.

**Quick solution to escalating the release process:**

I'm not sure what should we do as we don't know the actual cause of this issue, I think finding the reals cause will help us find the quick solution.

@nithusha21, it would be great if you can help us find the cause of this issue, asking you as you have worked on the stats related model recently and maybe you know what can cause this.   
   

Hi, I took a look at this and I am actually quite confused. So there are many exploration versions for which ExplorationIssuesModel is missing, for example, the exploration https://oppiatestserver.appspot.com/create/Kr_LjJHJ3Fu2 which Ankita mentions, has versions upto 7 but has issue models only for versions 5 and 6. Is it necessary for every exploration version to contain an issue model? If so what is missing in the complete flow of issue model creation which causes such a failure. 

I can confirm that the error is raised at line 538 which Sandeep mentions above (which is hit due to the missing model). @brianrodri @pranavsid98 could you PTAL and let us know what the expected behavior is?
Hi @brianrodri @pranavsid98 can you please help us with this issue since solving this will unblock the release. Thanks!

Resolving patch:
diff --git a/core/domain/exp_services.py b/core/domain/exp_services.py
--- a/core/domain/exp_services.py
+++ b/core/domain/exp_services.py
@@ -59,6 +59,7 @@
 datastore_services = models.Registry.import_datastore_services()
 memcache_services = models.Registry.import_memcache_services()
 taskqueue_services = models.Registry.import_taskqueue_services()
+transaction_services = models.Registry.import_transaction_services()
 (exp_models, feedback_models, user_models) = models.Registry.import_models([
     models.NAMES.exploration, models.NAMES.feedback, models.NAMES.user
 ])
@@ -539,11 +540,8 @@ def _save_exploration(committer_id, exploration, commit_message, change_list):
         Exception: The versions of the given exploration and the currently
             stored exploration model do not match.
     """
-    exploration_rights = rights_manager.get_exploration_rights(exploration.id)
-    if exploration_rights.status != rights_manager.ACTIVITY_STATUS_PRIVATE:
-        exploration.validate(strict=True)
-    else:
-        exploration.validate()
+    exploration_is_public = rights_manager.is_exploration_public(exploration.id)
+    exploration.validate(strict=exploration_is_public)
 
     exploration_model = exp_models.ExplorationModel.get(exploration.id)
 
@@ -558,8 +556,8 @@ def _save_exploration(committer_id, exploration, commit_message, change_list):
             'which is too old. Please reload the page and try again.'
             % (exploration_model.version, exploration.version))
 
-    old_states = exp_fetchers.get_exploration_from_model(
-        exploration_model).states
+    old_states = (
+        exp_fetchers.get_exploration_from_model(exploration_model).states)
     exploration_model.category = exploration.category
     exploration_model.title = exploration.title
     exploration_model.objective = exploration.objective
@@ -579,29 +577,41 @@ def _save_exploration(committer_id, exploration, commit_message, change_list):
     exploration_model.correctness_feedback_enabled = (
         exploration.correctness_feedback_enabled)
 
-    change_list_dict = [change.to_dict() for change in change_list]
-    exploration_model.commit(committer_id, commit_message, change_list_dict)
-    exp_memcache_key = exp_fetchers.get_exploration_memcache_key(exploration.id)
-    memcache_services.delete(exp_memcache_key)
-
     exploration.version += 1
-
+    change_list_dict = [change.to_dict() for change in change_list]
     exp_versions_diff = exp_domain.ExplorationVersionsDiff(change_list)
 
-    # Trigger statistics model update.
     new_exp_stats = stats_services.get_stats_for_new_exp_version(
         exploration.id, exploration.version, exploration.states,
         exp_versions_diff=exp_versions_diff, revert_to_version=None)
 
-    stats_services.create_stats_model(new_exp_stats)
+    exp_memcache_key = exp_fetchers.get_exploration_memcache_key(exploration.id)
+
+    def _update_storage_models():
+        """Groups all storage calls into a function call so that they can be
+        performed as an atomic transaction.
+        """
+        exploration_model.commit(committer_id, commit_message, change_list_dict)
+        # Note: the memcache will not be reinstated if the transaction fails.
+        # However, it must be deleted before creating the exploration summary
+        # because otherwise the wrong version will be used.
+        memcache_services.delete(exp_memcache_key)
+
+        stats_services.create_stats_model(new_exp_stats)
+
+        stats_services.update_exp_issues_for_new_exp_version(
+            exploration, exp_versions_diff=exp_versions_diff,
+            revert_to_version=None)
+
+    transaction_services.run_in_transaction(_update_storage_models)
 
     if feconf.ENABLE_ML_CLASSIFIERS:
         trainable_states_dict = exploration.get_trainable_states_dict(
             old_states, exp_versions_diff)
-        state_names_with_changed_answer_groups = trainable_states_dict[
-            'state_names_with_changed_answer_groups']
-        state_names_with_unchanged_answer_groups = trainable_states_dict[
-            'state_names_with_unchanged_answer_groups']
+        state_names_with_changed_answer_groups = (
+            trainable_states_dict['state_names_with_changed_answer_groups'])
+        state_names_with_unchanged_answer_groups = (
+            trainable_states_dict['state_names_with_unchanged_answer_groups'])
         state_names_to_train_classifier = state_names_with_changed_answer_groups
         if state_names_with_unchanged_answer_groups:
             state_names_without_classifier = (
@@ -614,11 +624,6 @@ def _save_exploration(committer_id, exploration, commit_message, change_list):
             classifier_services.handle_trainable_states(
                 exploration, state_names_to_train_classifier)
 
-    # Trigger exploration issues model updation.
-    stats_services.update_exp_issues_for_new_exp_version(
-        exploration, exp_versions_diff=exp_versions_diff,
-        revert_to_version=None)
-
 
 def _create_exploration(
         committer_id, exploration, commit_message, commit_cmds):
@@ -635,12 +640,9 @@ def _create_exploration(
             changes made in this model, which should give sufficient information
             to reconstruct the commit.
     """
-    # This line is needed because otherwise a rights object will be created,
-    # but the creation of an exploration object will fail.
     exploration.validate()
-    rights_manager.create_new_exploration_rights(exploration.id, committer_id)
 
-    model = exp_models.ExplorationModel(
+    exploration_model = exp_models.ExplorationModel(
         id=exploration.id,
         category=exploration.category,
         title=exploration.title,
@@ -657,16 +659,32 @@ def _create_exploration(
         param_specs=exploration.param_specs_dict,
         param_changes=exploration.param_change_dicts,
         auto_tts_enabled=exploration.auto_tts_enabled,
-        correctness_feedback_enabled=exploration.correctness_feedback_enabled
-    )
-    commit_cmds_dict = [commit_cmd.to_dict() for commit_cmd in commit_cmds]
-    model.commit(committer_id, commit_message, commit_cmds_dict)
+        correctness_feedback_enabled=exploration.correctness_feedback_enabled)
+
     exploration.version += 1
+    commit_cmds_dict = [commit_cmd.to_dict() for commit_cmd in commit_cmds]
 
-    # Trigger statistics model creation.
-    exploration_stats = stats_services.get_stats_for_new_exploration(
+    new_exp_stats = stats_services.get_stats_for_new_exploration(
         exploration.id, exploration.version, exploration.states)
-    stats_services.create_stats_model(exploration_stats)
+
+    def _update_storage_models():
+        """Groups all storage calls into a function call so that they can be
+        performed as an atomic transaction.
+        """
+        # Rights to the exploration need to be created before the actual
+        # exploration.
+        rights_manager.create_new_exploration_rights(
+            exploration.id, committer_id)
+
+        exploration_model.commit(committer_id, commit_message, commit_cmds_dict)
+        create_exploration_summary(exploration.id, committer_id)
+
+        stats_services.create_stats_model(new_exp_stats)
+
+        stats_services.create_exp_issues_for_new_exploration(
+            exploration.id, exploration.version)
+
+    transaction_services.run_in_transaction(_update_storage_models)
 
     if feconf.ENABLE_ML_CLASSIFIERS:
         # Find out all states that need a classifier to be trained.
@@ -680,12 +698,6 @@ def _create_exploration(
             classifier_services.handle_trainable_states(
                 exploration, state_names_to_train)
 
-    # Trigger exploration issues model creation.
-    stats_services.create_exp_issues_for_new_exploration(
-        exploration.id, exploration.version)
-
-    create_exploration_summary(exploration.id, committer_id)
-
 
 def save_new_exploration(committer_id, exploration):
     """Saves a newly created exploration.
@@ -1156,8 +1168,8 @@ def revert_exploration(
         Exception:  does not match the version of the currently-stored
             exploration model.
     """
-    exploration_model = exp_models.ExplorationModel.get(
-        exploration_id, strict=False)
+    exploration_model = (
+        exp_models.ExplorationModel.get(exploration_id, strict=False))
 
     if current_version > exploration_model.version:
         raise Exception(
@@ -1172,41 +1184,51 @@ def revert_exploration(
 
     # Validate the previous version of the exploration before committing the
     # change.
-    exploration = exp_fetchers.get_exploration_by_id(
+    exploration_is_public = rights_manager.is_exploration_public(exploration_id)
+    exploration_to_revert_to = exp_fetchers.get_exploration_by_id(
         exploration_id, version=revert_to_version)
-    exploration_rights = rights_manager.get_exploration_rights(exploration.id)
-    if exploration_rights.status != rights_manager.ACTIVITY_STATUS_PRIVATE:
-        exploration.validate(strict=True)
-    else:
-        exploration.validate()
+    exploration_to_revert_to.validate(strict=exploration_is_public)
 
-    exp_models.ExplorationModel.revert(
-        exploration_model, committer_id,
-        'Reverted exploration to version %s' % revert_to_version,
-        revert_to_version)
-    exp_memcache_key = exp_fetchers.get_exploration_memcache_key(exploration.id)
-    memcache_services.delete(exp_memcache_key)
+    exploration_at_current_version = exp_fetchers.get_exploration_by_id(
+        exploration_id, version=current_version)
 
-    # Update the exploration summary, but since this is just a revert do
-    # not add the committer of the revert to the list of contributors.
-    update_exploration_summary(exploration_id, None)
+    exp_memcache_key = exp_fetchers.get_exploration_memcache_key(exploration_id)
 
-    exploration_stats = stats_services.get_stats_for_new_exp_version(
-        exploration.id, current_version + 1, exploration.states,
+    new_exp_stats = stats_services.get_stats_for_new_exp_version(
+        exploration_id, current_version + 1,
+        exploration_to_revert_to.states,
         exp_versions_diff=None, revert_to_version=revert_to_version)
-    stats_services.create_stats_model(exploration_stats)
 
-    current_exploration = exp_fetchers.get_exploration_by_id(
-        exploration_id, version=current_version)
-    stats_services.update_exp_issues_for_new_exp_version(
-        current_exploration, exp_versions_diff=None,
-        revert_to_version=revert_to_version)
+    def _update_storage_models():
+        """Groups all storage calls into a function call so that they can be
+        performed as an atomic transaction.
+        """
+        exp_models.ExplorationModel.revert(
+            exploration_model, committer_id,
+            'Reverted exploration to version %s' % revert_to_version,
+            revert_to_version)
+
+        # Note: the memcache will not be reinstated if the transaction fails.
+        # However, it must be deleted before creating the exploration summary
+        # because otherwise the wrong version will be used.
+        memcache_services.delete(exp_memcache_key)
+
+        # Update the exploration summary, but since this is just a revert so
+        # not add the committer of the revert to the list of contributors.
+        update_exploration_summary(exploration_id, None)
+
+        stats_services.create_stats_model(new_exp_stats)
+
+        stats_services.update_exp_issues_for_new_exp_version(
+            exploration_at_current_version, exp_versions_diff=None,
+            revert_to_version=revert_to_version)
+
+    transaction_services.run_in_transaction(_update_storage_models)
 
     if feconf.ENABLE_ML_CLASSIFIERS:
-        exploration_to_revert_to = exp_fetchers.get_exploration_by_id(
-            exploration_id, version=revert_to_version)
-        classifier_services.create_classifier_training_job_for_reverted_exploration( # pylint: disable=line-too-long
-            current_exploration, exploration_to_revert_to)
+        (classifier_services
+         .create_classifier_training_job_for_reverted_exploration(
+             exploration_at_current_version, exploration_to_revert_to))
 
 
 # Creation and deletion methods.


Test patch:
diff --git a/core/domain/exp_services_test.py b/core/domain/exp_services_test.py
--- a/core/domain/exp_services_test.py
+++ b/core/domain/exp_services_test.py
@@ -17,6 +17,7 @@
 """Unit tests for core.domain.exp_services."""
 
 from __future__ import absolute_import  # pylint: disable=import-only-modules
+from __future__ import division  # pylint: disable=import-only-modules
 from __future__ import unicode_literals  # pylint: disable=import-only-modules
 
 import datetime
@@ -36,6 +37,7 @@
 from core.domain import rights_manager
 from core.domain import search_services
 from core.domain import state_domain
+from core.domain import stats_services
 from core.domain import subscription_services
 from core.domain import user_services
 from core.platform import models
@@ -47,6 +49,7 @@
 (exp_models, user_models) = models.Registry.import_models([
     models.NAMES.exploration, models.NAMES.user])
 gae_image_services = models.Registry.import_gae_image_services()
+memcache_services = models.Registry.import_memcache_services()
 search_services = models.Registry.import_search_services()
 transaction_services = models.Registry.import_transaction_services()
 
@@ -3881,6 +3884,55 @@ def test_update_exploration_with_invalid_commit_message(self):
                     'new_value': 'new title'
                 })], feconf.COMMIT_MESSAGE_ACCEPTED_SUGGESTION_PREFIX)
 
+    def test_update_exploration_does_nothing_if_create_stats_model_fails(self):
+        swap_create_stats_model = self.swap(
+            stats_services, 'create_stats_model',
+            lambda *_: python_utils.divide(1, 0))
+        assert_raises_regexp = (
+            self.assertRaisesRegexp(Exception, 'division or modulo by zero'))
+
+        self.save_new_valid_exploration('exp_id', 'user_id')
+        with swap_create_stats_model, assert_raises_regexp:
+            exp_services.update_exploration(
+                'user_id', 'exp_id', [exp_domain.ExplorationChange({
+                    'cmd': exp_domain.CMD_EDIT_EXPLORATION_PROPERTY,
+                    'property_name': 'title',
+                    'new_value': 'New title'})],
+                'Changed language code.')
+
+        self.assertIsNone(
+            exp_fetchers.get_exploration_by_id(
+                'exp_id', strict=False, version=2))
+
+    def test_memcache_does_not_persist_even_if_create_stats_model_fails(self):
+        self.save_new_valid_exploration('exp_id', 'user_id')
+
+        memcache_key = exp_fetchers.get_exploration_memcache_key('exp_id')
+        memcached_exploration = (
+            memcache_services.get_multi([memcache_key]).get(memcache_key))
+        actual_exploration = exp_fetchers.get_exploration_by_id('exp_id')
+        self.assertEqual(
+            memcached_exploration.to_dict(), actual_exploration.to_dict())
+
+        swap_create_stats_model = self.swap(
+            stats_services, 'create_stats_model',
+            lambda *_: python_utils.divide(1, 0))
+        assert_raises_regexp = (
+            self.assertRaisesRegexp(Exception, 'division or modulo by zero'))
+
+        with swap_create_stats_model, assert_raises_regexp:
+            exp_services.update_exploration(
+                'user_id', 'exp_id', [exp_domain.ExplorationChange({
+                    'cmd': exp_domain.CMD_EDIT_EXPLORATION_PROPERTY,
+                    'property_name': 'title',
+                    'new_value': 'New title'})],
+                'Changed language code.')
+
+        memcache_key = exp_fetchers.get_exploration_memcache_key('exp_id')
+        memcached_exploration = (
+            memcache_services.get_multi([memcache_key]).get(memcache_key))
+        self.assertIsNone(memcached_exploration)
+
     def test_update_language_code(self):
         exploration = exp_fetchers.get_exploration_by_id(self.NEW_EXP_ID)
         self.assertEqual(exploration.language_code, 'en')
@@ -4179,6 +4231,26 @@ def test_revert_exploration_with_mismatch_of_versions_raises_error(self):
             'from version 1. Please reload the page and try again.'):
             exp_services.revert_exploration('user_id', 'exp_id', 1, 0)
 
+    def test_revert_exploration_does_nothing_if_create_stats_model_fails(self):
+        swap_create_stats_model = self.swap(
+            stats_services, 'create_stats_model',
+            lambda *_: python_utils.divide(1, 0))
+        assert_raises_regexp = (
+            self.assertRaisesRegexp(Exception, 'division or modulo by zero'))
+
+        self.save_new_valid_exploration('exp_id', 'user_id')
+        exp_services.update_exploration(
+            'user_id', 'exp_id', [exp_domain.ExplorationChange({
+                'cmd': exp_domain.CMD_EDIT_EXPLORATION_PROPERTY,
+                'property_name': 'title',
+                'new_value': 'New title'})], 'Changed interaction_solutions.')
+        with swap_create_stats_model, assert_raises_regexp:
+            exp_services.revert_exploration('user_id', 'exp_id', 2, 1)
+
+        self.assertIsNone(
+            exp_fetchers.get_exploration_by_id(
+                'exp_id', strict=False, version=3))
+
 
 class EditorAutoSavingUnitTests(test_utils.GenericTestBase):
     """Test editor auto saving functions in exp_services."""