Repository: oppia/oppia

Issue:
Server error: AttributeError: 'NoneType' object has no attribute 'num_starts_v2'
This error is being repeatedly logged on stackdriver. It has occurred over 3500 times in the span on 24 hours. This looks like a serious bug, and would need to be handled ASAP. 

```
AttributeError: 'NoneType' object has no attribute 'num_starts_v2'
at update_stats (/core/domain/stats_services.py:127)
```

/cc @brianrodri could you take a look? 

**General instructions**
There are no specific repro steps available for this bug report. The general procedure to fix server errors should be the following:

* Analyze the code in the file where the error occurred and come up with a hypothesis for the reason.
* Get the logic of the proposed fix validated by an Oppia team member (have this discussion on the issue thread).
* Make a PR that fixes the issue, then close the issue on merging the PR. (If the error reoccurs in production, the issue will be reopened for further investigation.)


Discussion:
Hi @nithusha21, @brianrodri as the function ```update_stats```mentioned is 
```
def update_stats(exp_id, exp_version, aggregated_stats):
    """Updates ExplorationStatsModel according to the dict containing aggregated
    stats.

    Args:
        exp_id: str. ID of the exploration.
        exp_version: int. Version of the exploration.
        aggregated_stats: dict. Dict representing an ExplorationStatsModel
            instance with stats aggregated in the frontend.
    """
    exploration_stats = get_exploration_stats_by_id(
        exp_id, exp_version)

    exploration_stats.num_starts_v2 += aggregated_stats['num_starts']
    exploration_stats.num_completions_v2 += aggregated_stats['num_completions']
    exploration_stats.num_actual_starts_v2 += aggregated_stats[
        'num_actual_starts']
```
I think there is no check if the exploration_stats is None, so that's why it is giving an error on line no. 127.
This event occurs after url _/explorehandler/stats_events/<exploration_id>_ is hit. But when I try to hit the page with curl, another error is occurring _{"status_code": 500, "error": "'NoneType' object has no attribute 'get'"}_. Probably I need to hit the page with more consideration.

Looking at _get_exploration_stats_by_id_ I suspect that there can be no ExplorationStatsModel for given exp_id and exp_version. I don't know how to check it or if it's viable hypothesis.
One thing we could do is to write a one-off MapReduce validation job that iterates through all exploration IDs and tries to fetch all stats models for versions from 1 to the latest version. If any is missing, it prints out an error with the missing model IDs and versions. We can run the job and look to see what the errors are, to confirm/deny the hypothesis. (Note that the MR job would not write to the datastore in any way, it would just read it.)

Does this sound good, and if so, would you like to take a crack at it? See core/domain/stats_jobs_one_off.py for details/examples and [this page](https://github.com/oppia/oppia/wiki/Running-jobs-in-production) for some general process notes.

/cc @sshou14 
Hi @rafalk342, would you like to take this up, if yes feel free to assign you. Thanks!
Hi @rafalk342, ping!
I investigated this with Sean's help. So the error is in the way stats models are handled when exploration reverts happen. Say we are in a situation where we are at version 10 and we want to revert to version 5. While reverting https://github.com/oppia/oppia/blob/develop/core/domain/exp_services.py#L1418 is called, which passes in exploration.version (which is 10), and revert_to_version (which is 5). Now in https://github.com/oppia/oppia/blob/develop/core/domain/stats_services.py#L190, new_version is set as 10, and old_version is set as 9. (It should ideally have been 11 and 10, as the new version post revert is 11). Now, the stats model for v10 of the exploration would be rewritten, and the model for state 11 is missing. This causes these errors

As a plan of action I would suggest: 
1. Add a check in update_stats to make sure that the statsModel actually exists, before applying any operation on it (This is a quick fix, to clear up our server load). 
2. Now, regenerate all stats models, as it currently has some corrupt data. Fix the revert function to work correctly. 
3. Remove the check in update_stats as the error shouldn't happen again. 
Update: I chatted with @nithusha21 about this and we decided that it would be more robust to fix the corrupt data once and for all, without adding the check to silence the error. It's too late for this change to make the April release but we should target a fix for the May release.
Just a question came to my mind.
If we currently have version x and revert to version y. Is the behavior correct to revert the stats also to version y? I would expect the stats to persist from version x to version x+1.

As for the idea to regenerate data.
I see there is stats_jobs_one_off.py. So there is a possibility to retrieve all exploration and for every exploration check if some version of stats is missing. If stats version x is missing check if exploration version x was created by reverting (if not there can be other problem in the code). One we know which stats version is missing and from which exploration version y to get the stats (we can get that information from commit message in version x) just create appropriate stats as in the code visible here https://github.com/oppia/oppia/blob/26891d88fa21c0832500c5c04bd3f7bf82843afb/core/domain/stats_services.py#L199-L214
That's a good question and I'm not entirely sure.

What you say sounds reasonable, but there is the possibility of state names having changed from one exploration to another. So if you use the new stats then when reverting, there may be some state names in the new stats that don't correspond to anything in the older version of the exploration. That needs to be handled carefully. Do you have any thoughts on this?

For the regeneration idea, what you say sounds sensible to me. We can write an audit job to list the missing versions and their commit types, confirm that all of these are reverts, and then fix as you mentioned.
I think you are right that new state names are the reason why we should revert to earlier version of stats.
Hi just checking, what things remain to be done for this issue and what is the progress on those things?
> Hi just checking, what things remain to be done for this issue and what is the progress on those things?

Done:
- fixed the bug in the code

To do:
- regenerate missing data
- check if there are still these errors on the production
Hi @nithusha21 and @rafalk342, just checking in -- what progress has been made wrt this issue? Thanks! 

Resolving patch:
diff --git a/core/domain/stats_jobs_one_off.py b/core/domain/stats_jobs_one_off.py
--- a/core/domain/stats_jobs_one_off.py
+++ b/core/domain/stats_jobs_one_off.py
@@ -1275,3 +1275,83 @@ def reduce(key, stringified_values):
                     'all: %s sum:%s' % (
                         key, state_name, all_state_hit[state_name],
                         sum_state_hit[state_name]),)
+
+
+class RegenerateMissingStatsModels(jobs.BaseMapReduceOneOffJobManager):
+    """A one-off job to regenerate stats models which were missing due to
+    incorrect handling of exploration reverts. If a model is missing at version
+    x, we will regenerate all models from version x-1 till the max version of
+    the exploration.
+    """
+
+    @classmethod
+    def entity_classes_to_map_over(cls):
+        return [exp_models.ExplorationModel]
+
+    @staticmethod
+    def map(exp):
+        all_models = (
+            stats_models.ExplorationStatsModel.get_multi_stats_models(
+                [exp_domain.ExpVersionReference(exp.id, version)
+                 for version in range(1, exp.version + 1)]))
+        first_missing_version = None
+        for version, model in enumerate(all_models):
+            if model is None:
+                first_missing_version = version + 1
+                break
+
+        if first_missing_version is None:
+            yield ('No change', exp.id)
+            return
+
+        if first_missing_version == 1:
+            yield ('Missing model at version 1', exp.id)
+            return
+
+        for version in range(first_missing_version - 1, exp.version + 1):
+            commit_log = exp_models.ExplorationCommitLogEntryModel.get_commit(
+                exp.id, version)
+            exp_at_version = exp_models.ExplorationModel.get_version(
+                exp.id, version)
+            # Delete all old models.
+            if all_models[version - 1] is None:
+                if commit_log.commit_type != 'revert':
+                    yield ('Missing model without revert commit', exp.id)
+                    return
+
+                # Is a revert commit.
+                revert_to_version = commit_log.commit_cmds[0]['version_number']
+                try:
+                    stats_services.handle_stats_creation_for_new_exp_version(
+                        exp.id, version, exp_at_version.states, None,
+                        revert_to_version)
+                except Exception:
+                    yield (
+                        'Failed to create stats for exp at revert commit',
+                        {'exp_id': exp.id, 'version': version})
+                    return
+            else:
+                all_models[version - 1].delete()
+                change_list = (
+                    [exp_domain.ExplorationChange(commit_cmd)
+                     for commit_cmd in commit_log.commit_cmds])
+                exp_versions_diff = exp_domain.ExplorationVersionsDiff(
+                    change_list)
+                try:
+                    stats_services.handle_stats_creation_for_new_exp_version(
+                        exp.id, version, exp_at_version.states,
+                        exp_versions_diff, None)
+                except Exception:
+                    yield (
+                        'Failed to create stats for exp at non-revert commit',
+                        {'exp_id': exp.id, 'version': version})
+                    return
+
+        yield ('Success', exp.id)
+
+    @staticmethod
+    def reduce(key, items):
+        if key in ('Success', 'No change'):
+            yield (key, len(items))
+        else:
+            yield (key, items)


Test patch:
diff --git a/core/domain/stats_jobs_one_off_test.py b/core/domain/stats_jobs_one_off_test.py
--- a/core/domain/stats_jobs_one_off_test.py
+++ b/core/domain/stats_jobs_one_off_test.py
@@ -896,3 +896,96 @@ def test_standard_operation(self):
             stats_models.ExplorationStatsModel.get_entity_id(self.EXP_ID, 2))
         model = stats_models.ExplorationStatsModel.get(model_id)
         self.assertEqual(model.num_completions_v2, 3)
+
+
+class TestRegenerateMissingStatsModels(OneOffJobTestBase):
+    """Tests the regeneration of missing stats models."""
+    ONE_OFF_JOB_CLASS = stats_jobs_one_off.RegenerateMissingStatsModels
+
+    def setUp(self):
+        super(TestRegenerateMissingStatsModels, self).setUp()
+        self.EXP_ID = 'EXP_ID'
+        self.exp = self.save_new_valid_exploration(self.EXP_ID, 'owner_id')
+        self.state_name = self.exp.init_state_name
+
+        change_list = []
+        exp_services.update_exploration(
+            feconf.SYSTEM_COMMITTER_ID, self.EXP_ID, change_list, '')
+        change_list = [
+            exp_domain.ExplorationChange({
+                'cmd': exp_domain.CMD_RENAME_STATE,
+                'old_state_name': self.state_name,
+                'new_state_name': 'b',
+            })
+        ]
+        exp_services.update_exploration(
+            feconf.SYSTEM_COMMITTER_ID, self.EXP_ID, change_list, '')
+
+        exp_services.revert_exploration(
+            feconf.SYSTEM_COMMITTER_ID, self.EXP_ID, 3, 2)
+
+        change_list = [
+            exp_domain.ExplorationChange({
+                'cmd': exp_domain.CMD_RENAME_STATE,
+                'old_state_name': self.state_name,
+                'new_state_name': 'b',
+            })
+        ]
+        exp_services.update_exploration(
+            feconf.SYSTEM_COMMITTER_ID, self.EXP_ID, change_list, '')
+
+    def test_job_successfully_regenerates_deleted_model(self):
+        self.exp = exp_services.get_exploration_by_id(self.EXP_ID)
+        self.assertEqual(self.exp.version, 5)
+
+        model_id = stats_models.ExplorationStatsModel.get_entity_id(
+            self.EXP_ID, 4)
+        model = stats_models.ExplorationStatsModel.get(model_id)
+        model.delete()
+
+        output = self.run_one_off_job()
+        model_id = stats_models.ExplorationStatsModel.get_entity_id(
+            self.EXP_ID, 4)
+        model = stats_models.ExplorationStatsModel.get(model_id)
+        self.assertEqual(output, [u'[u\'Success\', 1]'])
+
+    def test_job_yields_correct_message_when_missing_model_at_version_1(self):
+        self.exp = exp_services.get_exploration_by_id(self.EXP_ID)
+        self.assertEqual(self.exp.version, 5)
+
+        model_id = stats_models.ExplorationStatsModel.get_entity_id(
+            self.EXP_ID, 1)
+        model = stats_models.ExplorationStatsModel.get(model_id)
+        model.delete()
+        model_id = stats_models.ExplorationStatsModel.get_entity_id(
+            self.EXP_ID, 2)
+        model = stats_models.ExplorationStatsModel.get(model_id)
+        model.delete()
+
+        output = self.run_one_off_job()
+        self.assertEqual(
+            output, [u'[u\'Missing model at version 1\', [u\''
+                     + self.EXP_ID + '\']]'])
+
+    def test_job_yields_no_change_when_no_regeneration_is_needed(self):
+        self.exp = exp_services.get_exploration_by_id(self.EXP_ID)
+        self.assertEqual(self.exp.version, 5)
+
+        output = self.run_one_off_job()
+        self.assertEqual(output, [u'[u\'No change\', 1]'])
+
+    def test_job_yields_correct_message_when_missing_model_at_non_revert_commit(
+            self):
+        self.exp = exp_services.get_exploration_by_id(self.EXP_ID)
+        self.assertEqual(self.exp.version, 5)
+
+        model_id = stats_models.ExplorationStatsModel.get_entity_id(
+            self.EXP_ID, 2)
+        model = stats_models.ExplorationStatsModel.get(model_id)
+        model.delete()
+
+        output = self.run_one_off_job()
+        self.assertEqual(
+            output,
+            [u'[u\'Missing model without revert commit\', [u\''
+             + self.EXP_ID + '\']]'])