Repository: oppia/oppia

Issue:
ExplorationStateIdMappingJob fails in production
The ExplorationStateIdMappingJob exhibits this error in production when run; we need to debug it:

```
Traceback (most recent call last): (/third_party/gae-mapreduce-1.9.17.0/mapreduce/handlers.py:508)
  File "/third_party/gae-mapreduce-1.9.17.0/mapreduce/handlers.py", line 494, in handle
    tstate.input_reader, shard_state, tstate, ctx)
  File "/third_party/gae-mapreduce-1.9.17.0/mapreduce/handlers.py", line 583, in _process_inputs
    entity, input_reader, ctx, tstate):
  File "/third_party/gae-mapreduce-1.9.17.0/mapreduce/handlers.py", line 634, in _process_datum
    output_writer.write(output)
  File "/third_party/gae-mapreduce-1.9.17.0/mapreduce/output_writers.py", line 1145, in write
    if len(data) != 2:
TypeError: object of type 'exceptions.Exception' has no len()
```

/cc @BenHenning 


Discussion:
@seanlip I think the error is being raised by the state id map generating functions because of some issue with commit_cmds of some exploration. If commit_cmds are not properly synchronized with exploration versions then it will create the problem. I can't find any other reason for it to happen. 
Chatted with @BenHenning.

- Noted that the `exploration` variable on line 365 is shadowed by the one on line 394, maybe that's relevant.
- @BenHenning suggests removing the yields and reintroducing them one at a time until the error presents, since this is something to do with the stuff yielded by map().
Hi, @seanlip if the current release can go ahead without this Job then feel free to do so. This job can go in next month, too, since I won't be working further on this part until after February.

Also, I don't think that error is because of stuff yielded from map() because all the yields are tuple objects and they have len() == 2. I think something is raising the exception in the map() function. I think that we should cover the body of the for loop (the part where functions generate state id map model for given exploration version) in try ... except block. I think those functions are generating exception.
Hi @prasanna08 -- thanks. We'll drop this job from the current release, then.

And what you suggest sounds fine -- feel free to make changes when you get a chance, and we can re-run in production! Thanks.
@prasanna08 any updates on this?
Hi @jacobdavis11, I was busy with my exams. I will fix this soon. Thanks
Hi @seanlip, last time this job raised Exception when running in production and we were not able to idenfity exact reason for it. I am also not sure why it is raising Exception. If I remember correctly @pranavsid98's stats job also had some issues in production. So how were those issues resolved? Maybe that will give us more idea about this Exception as well. Thanks.
Hm, I think you'll need to ask @pranavsid98. There were a bunch of issues, but I'm not sure how many of them relate to this PR. Also, I think @pranavsid98 is writing up a doc with his findings from the last study (about the issues with prod data) that might be helpful here, so that's another good reason to talk to him!
@prasanna08 I think I remember this issue. Are you yielding an Exception directly? If you are, it has to be something like this - yield(str(Exception)). 
@pranavsid98 that was happening before but then I fixed that issue. This error occurs after fixing that issue. I think this is happening because one of the explorations in production data does not have valid change_dict or something which would cause exception when generating state_id_map fo that exploration. So are there any illegal explorations in production (as in explorations that do not follow the rules they must follow, maybe because they were created before these rules?). Anyway, I think we can find out the cause by wrapping [this](https://github.com/oppia/oppia/blob/develop/core/domain/exp_jobs_one_off.py#L405-L412) portion of code in `try... except...` block and handle exceptions. However, I can't be sure until the job is run in production once again.
@prasanna08 A possible cause is line 365 in exp_jobs_one_off. When you execute get_explration_from_model, the complete schema migration is done, and there are a few explorations in prod which fails the schema migration. I handled it by adding a try-block around the retrieve command. The explorations that show these migration errors are usually the ones with redundant interactions and the like. During stats migration, we skipped such explorations.
@pranavsid98 I see. If that is the case then I will also do the same. Thanks.

Also @seanlip, by any chance would it be possible to see the message of the Exception raised in the production? This will be helpful in tracing the exception.

Resolving patch:
diff --git a/core/domain/exp_jobs_one_off.py b/core/domain/exp_jobs_one_off.py
--- a/core/domain/exp_jobs_one_off.py
+++ b/core/domain/exp_jobs_one_off.py
@@ -362,7 +362,12 @@ def map(item):
         if item.deleted:
             return
 
-        exploration = exp_services.get_exploration_from_model(item)
+        try:
+            exploration = exp_services.get_exploration_from_model(item)
+        except Exception as e:
+            yield ('ERROR with exp_id %s' % item.id, str(e))
+            return
+
         explorations = []
 
         # Fetch all versions of the exploration, if they exist.
@@ -401,15 +406,23 @@ def map(item):
                 return
 
             change_list = snapshot['commit_cmds']
-            # Check if commit is to revert the exploration.
-            if change_list and change_list[0]['cmd'].endswith(
-                    'revert_version_number'):
-                reverted_version = change_list[0]['version_number']
-                exp_services.create_and_save_state_id_mapping_model_for_reverted_exploration( # pylint: disable=line-too-long
-                    exploration.id, exploration.version - 1, reverted_version)
-            else:
-                exp_services.create_and_save_state_id_mapping_model(
-                    exploration, change_list)
+
+            try:
+                # Check if commit is to revert the exploration.
+                if change_list and change_list[0]['cmd'].endswith(
+                        'revert_version_number'):
+                    reverted_version = change_list[0]['version_number']
+                    # pylint: disable=line-too-long
+                    exp_services.create_and_save_state_id_mapping_model_for_reverted_exploration(
+                        exploration.id, exploration.version - 1, reverted_version)
+                    # pylint: enable=line-too-long
+                else:
+                    exp_services.create_and_save_state_id_mapping_model(
+                        exploration, change_list)
+            except Exception as e:
+                yield ('ERROR with exp_id %s' % item.id, str(e))
+                return
+
         yield (exploration.id, exploration.version)
 
     @staticmethod


Test patch: