dimagi/commcare-hq — pull request #5121 by snopoke (2014-12-18 16:43:21+00:00)
Title: restore performance improvements (part2)

2 main changes in this PR

## TL/DR
For a user with 500 cases:

*Only wrap if necessary*
1st restore = 172s (110s wrapping)
2nd restore (sync) using data from synclog = 23s (15s filter_cases_modified_elsewhere_since_sync)
2nd restore (sync) fetching data from couch = 23s (14s filter_cases_modified_elsewhere_since_sync)

*Wrap everything*
1st = 173s (112s fetch and wrap)
2nd = 133s (112s fetch and wrap)

## Don't wrap docs if you don't have to
The idea here is to do all the processing on raw cases and only wrap the ones that need to be sent down to the user right at the end.

Profiling this for a user with 500 cases, the full restore took 259 seconds to generate of which 167 seconds was spent wrapping the docs. A further improvement could be gained here if we didn't need to wrap the docs at all which requires changes to this: https://github.com/dimagi/commcare-hq/blob/master/corehq/ex-submodules/casexml/apps/phone/xml.py#L33

## Don't fetch docs from couch if you don't need to
This only works if the user is not part of any case sharing groups. The idea here is that all the calculations in the restore only require the case_id and case indices. Most of these (if not all) are already available in the SyncLog so just use those instead of fetching them from couch. At the end of the process when we know what cases we need to send to the use we fetch them.

This didn't produce as much improvement as I expected but I put that down to running locally where fetching from couch is fast.