mozilla/socorro — pull request #1681 by AdrianGaudebert (2013-11-19 14:22:08+00:00)
Title: Fixes bug 901977 - Store raw crash data into elasticsearch.

@twobraids r?

This is now a lot bigger than before. The ``crashstorage`` module indexes crashes like this:

```json
{
"crash_id": "xxx-xxx-xxx",
"processed_crash": {
"boo": "far"
},
"raw_crash": {
"foo": "bar"
}
}
```

All elasticsearch based services (Search and SuperSearch) have been changed to use the correct prefix for their fields (``processed_crash.``). Unit tests have been fixed accordingly. As the ``save_processed`` function of the ``crashstorage`` creates the same document structure (but with an empty ``raw_crash``), unit tests didn't need much work.

The backfill/reindex script I couldn't test locally for a lack of HBase, but I will test and time on stage where we have significant data. We should merge this right after a push so we have a full week to test the reindexing. Note that merging this will break search and supersearch until reindexing was done.

I have some ongoing work on the schema we use for elasticsearch that we will want to merge at the same time as this. And I am wondering when we will turn on the jdump, because when we do, we will certainly want to reindex as well, so we might want for that too.