ornicar/OrnicarMessageBundle — pull request #32 by jmikola (2011-12-13 04:02:19+00:00)
Title: Refactoring Message and Thread documents

This is currently a work in progress, but I wanted to open the PR early so you have adequate time to review it.

About two weeks ago, we started experiencing scalability problems with the messaging bundle. With some help from 10gen, we were able to identity poorly-performing queries, mainly caused by the hash fields on thread and message documents. Despite having indexes defined, queries on the dynamic keys (user ID's) were quite slow. 10gen explained that if a hash field is indexed, the query must use the full key/value pair in order to take advantage of the index.

To refactor this, I converted those hashes into arrays of embedded metadata objects. I extracted a root metadata class from the entity models to prevent duplication. Apart from the models themselves, the other big changes were to the document denormalization methods and respective manager classes.

The main sacrifice in BC here is sorting the index by last other-participant message date, and sorting the sent list by last participant message date. The original sorting done on the user-specific hash key was unable to use the index. As I allude to in one of the commits, preserving that functionality would require using the [positional operator](http://www.mongodb.org/display/DOCS/Updating#Updating-The%24positionaloperator) in `sort()`, which is not supported. I asked Eliot about the feature at MongoSV and he said such functionality was far off, but that an alternative solution should be possible using the new aggregation framework in Mongo 2.2. Currently, I removed the `sort()` calls from ThreadManager's queries. A compromise may be to sort threads by `createdAt` timestamps.