Findwise/Hydra — pull request #16 by jwestberg (2012-06-01 11:19:17+00:00)
Title: Boosting performance through per-stage indexes

Added measurements to fetching StressTest. Discovered how poor performance was in the usecase of batch-adding documents while Hydra is not working. Fixed those performance issues by adding per-stage indexes in MongoDB.

When a new stage connects to a Core, an index is now automatically created for that stage's "fetched" metadata. Increases performance over 100000 document batches by roughly 50x. Makes Hydra faster in general as well, but not at this dramatic level.