karmi/tire — pull request #724 by halfbrick (2013-05-08 09:50:31+00:00)
Title: Import task for Mongoid models should use the cursor, rather than skip()

Iterating through mongoid collections using skip() can get really slow for large collections, as it has to traverse from the beginning of the collection.
http://docs.mongodb.org/manual/reference/method/cursor.skip/

We were seeing our import task freeze up with high I/O after importing around 2.5 million records in a 6 million record collection.

This commit alters the mongoid import task so that it batches manually and makes the use of mongoid's cursor to only return a small batch of results.