jnunemaker/mongomapper — pull request #450 by jonpollock (2012-09-20 01:26:55+00:00)
Title: Dramatically speed up MongoMapper::Plugins::Keys#load_from_database

This pull request significantly enhances the performance of mongo-mapper when dealing with a large number of documents that may also have a large number of keys. When profiling our app, we noticed that MongoMapper::Plugins::Keys#load_from_database was taking a VERY long time. In fact, much longer than the time it took to get a response from mongo. For example, a 5kb response from mongo for a single flat document with about 40 keys would arrive in under 1ms, but the "load_from_database" method took over 17ms to complete. This method is only working with stuff that is already in memory and it was surprising that it took so long.

The first thing we noticed was the Keys#key? method. This was generating an array of keys and calling include? on it, instead of just calling has_key?. This was called quite a lot and the changes cut about 8ms off the single load_from_database call for that one example object; basically cutting the time in half. In our large test page, this was being called 34,667 times for 623 documents.

In the load_from_database method, we broke the different types of keys into separate lists, which saved doing all the respond_to? calls and allows us to more directly do what needs to be done for each key type. This also eliminated almost all calls to key?.

Then we deferred building the associations until needed. The Keys#load_from_database method will simply store the value for associations being loaded into an instance variable, "@_association_#{key}", for future evaluation.

These changes alone reduced the average response time for a particular page of ours which loaded 623 documents from 8.1 seconds to 1.4 seconds. keys#load_from_database went from 4.617 seconds to 0.23 seconds and these changes also helped other aspects of the app. Naturally, your mileage may vary.

I have re-run the mongo-mapper tests and they all still pass.