jnunemaker/mongomapper — pull request #521 by cheald (2013-06-22 08:05:57+00:00)
Title: Improve load speed from the database by 1000%

The tl;dr here is that MongoMapper is doing some really ugly things - notably, letting the Dirty plugin get its hands on content as it's being loaded from the DB, and performing #extends on instance variables, which is torching the entire Ruby method cache on a regular basis. This PR fixes this.

First, let's benchmark!

Here's the benchmark I used. I was interested in comparing Mongoid with MongoMapper, and was rather dismayed to find how abysmally MM was performing. It's worth noting that Mongoid uses the Moped driver, while MM uses the 10gen Mongo driver (which, IMO, is something of a mess; part of the blame is due there).

The benchmark is here: https://github.com/cheald/mongo_odm_bench

This comparison is MongoMapper 0.12.0 vs Mongoid 3.1.4

user system total real
MongoMapper Insert 0.480000 0.040000 0.520000 ( 0.542837)
Mongoid Insert 0.530000 0.040000 0.570000 ( 0.574484)

user system total real
MongoMapper Read 150 batches of 1 0.170000 0.010000 0.180000 ( 0.212139)
Mongoid Read 150 batches of 1 0.050000 0.010000 0.060000 ( 0.066210)
MongoMapper Read 3750 batches of 25 1.700000 0.180000 1.880000 ( 1.937555)
Mongoid Read 3750 batches of 25 0.180000 0.020000 0.200000 ( 0.211497)
MongoMapper Read 15000 batches of 100 6.540000 0.620000 7.160000 ( 7.263680)
Mongoid Read 15000 batches of 100 0.620000 0.050000 0.670000 ( 0.704031)
MongoMapper Read 75000 batches of 500 32.200000 2.960000 35.160000 ( 35.474875)
Mongoid Read 75000 batches of 500 2.930000 0.280000 3.210000 ( 3.277081)
MongoMapper Read 150000 batches of 1000 64.250000 5.950000 70.200000 ( 70.786035)
Mongoid Read 150000 batches of 1000 5.810000 0.580000 6.390000 ( 6.499160)

Now, with my patches:

user system total real
MongoMapper Insert 0.740000 0.080000 0.820000 ( 1.205486)
Mongoid Insert 0.530000 0.040000 0.570000 ( 0.580322)
user system total real
MongoMapper Read 150 batches of 1 0.050000 0.000000 0.050000 ( 0.058201)
Mongoid Read 150 batches of 1 0.050000 0.010000 0.060000 ( 0.073255)
MongoMapper Read 3750 batches of 25 0.190000 0.030000 0.220000 ( 0.229481)
Mongoid Read 3750 batches of 25 0.180000 0.020000 0.200000 ( 0.217780)
MongoMapper Read 15000 batches of 100 0.620000 0.120000 0.740000 ( 0.764084)
Mongoid Read 15000 batches of 100 0.620000 0.060000 0.680000 ( 0.709350)
MongoMapper Read 75000 batches of 500 2.990000 0.490000 3.480000 ( 3.558932)
Mongoid Read 75000 batches of 500 2.980000 0.290000 3.270000 ( 3.324602)
MongoMapper Read 150000 batches of 1000 5.970000 0.960000 6.930000 ( 7.050302)
Mongoid Read 150000 batches of 1000 5.900000 0.550000 6.450000 ( 6.564033)

So yeah, the time to read 15k records was reduced from 70.7 seconds to 7.05 seconds, an improvement of 1002%. I'm pretty sure that the difference in insert time is due to the updated "safe" parameter, which I didn't explicitly set in the 0.12.0 config.

* The big change here is that when loading from the database, we use an #internal_write_key method that isn't eligible for plugins to overwrite. Previously, things like #dirty_key's write_key would be invoked when loading from the DB, which in turn invoked read_key and a whole other chain of things. This massively slowed down read
speeds.
* In two places, runtime instances were being `#extend`'d. **This wipes the Ruby method cache** and slows down the entire application. Those two instances have been refactored out.
* Other changes have been made to improve internal_write_key. First off, uncast values are now stored in write (by the app, not the DB), and only then from the Rails plugin, as that's the functionality that is concerned with uncast values. It's now a simple hash lookup rather than being a whole mess of ivars.
* Keys#keys now copies the class.keys to an ivar when it's used. This method is used very frequently, and avoiding the extra function call has substantial benefit when you start adding it up.
* is_a? was replaced with instance_of? in a number of places; this is faster when you want to compare types directly, rather than the inheritance hierarchiy.
* We're testing !foo rather than foo.nil? in a number of places for similar reasons.
* In a few places, ivars were used rather than their prettier accessors. Avoiding the function call does add up when you're calling it hundreds of thousands of times.
* Several tests which were too brittle have been fixed.

I'm quite certain that there are a number of other places to make improvements, but I'm pretty happy with a 10x improvement for an evening's worth of hacking.