Shopify/identity_cache — pull request #128 by dylanahsmith (2014-02-27 02:06:07+00:00)
Title: Refactor benchmark so that it has easily comparable results.

@fbogsany & @arthurnn for review
cc @camilo

## Problem

Currently our cpu benchmark is taking almost 40 minutes to run on my machine. The results are also confusing and hard to compare:

```
Rehearsal ----------------------------------------------------------
FindRunner: 1.280000 0.100000 1.380000 ( 1.876371)
FetchMissRunner: 22.710000 1.300000 24.010000 (182.998031)
FetchHitRunner: 51.210000 3.300000 54.510000 (374.829770)
DoubleFetchHitRunner: 58.160000 4.000000 62.160000 (385.260937)
DoubleFetchMissRunner: 29.040000 1.980000 31.020000 (191.976452)
----------------------------------------------- total: 173.080000sec

user system total real
FindRunner: 1.260000 0.100000 1.360000 ( 1.848705)
FetchMissRunner: 22.260000 1.300000 23.560000 (181.462431)
FetchHitRunner: 50.180000 3.340000 53.520000 (375.619798)
DoubleFetchHitRunner: 57.440000 3.940000 61.380000 (382.727823)
DoubleFetchMissRunner: 28.640000 1.980000 30.620000 (191.175364)
```

Without looking at the code, it looks like IDC is horribly is 2 orders of magnitude slower on cache misses, and even slower on cache hits. Partly this is because it is measuring the prepare step, and partly it is because the FindRunner isn't loading the associations.

I found the DoubleFetch*Runner names to be unhelpful in determining what it actually does, which is that it adds another fetch for a normalized association.

## Solution

* Reduce the number of runs so that the benchmark so that it is painless to use
* Avoid measuring the prepare method, since it makes it look like cache misses are very slow
* Separate denormalized and normalized fetches into separate runners to reduce redundancy and to make it easier to see where a regression occurred
* Fetch the same associations across all runners so they are comparable

I find the output to be a lot clearer now:
```
/Users/dylansmith/.rbenv/versions/2.1.0-github/bin/ruby ./performance/cpu.rb
Rehearsal: -------------------------------------------------------------
FindRunner: 1.090000 0.090000 1.180000 ( 4.420599)
FetchEmbedMissRunner: 1.730000 0.100000 1.830000 ( 5.332514)
FetchEmbedHitRunner: 0.170000 0.010000 0.180000 ( 0.217751)
FetchNormalizedMissRunner: 2.940000 0.240000 3.180000 ( 6.988252)
FetchNormalizedHitRunner: 0.250000 0.020000 0.270000 ( 0.322094)
------------------------------------------------------------------------
FindRunner: 1.070000 0.060000 1.130000 ( 4.506542)
FetchEmbedMissRunner: 1.760000 0.110000 1.870000 ( 5.503062)
FetchEmbedHitRunner: 0.170000 0.010000 0.180000 ( 0.218063)
FetchNormalizedMissRunner: 3.050000 0.230000 3.280000 ( 7.007530)
FetchNormalizedHitRunner: 0.260000 0.020000 0.280000 ( 0.341007)
```

We can see that fetch is much faster than find for cache hits and there is some overhead for cache misses. We can also see that normalized associations are slower than denormalized associations. These are roughly make sense so a regression would more easily stand out.