artsy/garner — pull request #34 by fancyremarker (2013-06-17 23:18:33+00:00)
Title: Benchmarking for binding strategies

to: @dblock
cc: @joeyAghion

The purpose of this benchmarking suite is to sanity-check the performance of Garner binding strategy pairs (i.e., binding/invalidation pairs) against a no-cache baseline, and also to compare the performance of different binding strategy pairs. In particular, I wanted to have a standardized way to compare the performance of the `SafeCacheKey` strategy versus the `IdentityCache` strategy (based on the concepts of Garner 0.3.3 and earlier) which I'll send along next.

I'm not thrilled with how this code came out, so suggestions are welcome. My attempt to isolate profiler calls to just the primary methods of interest led to the ugly `BenchmarkContext` / `BenchmarkContextWrapper` separation. Practically, however, the code is useful. Here's the result of testing `SafeCacheKey` against baseline, with `N=100` iterations, `D=12850` approximate Mongoid document size, and `R=8` "recursive" database calls per cache block. This is based off averages computed for one of Artsy's most popular API endpoints, `/api/v1/artwork/:id`. The test was run on a c1.medium instance on AWS, connecting to a Heroku "memcache:5mb" server (with compression enabled) and a MongoHQ "experimental" server.

```
$ N=100 D=12850 R=8 bundle exec rake benchmark
Key: Base
Invalidation: Base
MethodProfiler results for: BenchmarkContext
+-----------------------+-----------+-----------+--------------+-------------+
| Method | Min Time | Max Time | Average Time | Total Time |
+-----------------------+-----------+-----------+--------------+-------------+
| #a_warm_virtual_fetch | 19.834 ms | 74.019 ms | 33.432 ms | 3343.164 ms |
| #b_cold_virtual_fetch | 20.896 ms | 84.498 ms | 38.979 ms | 3897.856 ms |
| #c_warm_class_fetch | 19.782 ms | 73.236 ms | 30.223 ms | 3022.333 ms |
| #d_cold_class_fetch | 21.082 ms | 82.281 ms | 35.234 ms | 3523.416 ms |
| #e_force_invalidate | 0.014 ms | 0.045 ms | 0.028 ms | 2.783 ms |
| #f_soft_invalidate | 0.016 ms | 0.044 ms | 0.031 ms | 3.058 ms |
+-----------------------+-----------+-----------+--------------+-------------+

Key: SafeCacheKey
Invalidation: Touch
MethodProfiler results for: BenchmarkContext
+-----------------------+-----------+-----------+--------------+-------------+
| Method | Min Time | Max Time | Average Time | Total Time |
+-----------------------+-----------+-----------+--------------+-------------+
| #a_warm_virtual_fetch | 2.651 ms | 54.156 ms | 3.426 ms | 342.645 ms |
| #b_cold_virtual_fetch | 24.641 ms | 81.999 ms | 44.719 ms | 4471.918 ms |
| #c_warm_class_fetch | 2.964 ms | 55.139 ms | 4.327 ms | 432.680 ms |
| #d_cold_class_fetch | 25.211 ms | 83.124 ms | 42.614 ms | 4261.439 ms |
| #e_force_invalidate | 0.852 ms | 2.112 ms | 1.341 ms | 134.080 ms |
| #f_soft_invalidate | 0.016 ms | 0.078 ms | 0.029 ms | 2.851 ms |
+-----------------------+-----------+-----------+--------------+-------------+
```

A clear advantage on warm cache fetches, both for virtual (i.e., `Garner::Mixins::Mongoid::Identity`) and class-bound cache block, with a minimal hit on cold cache fetches. Note that the benefit of Garner appears completely uncorrelated with `D`, the document size and highly correlated with `R`, the number of recursive database calls (or really any time- or I/O-intensive operations) within a cache block.