puppetlabs/puppet — pull request #2329 by zaphod42 (2014-02-05 19:36:34+00:00)
Title: (PUP-753) Benchmark puppet compiling w/many modules

This sets up a scenario in which we are benchmarking the puppet compiler
with a large number of modules, but very simple manifests. The generated
modules have 2 manifests each an public and an internal. The site.pp for
the environment simply includes all of the modules' public classes. The
public classes in turn include their own internal class. The internal
class then places a single "notify" resource in the catalog.

This also ends up laying down a general structure for benchmark scenarios.
Each scenario has support files under `bencmarks/`. Each
scenario needs to provide a file name `benchmarker.rb` that contains a class
named `Benchmarker`, which is constructed with 2 values the "target" directory
to use for the benchmark data and the "size" of the benchmark. The instance of
the benchmark class needs to respond to 3 methods: `setup`, `generate`, and
`run`.

Rake tasks are generated for each benchmark scenario following the scheme
`benchmark::`. There is also a task `benchmark:` which
is an alias for `benchmark::run`. The `run` task will run the benchmark
10 times (controllable with the `ITERATIONS` environment variable) and output
timings of each run as well as totals. The `profile` task will run the
benchmark a single time and create a calltrace file that can be used with a
cachegrind implementation.