boltdb/bolt — pull request #135 by benbjohnson (2014-04-19 03:11:56+00:00)
Title: Bench

## Overview

This pull request adds the `bolt bench` command to allow standardized, synthetic benchmarks. This initial cut aims at providing a simple system that can be built upon. It currently only supports sequential read and write workloads with customizable key sizes, value sizes, and item counts. It also provides the ability to turn on profiling for read, write, or read-and-write workloads.

## API

To use `bolt bench`, build the binary and run it from the command line:

```sh
$ go install github.com/boltdb/bolt/...
$ bolt bench
```

The default configuration will write 1000 key/value pairs and then sequentially iterate over them for at least 1 second. You can also specify options to tweak this setup. Below is the help page:

```sh
$ bin/bolt bench --help
NAME:
bench - Performs a synthetic benchmark

USAGE:
command bench [command options] [arguments...]

OPTIONS:
--pmode 'rw'	Profile mode
--wmode 'seq'	Write mode
--rmode 'seq'	Read mode
--iter '1000'	Iteration count
--ksize '8'	Key size
--vsize '32'	Value size
--cpuprofile ''	CPU profile output path
--memprofile ''	Memory profile output path
--blockprofile ''	Block profile output path
```

## Output

When the benchmark is complete, two lines will print:

```
# Write	2.282639ms	(2.282us/op)	(438212 op/sec)
# Read	1.000034244s	(31ns/op)	(32258064 op/sec)
```

Each line shows 3 metrics: total time, time per operation, and number of operations per second.

## Profiling

One issue that was particularly frustrating about using the benchmarking in `go test` was that profiling starts from the beginning. That means that I couldn't isolate the profile to only the read portion of the benchmark. With this pull request, the `bolt bench` command accepts a "profile mode" (`--pmode`) that can be `r`, `w`, or `rw`. This specifies which portion(s) of the benchmark to profile.

To run a profile, simply specify the `--cpuprofile`, `--memprofile`, and/or `--blockprofile`:

```sh
$ bolt bench --cpuprofile /tmp/cpu.pprof
$ go tool pprof `which bolt` /tmp/cpu.pprof
```

## Other features

The next features to add to `bolt bench` include:

1. Random write mode

2. Random read mode

3. Parallel read mode

4. Deletion mode(s)

5. Streaming counters

6. Customizable minimum benchmark times - currently hardcoded to `1s` for reads.

7. Add benchmarks to CI.

The "streaming counters" feature is especially interesting. The idea is to kick off a separate goroutine that simply diffs the `DB.Stats()` on a given interval (e.g. `200ms`) and prints it to STDOUT as tab-delimited output. We can then pipe that directly to something like `gnuplot`.

## Acknowledgments

Thanks to @snormore for providing the benchmarking groundwork and several commits in this PR.