basho/bitcask — pull request #83 by evanmcc (2013-03-19 21:48:57+00:00)
Title: Improve fold speed for large files filled with small objects.

This patch rewrites all of the file folds as subfolds of a function that folds over adaptively-sized chunks of the file in question, using return values as a simple protocol to communicate with the main folding function.

Some simple benchmarking. See test/bcfold_setup for info on key and value sizes.
```
pevm-fast-fold:

Report for run: small
Open: 23843056
Datafile Key Fold: 8119489 6795835
Datafile Fold: 31599952 6795835

Report for run: medium
Open: 1866682
Datafile Key Fold: 6934109 509849
Datafile Fold: 9501606 509849

Report for run: large
Open: 205041
Datafile Key Fold: 7065257 20948
Datafile Fold: 8051616 20948

Report for run: cs
Open: 150339
Datafile Key Fold: 573293 1024
Datafile Fold: 8601723 1024

master:

Report for run: small
Open: 298579027
Datafile Key Fold: 397753135 6795835
Datafile Fold: 168100962 6795835

Report for run: medium
Open: 22597198
Datafile Key Fold: 72591363 509849
Datafile Fold: 17351239 509849

Report for run: large
Open: 1095118
Datafile Key Fold: 9041969 20948
Datafile Fold: 7874795 20948

Report for run: cs
Open: 210885
Datafile Key Fold: 590771 1024
Datafile Fold: 7196062 1024
```

These tests still need to be re-run on some machines with slower disks, but I wanted to get some feedback on the code because I am concerned with the fragility/complexity of the current interface.