ayende/raven.voron — pull request #8 by KrzysFR (2013-12-01 15:58:58+00:00)
Title: Updated the FoundationDB performance comparison test to be more idiomatic

As requested in ayende/raven.voron#7 , I updated the FoundationDB perf test to be more idiomatic.

The write test logic was actually pretty ok (except the encoding of the keys). It could be further optimized by packing all the keys in batch (less overhead for the Garbage Collector) but it should be negligible for this benchmark.

I changed the read test to read values by batch using GetValuesAsync(IEnumerable<...>) instead of a single item at a time using GetAsync(...). The previous implementation was stuck at about 100 read per second. With batching I'm getting 80 to 90.000 read/sec.

My test environment:
- CPU: Core i7-2600 @ 3.40 Ghz
- Memory: 12 GB
- Hard drive: SSD, Samsung 840 EVO (500 GB)
- OS: Windows 7, x64
- FoundationDB: v1.0.1, running locally, configured with two process, ssd engine, no replication.

```
Testing: FoundationDB
Write Seq: 1 000 000 items in 114 457 sec, 8 737 ops/s.
Write Seq [2]: 1 000 000 items in 67 791 sec, 14 751 ops/s.
Write Seq [4]: 1 000 000 items in 43 067 sec, 23 220 ops/s.
Write Seq [8]: 1 000 000 items in 30 566 sec, 32 716 ops/s.
Write Seq [16]: 1 000 000 items in 23 226 sec, 43 055 ops/s.
Read Seq: 1 000 000 items in 21 765 sec, 45 945 ops/s.
Read Seq [2]: 2 000 000 items in 22 042 sec, 90 736 ops/s.
Read Seq [4]: 4 000 000 items in 43 435 sec, 92 092 ops/s.
Read Seq [8]: 8 000 000 items in 91 175 sec, 87 743 ops/s.
Read Seq [16]: 16 000 000 items in 193 965 sec, 82 489 ops/s.
Write Rnd: 1 000 000 items in 146 597 sec, 6 821 ops/s.
Write Rnd [2]: 1 000 000 items in 89 053 sec, 11 229 ops/s.
Write Rnd [4]: 1 000 000 items in 63 200 sec, 15 823 ops/s.
Write Rnd [8]: 1 000 000 items in 44 898 sec, 22 273 ops/s.
Write Rnd [16]: 1 000 000 items in 42 767 sec, 23 383 ops/s.
Read Rnd: 1 000 000 items in 22 979 sec, 43 518 ops/s.
Read Rnd [2]: 2 000 000 items in 24 395 sec, 81 984 ops/s.
Read Rnd [4]: 4 000 000 items in 48 115 sec, 83 134 ops/s.
Read Rnd [8]: 8 000 000 items in 97 933 sec, 81 689 ops/s.
Read Rnd [16]: 16 000 000 items in 201 652 sec, 79 345 ops/s.
```

I suspect that you would need more than 16 threads to max out on the write sequential test (where Voron's sweet spot seems to be between 4 and 8 threads)

Also, the rest test are hitting a known limitation of a the current client that is only able to process 80-90k ops/sec per process. Currently, you whould need to have multiple processes to scale up before starting to bottleneck the server.

For comparison, here is the Voron test on the same hardware (using the latest 'voron' branch)

```
Testing: Voron
Write Seq: 1 000 000 items in 18 846 sec, 53 062 ops/s.
Write Seq [2]: 1 000 000 items in 21 809 sec, 45 853 ops/s.
Write Seq [4]: 1 000 000 items in 16 292 sec, 61 380 ops/s.
Write Seq [8]: 1 000 000 items in 15 907 sec, 62 865 ops/s.
Write Seq [16]: 1 000 000 items in 25 571 sec, 39 107 ops/s.
Read Seq: 1 000 000 items in 2 296 sec, 435 540 ops/s.
Read Seq [2]: 2 000 000 items in 2 599 sec, 769 527 ops/s.
Read Seq [4]: 4 000 000 items in 3 789 sec, 1 055 688 ops/s.
Read Seq [8]: 8 000 000 items in 6 676 sec, 1 198 322 ops/s.
Read Seq [16]: 16 000 000 items in 14 263 sec, 1 121 784 ops/s.
Write Rnd: 1 000 000 items in 43 745 sec, 22 860 ops/s.
Write Rnd [2]: 1 000 000 items in 49 376 sec, 20 253 ops/s.
Write Rnd [4]: 1 000 000 items in 47 342 sec, 21 123 ops/s.
Write Rnd [8]: 1 000 000 items in 49 319 sec, 20 276 ops/s.
Write Rnd [16]: 1 000 000 items in 51 624 sec, 19 371 ops/s.
Read Rnd: 1 000 000 items in 3 205 sec, 312 012 ops/s.
Read Rnd [2]: 2 000 000 items in 3 360 sec, 595 238 ops/s.
Read Rnd [4]: 4 000 000 items in 4 326 sec, 924 642 ops/s.
Read Rnd [8]: 8 000 000 items in 7 540 sec, 1 061 008 ops/s.
Read Rnd [16]: 16 000 000 items in 15 821 sec, 1 011 314 ops/s.
```

I must say I am VERY impressed with the read speed (both seq and random). For the write tests, FoundationDB is actually faster with 16 threads, though in reality I guess both tests are just two fancy ways to measure the performance bottleneck of my single consumer-grade SSD disk, more than anything.

Lastly, I should point out that FoundationDB is not really targeted for local, single machine, usage. It is optimized for use with distributed clusters, with fail over and data redundancy.

I repeated the test using our slowest test cluster (4 x Intel NUC with consumer grade SSDs, running Ubuntu 13.10), and got the following results (again, reads are hitting the current client bottleneck).

```
Testing: FoundationDB (remote cluster)
Write Seq: 1 000 000 items in 136 642 sec, 7 318 ops/s.
Write Seq [2]: 1 000 000 items in 86 482 sec, 11 563 ops/s.
Write Seq [4]: 1 000 000 items in 49 817 sec, 20 073 ops/s.
Write Seq [8]: 1 000 000 items in 36 426 sec, 27 453 ops/s.
Write Seq [16]: 1 000 000 items in 30 380 sec, 32 916 ops/s.
Read Seq: 1 000 000 items in 26 574 sec, 37 631 ops/s.
Read Seq [2]: 2 000 000 items in 28 465 sec, 70 262 ops/s.
Read Seq [4]: 4 000 000 items in 58 852 sec, 67 967 ops/s.
Read Seq [8]: 8 000 000 items in 122 927 sec, 65 079 ops/s.
Read Seq [16]: 16 000 000 items in 245 300 sec, 65 226 ops/s.
Write Rnd: 1 000 000 items in 196 832 sec, 5 080 ops/s.
Write Rnd [2]: 1 000 000 items in 125 254 sec, 7 984 ops/s.
Write Rnd [4]: 1 000 000 items in 80 938 sec, 12 355 ops/s.
Write Rnd [8]: 1 000 000 items in 59 221 sec, 16 886 ops/s.
Write Rnd [16]: 1 000 000 items in 48 452 sec, 20 639 ops/s.
Read Rnd: 1 000 000 items in 19 847 sec, 50 385 ops/s.
Read Rnd [2]: 2 000 000 items in 27 633 sec, 72 377 ops/s.
Read Rnd [4]: 4 000 000 items in 59 461 sec, 67 271 ops/s.
Read Rnd [8]: 8 000 000 items in 118 563 sec, 67 475 ops/s.
Read Rnd [16]: 16 000 000 items in 236 572 sec, 67 633 ops/s.
```

Overall, I'm very impressed by the performance of Voron. I have been searching for a very fast storage engine on top of which I could put the Layer API of FoundationDB, to be able to have the same application code scale from large distributed clusters, down to in-process / single file on disk. Similar to how you would use either PostgreSQL or SQLite with your app, depending on the deployment scenario.