shuttler/nessDB — Issue #9: benchmark feedback
Author: laurent-charmet
Created: 2011-10-29 15:24:07+00:00
State: open

Hi, I repost here what I post on googlecode

- I'm doing a bench and it become very slow after a while

Here the diff with my tunning:
-#define KEYSIZE 20
-#define VALSIZE 100
-#define NUM 1000000
+#define KEYSIZE 16
+#define VALSIZE 24
+#define NUM 100000000
#define R_NUM 10000
#define REMOVE_NUM 10000
-#define BUFFERPOOL (1024*1024*1024)
-#define BGSYNC (1)
+#define BUFFERPOOL (2*1024*1024*1024)
+#define BGSYNC (0)

@@ -88,10 +88,16 @@ void random_value()
}

void random_key(char *key,int length) {
+/*
char salt[36]= "abcdefghijklmnopqrstuvwxyz0123456789";
memset(key,0,length);
for (int i = 0; i < length; i++)
key[i] = salt[rand() % length];
+*/

+ *(int*)key = rand();
+ *(int*)(key+4) = rand();
+ *(int*)(key+8) = rand();
+ *(int*)(key+12) = rand();

What version of the product are you using? On what operating system?
- version 1.7
- Linux hyperstream 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
- my config: core i7 2600, 16 GB of ram, SSD ocz vertex 3

Please provide any additional information below.
- it's not possible to have a buffer pool > to 2 GB since it's an int and not an int64 (I have 16 GB of ram)
- DB size is very huge with big overhead: I have a key of 16 bytes and a records of 24 bytes. So it's 40 bytes per records. But after inserting 100 millions records, the DB size (*.db+*.idx) is about 14 GB, so it's an average of 151 bytes per records !!! With Kyoto using hash DB it's rather ~ 55 bytes
- DB inserting is quite fast, but speed decrease quickly. I suppose that the bufferpool is too small.
- performance: it's an average of 29.000 insert/s (but it start at 140.000 and finish at a very slow rate) after 100 millions insert. Kyoto is at 90.000 insert/s after 250 millions insert !! But I set the cache to 12 GB of ram
- iostat show a very important write I/O. Often between 50 to 120 MB/s during the whole benchmark. It seems there is a lot of update (index balancing ??)

Some others information:

|Random-Write (done:99999999): 0.000035 sec/op; 28911.6 writes/sec(estimated); 2.6 MB/sec; cost:3458.821(sec)
lch@hyperstream:/DT/local/nessDB/src$ du -k .
14781888 ./ndbs
=> 151 bytes per records instead of 24 ...

-rw-r--r-- 1 lch users 245325984 2011-10-26 15:58 ness0.db
-rw-r--r-- 1 lch users 918359689 2011-10-26 15:58 ness0.idx
-rw-r--r-- 1 lch users 245150720 2011-10-26 15:58 ness10.db
-rw-r--r-- 1 lch users 918612711 2011-10-26 15:58 ness10.idx
-rw-r--r-- 1 lch users 245293760 2011-10-26 15:58 ness11.db
-rw-r--r-- 1 lch users 918339284 2011-10-26 15:58 ness11.idx
-rw-r--r-- 1 lch users 257679456 2011-10-26 15:58 ness12.db
-rw-r--r-- 1 lch users 917527165 2011-10-26 15:58 ness12.idx
[...]

Hope you can continue to enhance this promising DB :)