WiredTiger Backend for OpenLDAP
Open Source Solution Technology Corporation
HAMANO Tsukasa <hamano@osstech.co.jp>
LDAPCon 2015 Edinburgh November 2015

Abstract

sub scope searching, and also % prefix entries areneeded for one scope searching. If we choose theThis paper introduces WiredTiger backend for hierarchical structure, modrdn is fast but lookup
OpenLDAP. WiredTiger is an embedded database and sub scope search need extra cost.
having characteristics of multi-core scalability andlock-free algorithms. We implemented a new
OpenLDAP backend called back-wt that is using
WiredTiger database and then measured the performance.

1

Motivation

BerkeleyDB is a legacy embedded database. The
write performance of back-bdb (OpenLDAP backend using BerkeleyDB) is painfully slow and notscalable. If we flush disk asynchronously in orderto improve the write performance, data durability
will be sacrificed. Although OpenLDAP is a multithreaded application, the existing backends don’t
scale well with number of CPUs. The WiredTiger
backend will bring about highly concurrent writeperformance.

2Figure 1: Plain structure vs Hierarchical structure
We followed basically plain data structure but
we made some enhancements to the data structure for performance and database footprint. Before adding an entry, we reverse the DN per RDN
and then add the Reverse DN as the key into
WiredTiger’s B-Tree table. At this point, entries
are sorted by Reverse DN, so we can search rapidly
with a sub scope using WiredTiger’s range search.
The range search is an efficient method that only
needs WT_CURSOR::search_near() and increment
cursor operations for this purpose.

Data StructureFirst, we had to choose data structure either plain
structure such as back-bdb or hierarchical structure such as back-hdb. If we choose the plain
structure, sub scope searching is fast but modrdn
and add operations need extra cost. Actually wecan’t use modrdn with sub directories on back-bdb.
The plain structure need many @ prefix entries for1

wtconfig log=(enabled)
wtconfig transaction_sync=(enabled=false)

Listing 2: slapd.conf for writing transaction log
without sync
Figure 2: Making Reverse DN
3. Write transaction log into file, and log records
are flushd for each commit of the transaction.
wtconfig log=(enabled)
wtconfig transaction_sync=(enabled=true)

Listing 3: slapd.conf for writing transaction log
with sync

4 Current Status
Figure 3: back-wt data structure3• slapadd, slapcat, slapindex have been implemented.
• basic LDAP operations (BIND, ADD,DELETE, SEARCH, MODIFY, MODRDN)
have been implemented.
• Password Modify Extended Operation (RFC3062) works.
• deref search has not been implemented yet.
• alias and glue entry have not been implemented yet.
• WiredTiger does not support multiprocess access yet. It means that we can’t do slapcat
while running slapd at the moment. However,WiredTiger is planning to support RPC in thefuture. If it is realized, we can do hot-backup
while avoiding multi-process locking.
• We do not implement entry cache similar toback-bdb. It’s not absolutely necessary since
WiredTiger cache is fast enough.
• back-wt currently uses B-Tree table. We will
test LSM table in the future.

Data Durability

WiredTiger has several durability levels of transaction. Here is the back-wt settings corresponding to each durability level. In back-wt, wecan set wtconfig patameter in order to set durability level. This parameters are just passed towiredtiger_open().
1. Write transaction log into memory. This is thefastest, but it only ensure durability at checkpoint.
wtconfig log=(enabled=false)

Listing 1: slapd.conf for in-memory transactionlog

2. Write transaction log into file, but log records
aren’t flushd for each commit of the transac- 5 Benchmarking
tion. This is equivalent to dbnosync in backbdb.
We have measured benchmarks that focus on concurrency performance by new benchmarking tool2

that called lb.1 This benchmark tool can generatemany concurrency load by goroutines of Go. See
our wiki page for detail of benchmarks.2

back-wt req/sec
back-bdb req/sec
back-mdb req/sec

25000
Requests per second

5.1

30000

Enviroments

20000

15000

10000

5000

We have executed benchmarks on following environments:

01

4

8

16

32

64

128

256

512

Concurrency Level

Figure 5: LDAP ADD Rate (nosync txn log)

• 12 Core x 2 Hyper Threading = 24 LogicalCPUs.
• 15,000 RPM SAS Disks, not used RAID
cards.
• Database directory was placed on ext4 filesystem on Linux box.
• 60G Memory• OpenLDAP of git master at Sep 2015 and applied some back-wt patches.
• No checkpoint was performed during thebenchmarking.
• We measured two methods for ADD benchmarking, the first flushes disk transaction log
each request and the second doesn’t flush disk
transaction log each request.

45000
back-wt req/sec
back-bdb req/sec
back-mdb req/sec

40000

Requests per second

35000
30000
25000
20000
15000
10000
5000
0
1

2

4

8163264128256512256512

Concurrency Level

Figure 6: LDAP BIND Rate
30000
back-wt req/sec
back-bdb req/sec
back-mdb req/sec

Requests per second

25000

5.22Results

20000

15000

10000

6000
back-wt req/sec
back-bdb req/sec
back-mdb req/sec

5000

5000

Requests per second

012

4000

48163264128

Concurrency Level

3000

Figure 7: LDAP SEARCH Rate

2000

1000

5.3 Analysis

012

48163264128256• We have only used 24 logical CPUs. We may
get more scalability on more CPUs.
• The reading performances are much the same.
• The concurrency writing performances ofback-wt are pretty good.

512

Concurrency Level

Figure 4: LDAP ADD Rate (sync txn log)
12

https://github.com/hamano/lb
https://github.com/osstech-jp/openldap/wiki/3