/dev/docs/infobase
Type: /type/page
body: value: **Created on July 5, 2008**

Reports about infobase performance and ideas and suggestions to improve the performance.

* [[performance_issues|Open Library Performance Issues]] - *Anand Chitipothu*
* [[infostore|Infostore: Generalizing Infobase to support mulitiple database schemas]] - *Anand Chitipothu*

If you have any ideas to improve Infobase performance or if you have any numbers on Infobase performance please add them here.

## David's comments
We have a difficult decision: optimize the implementation of Infobase as an RDBMS-based system or choose a more efficient storage back-end. While I'm most qualified to help with the former strategy, I prefer the latter one.

And there's a fairly obvious choice if we change the back-end: it should be an LDAP server. Open Library needs (or at least could use) the following capabilities that LDAP servers provide more cleanly than RDBMS servers: (1) multivalued fields for a large percentage of fields, (2) ability to make schema modifications without excessive delay in the database, (3) the ability to store all attributes with spacial locality (grouped in the same place on disk), (4) proven ability to store millions to billions of "things" with high performance (something RDBMSs cannot do with triple-stores), (5) ability to index attributes as substrings and other useful things for querying, (6) ability to create indexes across multiple multivalued fields, (7) data quality controls (using object classes) that truly work the way Open Library does, (8) useful replication options for distributing the directory across servers, (9) a robust, LDAP-based API, (10) LDIF for record interchange (in addition to the current JSON-based protocols).

Basically, the only thing not directly supported is revision control of things, but the abstraction provided by Thing DB already supports layering revision control on top of a non-versioned system like we have right now with PostgreSQL. We could also use the opportunity to have the system store old revisions on a different server or cluster instead of wasting the prime space that stores current revisions.

My thesis adviser is one of Sun's lead directory services engineers, and Sun has expressed interest in the past at devoting development resources to optimizing their directory technologies for scientific work. An rRNA sequence alignment lab I worked in started migrating to Sun-based directory server storage in 2005.

I would look at Fedora Directory Server, ApacheDS, OpenDS, and OpenLDAP. Avoid any LDAP server or configuration that uses a triple-store in a relational database; it would put you right back at square one. OpenDS looks particularly promising. NintendoDS had disappointing performance.
title: Infobase