FoundationDB/sql-layer — pull request #999 by MMcM (2013-10-23 01:18:03+00:00)
Title: Add extensible storage formats to AIS

This is a pretty big diff in some pretty old code, but I couldn't figure out any smaller pieces.

AIS objects that describe storage areas (Groups, Indexes and Sequences) have a new StorageDescription field. In Protobuf, this is a Storage message with nothing but extension fields. Descriptions and extensions are defined for the four storage formats currently supported:
* Persistit tree
* FDB directory layer
* Full text index in the file system
* Memory table factory

This branch does not actually introduce any new storage formats. It does not even add a class called StorageFormat (which would be a factory for StorageDescription). That will be coming soon.

Most of the changes are around plumbing around the new StorageFormatRegistry that needs to be present when objects are created or protocol buffers decoded.

The tricky parts are around bootstrapping and testing.
* Tables (etc.) are created initially without any storage linkup and then introduced to the AIS. But along the way the partial AIS still needs to validate. Formerly, this happened by generating tree names that were unique in a very small scope and throwing them away. Now there is a standin to mark that the object came from AISBuilder.
* A memory table is created first in that way and then registered with a factory. Since the factory itself cannot serialize, it needs to be remembered by TableName for later recovery. Persistit serializes memory tables to the B-tree. FDB just keeps them in memory, but there is still a copy as part of the registration in the AIS and serialization as part of AIS cloning.

Some things noted but not addressed here:
* NameGenerator is synchronized by a wrapper class rather than using that declaration on methods. That makes is somewhat harder to have a base class with common name handling (table ids) and subclasses for tree name generation and prefix bytes.
* A fair amount of effort is invested in getting unique tree names to be close to the table / index names because the Persistit GUI tool knows (next to) nothing about the SQL layer. With a different tool, we could perhaps just use the GUID for the tree name.
* ApiTestBase.createTable (etc.), when used in an IT, does some very messy things with SchemaFactory, rather than just calling AISDDL the way a SQL connection would. If we keep this, it will get even messier to support ITs that test specific storage formats.