Adds storage driver interface, tests, and three basic implementations

Re-opening of #630 with the new orphan next-generation branch.

@dmp42 @wking @bacongobbler @noxiouz 
On Mon, Nov 03, 2014 at 09:07:51PM -0800, Stephen Day wrote:
> > Using either a marker string [1] or an opaque marker type [2]
> > seems to support all proposed backends.  With the marker stored in
> > the registry, the storage backend doesn't need to keep track of
> > open list calls, so a reentrant interface is easy.  Or am I
> > missing something?
> 
> Yes, the marker or index will support most backends, but the
> behavior may be different on updates to that directory. With a
> marker string, an insert before the marker after the first call will
> lead to a missed entry.

That entry wouldn't have been part of the results from an initial
list-all call anyway.  I'm fine skipping over it during an iterative
list.

> With an index, one may have duplicate entries.

So just have the registry remember the last entry it got (as well as
the opaque marker), and then skip anything <= that entry when parsing
the next batch of entries.

> > How so?  ZRANGEBYSCORE is O(log(N)+M) with N being the number of
> > elements in the sorted set and M the number of elements being
> > returned.  That scales pretty well.  More importantly, it lets you
> > start streaming results from the registry to the client before
> > you've finished listing the whole directory.
> 
> That's not the case I'm worried about. If we want to iterate over
> the entries of a large directory to support an API operation, the
> latency of the registry request will still be proportional to the
> size of the directory, reentrant or not.

No, if the request is “give me a list of tags and their associated
image IDs so I can pull them” (#614), you can start writing back to
the client immediately, without waiting for the tag list to complete.
That's just a:

  storage → registry → client

pipe, and there's no reason to block the registry → client leg.

> Let's proceed with this simplified API and table this discussion,
> for now. We have collected some really good suggestions but we
> really need to look at how we can avoid this bottleneck. Once the V2
> paths are laid out, we can come back and see if we really need the
> reentrant API call.

Well, the API is versioned, so it shouldn't be too hard to adopt this
later.  Still, it doesn't seem that complicated to me, and I think
we'll need to adopt something that lets us stream results.  This is
what other storage engines already support for streamed listings, so I
expect it's a reasonable choice for how to structure this ;).