Some design and first stab at a structured data api.

Summary:
My first instinct is to use YAML for structured data schema definitions.
Much easier to read than any other formats. Also accepts javascript as
valid YAML.

The idea behind the go client is to define schemas via go structs and
simply create the YAML schema directly from the client application
source.

Test Plan: Unittests

Reviewers: petermattis, andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D8
Fixed errors in client_test.go

Test Plan: unittests

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D9
Fleshed out more documentation for the schema along with the rest of the YAML parsing directives. Also now parsing all "roach" tags from Go structs to generate the schema automatically from a Go application.

Test Plan: unittests

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D10
fix build error with correct import path for go-yaml
Added schema validation. Need more unittests.

Summary:
Renamed "numberset" and "numbermap" to "integerset" and "integermap" to
avoid any confusion that these types may be used with floating point
values.

Test Plan: unittests

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D11
Consolidated previously-broken out directories of structured/{client, schema,db} into a single structured directory. Breaking them out was premature. Combined the doc.go files and changed some wording.

Test Plan: None necessary

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D12
Added additional tests for validation. Shifted some code around between go_schema_test and schema_test.

Summary:
Fixed a bug where we weren't setting Interleave from struct field tag
"interleave".

Test Plan: this commit adds unittests! hurrah.

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D13
Added incoming foreign key map to table for implementing on-delete policy.

Test Plan: unittests

Reviewers: andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D14
Removed helper methods on gossip for adding and getting info objects of the basic types. This was just a lot of extra lines of code without much of a purpose.

Summary:
Added support for arbitrary types of info objects, so long as they
implement the gossip.Ordered interface. This will allow us to add
more complicated struct types to groups, something we'll need to do
virtually everywhere. For example, when keeping track of disks with
available capacity per datacenter, we're going to want to store a
struct which describes the location of the disk host, port, device
and also contains more detailed information about free space. We
could previously have stored this all in the key, but not worth
having to parse it out!

Test Plan: unittests included

Reviewers: andybons, levonlloyd

Reviewed By: andybons, levonlloyd

Differential Revision: http://phabricator.andybons.com/D16
Fix build after a recent change to goraft.

Summary:
The timeout used to be hard-coded 150ms, but is now a required parameter
to NewHTTPTransporter.

You may need to update your copy of goraft after this commit.

Test Plan: go test ./...

Reviewers: spencerkimball, andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D17
fix build broken by removing helper methods
Fleshed out layering and general structure of cockroach. The devilish details remain; I tried to capture many of the TODOs in comments.

Summary:
server/server.go is the entry point to the system. A server now creates
the following objects:

 - An HTTP server for handling RESTful APIs

 - An RPC server for handling Raft, Gossip, & Roach Key-Value RPCs

 - A Gossip instance which connects P2P to the larger gossip network
   by spawning clients in order to minimize total hops to all known
   data while still remaining under a maximum fan out limit. Each
   gossip instance also registers a gossip service on the RPC
   server.

 - A local storage Node which multiplexes incoming requests to the
   appropriate storage engine (one storage engine per physical storage
   device). Nodes register a Node service on the RPC server, which
   accepts incoming Roach Key-Value RPCs. The storage node is passed
   the gossip instance so the node can advertise its availability. It
   also uses the gossip instance to access information about other
   available nodes when allocating new replicas to build consensus
   groups.

 - A Roach Key-Value database (kvDB). This is a client which provides
   virtual access to the monolithic sorted map by looking up ranges to
   satisfy requests to read or write the map. For range scans, for
   example, it may decompose the request into multiple RPCs. This layer
   provides an async API for higher-level client APIs.

 - A Key-Value RESTful API server. Provides CRUD functionality via HTTP
   POST, GET, PUT & DELETE methods. The KV REST server is instantiated
   with the roach node's HTTP serve mux.

 - A structured schema database (structuredDB). This is a client built
   on top of the key-value database that understands how to construct
   schemas comprised of familiar RDBMS concepts such as tables, columns,
   keys, indexes, foreign keys and a query language. A structuredDB is
   intantiated with a reference to the underlying kvDB.

 - A structured schema RESTful API server. Provides CRUD functionality
   via HTTP methods. Unlike the simple KV RESTful server, the structured
   one generates endpoints for each table in the schema and depending
   on the indexes and foreign keys defined. It automatically exports a
   complete interface as well as documentation.

---

Extracted RPC server and client code from gossip/ into a more general
location in rpc/. This still has a ways to go.

Also, moved contents of former db/ package into storage/ and for the
moment decommissioned the Raft code. It's currently commented out in
storage/raft.go. I removed the delete and put commands with the
expectation that the commands being serialized into the Raft log match
those of the Cockroach "Node" RPC service. That API is defined by
storage/node.go and kv/messages.go.

As mentioned in a previous comment, a single instance of cockroach
exports a single Node server. That server multiplexes between one or
more "stores". Each store corresponds to one physical device. E.g.
you might have a store for ssd01, ssd02, sda1, sda2, etc. Each store
is backed by a storage engine. The interface is in storage/engine.go;
there are two implementations: storage/in_mem.go & storage/rocksdb.go.

Although each store has a single engine, there can be many ranges
(as high as 2^18). A store's job is to maintain a map of range objects.
Each range hosts a non-overlapping, probably non-contiguous region of
the overall keyspace. Each range maintains its own Raft instance to
mediate writes between the replicas in the range. Whichever replica
is the established leader makes decisions about new allocations. Each
range has a reference to the underlying engine (it can only write to
its key range), and to an allocator for maintaining range replicas.

I removed the protobuf API for the key value store. We won't need it
in the form originally envisaged. It's contents have been reworked into
kv/messages.go.

Also added an LRU cache, taken from another go package. Not in use,
but will need to press it into service soon.

Test Plan:
This is a partial review submission. It doesn't even compile.
I just wanted to get it looked over now rather than in a few more days.

Reviewers: petermattis, andybons

Reviewed By: andybons

Subscribers: levonlloyd

Differential Revision: http://phabricator.andybons.com/D15
Fixed race condition in gossip simulation startup which would cause client c

Summary:
Added a very basic heartbeat mechanism to the rpc/client code so a
disconnected client is not kept cached and continually returned on
calls to rpc.NewClient().

Test Plan: unittests

Reviewers: petermattis

Differential Revision: http://phabricator.andybons.com/D19
Initial commit of multiraft.

Summary: Contains basic scaffolding and the beginning of a RequestVote implementation.

Test Plan: soon

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D20
Port all logging to glog.

Summary: Satisfy golint on all modified files to get past the presubmit.

Test Plan: go test ./...

Reviewers: spencerkimball, andybons

Reviewed By: andybons

Differential Revision: http://phabricator.andybons.com/D21
Replace the pass-through LocalTransport with LocalRPCTransport.

Summary:
Remove some redundant shims by borrowing the net/rpc client interface
more directly.  rpc.Call is a suitable container for a request/response
pair even when net/rpc is not used as the underlying transport.

Test Plan: go test

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D22
Remove the cockroach binary which was mistakenly checked in.

Summary: Add it to gitignore.

Test Plan: n/a

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D23
Introduce a Clock abstraction and use it to make a more precise test.

Test Plan: go test

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D26
Remove goprotobuf install for now since it is not used.

Test Plan: go test ./...

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D27
***WIP*** First pass at range allocation. Lays down a basic algorithm for selecting a set of peers to send a new range to. MISSING 1) A lot of the config data structures are straw men right now 2) The actual data in the gossip 3) Most Tests

Test Plan: go test ./...

Reviewers: spencerkimball

Reviewed By: spencerkimball

Differential Revision: http://phabricator.andybons.com/D25