serverinit: add support for memory-only indexer. Enable in devcam.

To avoid confusion, the previously named "memoryIndex" configuration
parameter, which actually meant to slurp the index into the memory
corpus, has been renamed to "copyIndexToMemory". "memoryIndex" is now
the parameter to enable a memory-only indexer.

http://camlistore.org/issue/416

Change-Id: I7026e5463fda3a1887a88e0b07d4e3431c60c8b7
Send system status JSON on websockets upon connect and when it changes.

Change-Id: Ic0f211611d63b9d519ee3c5da0dd96c6b77459d5
schema: add FileReader.ForeachChunk

Will be used by the blobpacked storage server.

Also, we can then rewrite much of the legacy crufty methods on
FileReader in terms of this.

Change-Id: Ia371001f1ed502fa1da2dbc4a8e1c29e5f0c9e8c
schema: rewrite FileReader.LoadAllChunks, delete GetChunkOffsets

LoadAllChunks is now written in terms of ForeachChunk and is bounded.

GetChunkOffsets was never meant to be exported and was unused elsewhere.
It's now unused entirely.

Change-Id: I0653385fe805fda721e0b45ea0a17d9839c3ba6e
Aggregate all errors with their URLS in status.json in one place.

Change-Id: If7a540fc3d1a6caa94b5f04d44de3207f15ac4cf
Show server errors in the piggy menu.

When there is a server error, the piggy becomes sad and periodically
shakes his head. When you expand the piggy menu you see one or more
entries explaining why he is sad.

Change-Id: If1e28dbf02c9befaa46b62838cb704220f719bc6
Adjust position of error icon.

Change-Id: I2b86d8b52a3c616ca3f2408973d85e70fb27a2d7
blobpacked: first cut at packing, creating zip files with contiguous chunks.

See package docs in blobpacked.go for background & rationale.

Will minimize disk seeks, regardless of the underlying storage
system. I always knew this day would come, but we've been putting it
off. It's finally time to separate out logical (tons of small chunks)
storage from physical (big contiguous layout for how they'll be
accessed).

First 80% is done. Last 80% remains: mostly lots of tests and more
paranoia, and fsck things, for when we inevitably lose indexes. (but
each zip file is self-describing, so the index is redundant).

Change-Id: I13f9aa79c4d9d87b5fdcece0d99d97eba031d608
Merge branch 'master' of https://camlistore.googlesource.com/camlistore
Move importerRoot from status.json to discovery.

It seems to make more sense there...

Change-Id: I8450f1cb30ed331cfe15ddb49f55438c5bf9fd5b
Get continual server status updates over websocket.

Also show channel-level websocket errors in piggy menu.

Change-Id: Iad5f00e5578f28f837cd2e88cbdab3f715a9fed1
Checking in an unmodified copy of Go's archive/zip package.

The import is happening at hg 47de34bbcca1 but the last modification
of the package was at:

changeset:   20518:8ab50c14cd95
user:        Brad Fitzpatrick <bradfitz@golang.org>
date:        Mon Aug 04 16:12:55 2014 -0700
summary:     archive/zip: accept bogus trailing zeros in extras

... which will go into Go 1.4.

We don't need that fix in Camlistore, but rather we need additional
data about the state of the *zip.Writer while it's writing.  So this
is a pristine import in preparation for future changes so the changes
will be easy to see in the history, in case we want to move the
changes to the Go standard library (likely). But Go goes into 1.4
freeze in two days and it's the weekend.

Change-Id: I07117b7c5a7f58bd1e8abf6227c5fb0fa8047fa3
zip: add LastDataOffset field to zip.Writer.

For blobpacked.

Change-Id: I23c325d945b0eff2ec8f47259bc527f1d4f6031f
blobpacked: populate manifest.json chunks and verify them in tests

This switches to using a fork of archive/zip for now, until we can upstream
changes and wait a release cycle.

Plenty of TODOs remain, but making progress.

Change-Id: I3d7439dc405ba7f64b8d791952f1623a3a7e2024
Merge branch 'master' of https://camlistore.googlesource.com/camlistore
blobserver: add memory storage, mainly for tests.

Mostly code moved from pkg/test/fetcher.go

Also got rid of blob.MemoryStorage

http://camlistore.org/issue/416

Change-Id: I8d17b80942e2fde1c869f71738bfb8a67ed8ab31
Merge "blobserver: add memory storage, mainly for tests."
Merge "serverinit: add support for memory-only indexer. Enable in devcam."
First version of pinboard importer.

Change-Id: I2b06b20b2609981b03103b4a26c9c4633abe8473
schema: fix test I broken when removing GetChunkOffsets

Rewrite in terms of ForeachChunk instead.

Change-Id: Id018420c47ffe116246c92b06c61a9dc9bbbee62