This checkin adds a new, non-default shutdown mode, in which we don't
don't write all the timelines to the db, but instead write a blob
containing all the start times in the accumulators.  This will
provide a much faster shutdown/startup sequence in prod.

This checkin will be followed by unit tests that exercise the
capabilities.

util/src/main/java/com/ning/arecibo/util/timeline/ShutdownSaveMode.java:

    An enum defining the two shutdown modes

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:

    Add config parameter saying which shutdown mode we're using.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorModule.java:

    When stopping the core, call forceCommit(true) to tell it we're
    shutting down.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    In the LoadingCache listener onRemoval() method, if we're doing
    fast shutdown, saving only start times, add to them to the
    StartTimes instance instead of saving the chunk to the db.

    In the scheduled forceCommit(), pass false indicating it's not a
    shutdown.

    In replay(), if there is a StartTimes instance for the
    hostId/categoryId, it's time to filter events.

    In forceCommit(), if we're doing fast shutdown, create and bind a
    StartTimes instance, and after the accumulators are invalidated,
    save the start timesin the db.

collector/src/main/resources/collector.sql:

    Create the last_start_times table, which will never have more than
    1 row.

util/src/main/java/com/ning/arecibo/util/timeline/StartTimes.java:

    New class to hold the map from hostId to map from category id to
    time, used to preserve the start times from host timeline
    accumulators.

util/src/main/java/com/ning/arecibo/util/timeline/StartTimesBinder.java:
util/src/main/java/com/ning/arecibo/util/timeline/StartTimesMapper.java:

    New units that bind and map StartTimes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:
collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Add methods insertLastStartTimes(StartTimes startTimes) and
    deleteLastStartTimes();

util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:
util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Provide the sql templates and binders for the new TimelineDAO
    methods.

collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:

    Call forceCommit(false), because it's not testing a shutdown.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineHostEventAccumulator.java:

    Add a TODO mourning the high cost of
    TimelineHostEventAccumulator.addHostSamples().
This checkin adds bulk insert operations for all the tables.  These
help a lot in running the TimelineLoadGenerator, but the timeline bulk
insert operations are also needed to provide acceptable performance in
prod.

pom.xml:

    Update jdbi version to 2.33, because Brian fixed a bug a reported.

collector/pom.xml:

    Remove the explicit jdbi version - - inherit it from the master
    pom.

collector/src/main/resources/collector.sql:

    Insert timeline_times and timeline_chunk sample rows.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:
collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:

    Add bulkInsertHosts(), bulkInsertEventCategories,
    bulkInsertSampleKinds, bulkInsertTimelineChunks, and
    bulkInsertTimelineTimes.

util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:

    Add new methods, and also if hostId is null in
    getOrAddSampleKind(), don't add it to the
    hostIdsSampleKindIdsCache.

util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Add new methods, but in the case of bulkAddTimeline*(), in a
    single transaction, get the highest id, and base the ids of the
    records to be added on that highest id.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunk.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimes.java:

    Add new constructors that take an objectId and an existing
    instance, and clones the instance with the new object id.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesBinder.java:

    In the bind code, bind the id to null if it's zero, and to its
    value if non-zero.  That way we get autoincrement assignment if
    the id is zero.

util/src/main/java/com/ning/arecibo/util/timeline/CategoryIdAndSampleKindBinder.java:

    New binder needed for bulkInsertEventCategories.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesMapper.java:

    Move MAX_IN_ROW_BLOB_SIZE to here.

util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:
util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Add templates and bind methods for the new bulk methods.
Add method returning boolean true if the sample represents a default value for the type; false otherwise
Initial checkin of TimelineLoadGenerator, which at this point only does inserts
In looking at statistics from xnb3, it became clear that the sampling
time compression is so good that there is only downside in sharing
sampling time information between timeline chunks.

So this dev-dstryker checkin is a major overhaul, eliminating
TimelineTimes and TimelineChunkAndTimes and all the associated
machinery.  After the checkin, all unit tests pass.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Aggregate directly into TimelineChunks.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.java:

    Remove TimelineTimes methods, and choose more appropriate names.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    Replace use of TimelineChunkAndTimes with TimelineChunk.

collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java:
collector/src/test/java/com/ning/arecibo/collector/AccumulatorConsumer.java:
collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:
collector/src/test/java/com/ning/arecibo/collector/TestTimelineChunkAndTimes.java:
collector/src/test/java/com/ning/arecibo/collector/persistent/TestDefaultTimelineDAO.java:
collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineAggregator.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestCollectorEventProcessor.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestInMemoryCollectorEventProcessor.java:

    Replace use of TimelineChunkAndTimes* with TimelineChunk*.

    Conform to new TimelineChunk signature.

collector/src/main/resources/collector.sql:

    Delete table TimelineTimes.

    Add attributes not_valid and aggregation_level to table
    timeline_chunk, and modify indexes appropriately.

    Insert a test record that can be used by the health check.

collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:

    Query the TimelineChunks to get aggregation candidates.

    Remove templates for TimelineTimes, and change names
    appropriately.

collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Replace use of TimelineChunkAndTimes* with TimelineChunk*.

    Remove TimelineTimes methods

collector/src/test/java/com/ning/arecibo/collector/persistent/TimelineLoadGenerator.java:

    Remove references to TimelineTimes.

    Conform to new TimelineChunk constructor signature.

util/src/main/java/com/ning/arecibo/util/timeline/CSVSampleConsumer.java:
util/src/main/java/com/ning/arecibo/util/timeline/StartTimesMapper.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimeRangeSampleProcessor.java:

    Conform to move of static methods to DateTimeUtils.

util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:

    Remove TimelineTimes methods.

util/src/main/java/com/ning/arecibo/util/timeline/SampleCoder.java:

    Change scan() signature: now takes TimelineChunk and a
    SampleProcessor.

    Conform to new TimeCursor method signatures.

util/src/main/java/com/ning/arecibo/util/timeline/StartTimesBinder.java:

    Fix typo - - referred to TimelineTimesBinder.

util/src/main/java/com/ning/arecibo/util/timeline/TimeCursor.java:

    Changme TimeCursor() constructor to take a byte[] times instead of
    a TimelineTimes.

    Change the signature of consumeRepeat(), but it's still broken.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineCacheManager.java:

    Remove stuff about TimelineTimes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunk.java:

    Remove property timelineTimesId and add properties notValid,
    aggregationLevel and times, and make corresponding constructor
    changes.

    Move the getSamplesAsCSV and friends here from
    TimelineChunkAndTimes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAccumulator.java:

    Conform to new TimelineChunk constructor(), and add overloading of
    extractTimelineChunkAndTimes that takes a List<DateTime>.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:

    Bind the new attributes of TimelineChunk.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkMapper.java:

    Parse out the new attributes of TimelineChunk.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineCoder.java:

    In compressTimes(), flush the right stream.

    Provide new method compressDateTimes(List<DateTime>).

util/src/main/java/com/ning/arecibo/util/timeline/TimelineHostEventAccumulator.java:

    Conform to new extractTimelineChunkAndReset() signature.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineOpcode.java:

    Don't use 0xff and 0xfe as opcodes - - it's less confusing to use
    0x7f and 0x7e.

util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Remove references to timeline_times.

util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCoder.java:

    Remove TimelineTimes stuff.

util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCompression.java:

    Create real times, because the new accumulator
    extractTimelineChunkAndReset() signature needs them.

util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAccumulator.java:

    Conform to new scan() signature.

    Remove refs to TimelineTimes and *AndTimes* clases.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAndTimesConsumer.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAndTimesDecoded.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAndTimesToJson.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunksAndTimesViews.java:

    Renamed to remove the AndTimes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkConsumer.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkDecoded.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunksViews.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkToJson.java:

    Renamed units.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAndTimes.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimes.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesBinder.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesMapper.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineTimes.java:

    Deleted

util/src/main/java/com/ning/arecibo/util/timeline/DateTimeUtils.java:

    New unit to hold static methods that convert to/from unix time.

util/src/main/java/com/ning/arecibo/util/timeline/TimesAndSamplesCoder.java:

    New unit to hold static methods that combine times and samples
    byte arrays.

util/src/main/java/com/ning/arecibo/util/timeline/TimeBytesAndSampleBytes.java:

    New unit to hold times and samples byte arrays.
Fix comments, JavaDoc and a few method names that erroneously refer to TimelineTimes
Provide a much cheaper, counter-based way of figuring out when we have to add null samples to an accumulator
Rename getInMemoryTimelineChunk to the more accurate getInMemoryTimelineChunks
Rename getInMemoryTimelineChunk to the more accurate getInMemoryTimelineChunks
This checkin makes batched background db writes work, which we need to
achieve an adequate db insert rate, required for prod deployment.  It
also spreads writes out over the entire collection period, rather than
burst-writing at a single point in the hour.

It also eliminates use of LoadableCache from TimelineEventHandler.  We
don't want the accumulators for a host to ever be evicted, and there
is a downside to the opacity imposed by Loadable cache.

With this checkin, all the unit tests pass.  However, we don't yet
have unit tests for the new background writing functionality, because
all the tests currently write in the foreground.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:

    Add the attributes that control the background writing:
        int backgroundWriteBatchSize
        TimeSpan backgroundWriteCheckInterval
        TimeSpan backgroundWriteMaxDelay.
        boolean performForegroundWrites

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorModule.java:

    Add method configureBackgroundDBChunkWriter() which hooks up the
    background writer to the lifecycle.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    Make accumulators a ConcurrentHashMap rather than a cache,
    supported by new synchronized method getOrAddHostEventAccumulator(),
    and use the new method in processSamples()

    Remove bogus accumulatorsMonitor, which was only protecting some
    accumulators accesses.

    Use eventCategoryIds in the maps; they are cheaper than the
    strings.

    In getInMemoryTimelineChunks(), include the pending chunks in the
    chunks returned if they fit in the time window.

    Replace the LoadingCache metrics with CounterMetrics, and also
    provide counters for the rest of the functionality.

collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java:

    Log an error if we get an exception writing the StreamingOutput.

collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:

    The first hostId should be 1, not 0.

collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:

    Use the injected BackgroundDBChunkWriter.

collector/src/test/java/com/ning/arecibo/collector/TestModulesFactory.java:

    Cause the BackgroundDBChunkWriter to write in the foreground for
    tests, if the relevant property is not set, because otherwise the
    tests have to have sleeps in them.

collector/src/test/java/com/ning/arecibo/collector/TestTimelineHostEventAccumulator.java:

    Import change for the move of TimelineHostEventAccumulator.

collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineAggregator.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestSamplesReplayer.java:

    Call accumulator method extractAndQueueTimelineChunks()
    rather than extractAndSaveTimelineChunks()

collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineEventHandler.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestInMemoryCollectorEventProcessor.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestFileBackedBuffer.java:

    Supply new BackgroundDBChunkWriter arg to TimlineEventHandler
    constructor.

collector/src/test/java/com/ning/arecibo/collector/process/TestCollectorEventProcessor.java:

    Supply new BackgroundDBChunkWriter arg to TimlineEventHandler
    constructor.

    Comment out a few asserts that only make sense in the face of
    cache eviction of accumulators, which we no longer do.

collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java

    Supply BackgroundDBChunkWriter arg; use extractAndQueueTimelineChunks();
    in sendSamples(), call getOrAddHostEventAccumulator() rather than new'ing one.

util/src/main/java/com/ning/arecibo/util/timeline/HostTimelines.java:

    Unused; deleted.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAccumulator.java:

    Add some debug logging.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineHostEventAccumulator.java:
collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineHostEventAccumulator.java:

    Move TimelineHostEventAccumulator from util to collector, because it needs
    to access the background writer.

util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Fix bug in bulkInsertTimelineChunks template.

collector/src/main/java/com/ning/arecibo/collector/persistent/BackgroundDBChunkWriter.java:

    New unit

collector/src/main/java/com/ning/arecibo/collector/persistent/PendingChunkMap.java:

    New class to hold the TimelineChunks for one host event category
    and time.
Add a raft of TODOs needed to allow the TimelineAggregator to compress time and sample sequences across timelines
Fix bugs from the previous fast shutdown and background write checkin.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorModule.java:

    Don't have the BackgroundDBChunkWriter listen for the STOP event,
    because only the TimelineEvent handler knows when to shut it down.

collector/src/main/java/com/ning/arecibo/collector/persistent/BackgroundDBChunkWriter.java:

    Rename finishBackgroundWritingAndExit with the more descriptive
    shuttingDown.

    Fix bug: The CounterMetric counters should be associated with
    BackgroundDBChunkWriter, not TimelineAggregator.

    Provide boolean method getShutdownFinished() for
    TimelineEventHandler to wait on.

    Rename finishBackgroundWriting() to initiateShutDown().

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Spell subtract correctly.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    Provide a Comparator that sorts first by hostId, then by
    sampleKindId, then by startTime, used to sort memory chunks.

    Add AtomicBoolean shuttingDown()

    startTimes doesn't need to be in an AtomicReference().

    In handle(), if shuttingDown is true, just increment a counter.

    After collecting memory chunks, sort them using the new Comparator.

    In forceCommit(), if shutdown is true, set shuttingDown to true,
    and at the end of the method, loop waiting for the
    backgroundWriter to shut down.

collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:

    Conform to background writer method rename.

util/src/main/java/com/ning/arecibo/util/timeline/TimesAndSamplesCoder.java:

    Provide static method getEncodedLength(TimelineChunk) to get the
    length of the bytes of encoded stuff.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:

    Decide whether to use the in-row samples or the blob samples by
    getting the size from TimesAndSamplesCoder.getEncodedLength()
This checkin fully encaches hosts, event categories and sample kinds
at startup, and kepts the caches updated across operations that create
new ones.

This checkin also fixes a problem with bulk timeline insertion: that
insertion cannot use auto increment columns.  Instead, use an
AtomicLong in DefaultTimelineDAO to supply chunk ids.

util/src/main/java/com/ning/arecibo/util/timeline/HostIdAndSampleKindId.java:
util/src/main/java/com/ning/arecibo/util/timeline/HostIdAndSampleKindIdMapper.java:

    New units used by the new TimelineDAO method,
    getSampleKindIdsForAllHosts.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    The initial value of the sampleTimelineId should be 0, not -1.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    Make getInMemoryTimelineChunks() be synchronized.

    In getInMemoryTimelineChunks(), for each host event collector, do
    the times compression once, not once for each chunk.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineHostEventAccumulator.java:

    When creating pendingChunkMaps, for each host event collector, do
    the times compression once, not once for each chunk.

    Fix typos in error messages.

collector/src/main/resources/collector.sql:

    Remove auto_increment in the sample_timeline_id for
    timeline_chunks.

collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:

    Add currently unused templates that might provide a more efficient
    way to query for agregation candidates.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:

    Change insertTimelineChunk to return a Long, and change
    bulkInsertTimelineChunks to return List<Long>.

    Add method getSampleKindIdsForAllHosts().

collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:

    Maintain a Map<Integer, Set<Integer>> to keep records of the
    correspondence between hosts and the sampleKindIds they support.

    Conform to TimelineDAO method return value changes, and implement
    the new method.

util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Since the auto increment stuff can't work when there are
    operations like bulkInsertTimelineChunks that need to insert
    multiple records, instead maintain a chunk id counter which is
    initialized with the highest chunk id in the timeline_chunks
    table.

    Change the methods that insert into timeline_chunks to get the
    ids from the new counter.

    Remove unneeded method-level synchronize operations.

    Conform to TimelineDAO method changes.

util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:

    Eliminate all the LoadableCaches, because we don't ever want to
    evict hosts, event categories, sample kinds and timeline host
    accumulators.  Replace them with BiMaps and Maps, and synchronize
    the methods that access the maps.  At startup, load up the maps
    with _all_ values, and update the maps as new hosts, event
    categories and sample kinds are encountered.

    Conform to TimelineDAO method changes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAccumulator.java:

    Remove now-unreferenced overloading of
    extractTimelineChunkAndReset().

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:

    We already have the time and samples bytes, so don't create them
    again.

util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:
util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Add getSampleKindIdsForAllHosts().

util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCompression.java:

    Compress bytes so before calling
    accum.extractTimelinechunkAndReset().
Rename sampleTimelineId to chunkId, and sample_timeline_id to chunk_id.
Check in small fix for getInMemoryChunks()
This checkin reverts explicit setting of chunk ids in
bulkInsertTimelineChunks(), because none of the callers actually care
about the ids.  With this checkin, timeline aggregation seems to
work properly, but in-memory samples are broken for some reason.

collector/src/main/resources/collector.sql:

    Restore auto_increment treatment of chunk_id.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:
util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:
collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:

    Change bulkInsertTimelineChunks to have a void return value.

util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Eliminate the AtomicLong used to initialize chunkIds, and don't
    use it when inserting TimlineChunks.

    Change bulkInsertTimelineChunks to have a void return value.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:

    Eliminate a check for 0 chunkIds - - 0 is the normal case; only
    tests make it non-zero.
Fix the problem with in-memory samples.  The bug was outputing memory
chunks before db chunks in writeJsonForAllChunks().  Since the db chunks
have earlier times, they get rejected because the memory chunks have already
established later times.

This checkin reverses the order so db chunks are output first, then memory
chunks.  With this change, I don't see any problems with dashboard graphs
that span db and memory chunks.
This checkin does batching of TimelineAggregator chunk creates and
invalidates/deletes.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:

    Add int parameter maxChunkIdsToInvalidateOrDelete, defaulting to
    1000.  This means that by default, we'll batch up chunk inserts
    and deletes or invalidates whenever the count of chunks aggregated
    exceeds 1000.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Add some counters to distinguish between queued write operations
    and the actual writes.

    Add a pair of lists, one to hold new chunks produced by
    aggregation, and the other to hold the chunkIds of aggregated
    chunks to be deleted or marked not valid.  In
    aggregateHostSampleChunks(), add the new chunk and the ids of the
    chunks aggregated to these lists, and if the list of ids exceeds
    the threshold, call performWrites().

    When we're done with an aggregation level, of there are any writes
    queued, call performWrites().

    Add new method performWrites(), which in a single transaction,
    inserts the new chunks and deletes or marks not valid the old
    chunks.

    Fix bug: Remove redundant timelineChunksCreated.inc().

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.java:

    In makeTimelineChunkValid(), change the chunkId to be a long.

util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    In insertTimelineChunk, explicitly initialize aggregation_level
    and not_valid, rather than letting the table default be used.
This checkin adds timeline and sample coding support sufficient to
allow the TimelineAggregator to combine the compressed encodings
intelligently, rather than just concating them.  This greatly
compresses aggregations of default-valued timelines.  Since this is
fairly tricket code, this checkin also adds a ton of unit tests for
combining.

It also optimizes the basic search query for chunks to aggregate.
This should reduce the number of chunks scanned in prod by a factor of
6 or more.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Use the new coder combine() methods to compress aggregated
    time and sample byte stream.

    Pass chunksToAggregate to the dao method that gets candidate
    chunks.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.java:

    Remove now unused insertNewInvalidTimelineChunk.

    In getTimelineAggregationCandidates, bind chunksToAggregate.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineHostEventAccumulator.java:

    Treat opcodes as ints, since byte is signed and we have opcodes
    above 127.

    The max repeat count is now MAX_SHORT_REPEAT_COUNT.

collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:

    Remove never-used -getHostIdsAndSampleIdsOfAggregationCandidates.

    Update getTimelineAggregationCandidates to only return chunks for
    which aggregation is possible.

util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Remove unused imports.

util/src/main/java/com/ning/arecibo/util/timeline/RepeatSample.java:

    There are two sample repeat codes, one allowing up to 255 repeats,
    and the other allowing up to 65535 repeats.

    Add incrementRepeatCount(int addend), because combining timelines
    means combining repeats.

    Overload getOpcode() to return either the BYTE or SHORT opcode,
    depending on the repeat count.

util/src/main/java/com/ning/arecibo/util/timeline/SampleCoder.java:

    Make this instantiable, holding the state and methods necessary to
    do sample coding, moved here from TimelineChunkAccumulator.

    Add compress() and decompress() methods for sample lists.

    Add coder and scanner support for both BYTE and SHORT sample
    repeat opcodes.

    Add new method combineSampleBytes(), which combines and compresses
    a list of timelines.

    Use DataInputStream.read() when we read a byte; it has a cleaner
    API.

util/src/main/java/com/ning/arecibo/util/timeline/SampleOpcode.java:

    Make opcode index values be ints, not bytes, to side-step signed
    byte problems with opcodes > 127.

    Add BYTE and SHORT versions of the repeat opcodes.

util/src/main/java/com/ning/arecibo/util/timeline/ScalarSample.java:

    Add an equals() overloading so it can be used in tests.

util/src/main/java/com/ning/arecibo/util/timeline/TimeCursor.java:

    Add support for scanning both BYTE and SHORT repeats.

    Fix bug: for unrepeated delta times, it read one ahead.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAccumulator.java:

    Move all the coding machinery to base type SampleCoder.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineCoder.java:

    Move MAX_DELTA_TIME to TimelineOpcode.

    Add support for both BYTE and SHORT delta repeaters.

    Add major new method: combineTimelines(), called by the
    aggregator.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineOpcode.java:

    Add support for both BYTE and SHORT delta repeaters opcodes.

    Make the non-delta-valued opcodes  run from 0xFF down.

    Change the max delta time to 240, which leaves 13 unused opcodes
    above it.

util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCoder.java:

    Add comprehensive tests of combining sample byte sequences and use
    of the SHORT repeat facility.

util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAccumulator.java:

    Add tests of BYTE and SHORT repeaters.

util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineCoder.java:

    Add comprehensive tests of combining time byte sequences and use
    of the SHORT repeat facility.
collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    If we didn't make new aggregates at the current level, there is no
    point in trying later levels, so exit the loop.

util/src/main/java/com/ning/arecibo/util/timeline/SampleCoder.java:

    The aggregator was getting errors because the combiner was using
    sample value equals(), which will return true for objects of
    different types.  Add a new static method sameSampleValues() which
    only returns true if the objects have the same class.

util/src/main/java/com/ning/arecibo/util/timeline/ScalarSample.java:

    Use new static method SampleCoder.sameSampleValues()