collector: fix test exception

22839f032c5a0e3502dbe0e1e5822b6294bdc987 broke the TestModulesFactory
by not installing the EmbeddedJettyJerseyModule module.

Reported-by: David Stryker <dstryker@ning.com>
Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
collector: fix test exception

22839f032c5a0e3502dbe0e1e5822b6294bdc987 broke the TestModulesFactory
by not installing the EmbeddedJettyJerseyModule module.

Reported-by: David Stryker <dstryker@ning.com>
Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
This checkin, in branch dev-dstryker, adds an event_categories table
that maps string names of event categories into integer category ids,
as well as the caches for the mapping.  Everywhere that we used to use
event categories in the timeline mechanisms, we now use event category
ids.

This checkin also fixes a bug in timeline aggregation; provides for
per-aggregation-level counts of timelines to aggregate; and increases
the max aggregation level to 3.

Finally, this checkin adds an end_time column to timeline_chunks table
and TimelineChunks class.

All unit tests pass with these changes.

util/src/main/java/com/ning/arecibo/util/timeline/CategoryIdAndSampleKind.java:

    New unit that encapsulates an event category id/sample kind pair.

collector-client-support/src/main/java/com/ning/arecibo/collector/rest/DefaultCollectorClient.java:

    Add a fix for a Sun JDK compile error.

collector/src/main/java/com/ning/arecibo/collector/guice/CachingDefaultTimelineDAOProvider.java:

    Conform to CachingTimelineDAO constructor arg change.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:

    Add a parameter for the max number of event categories.

    Change maxAggregationLevel to 3.

    Add a parameter that gives, for each aggregation level, the number
    of chunks we must find in order to aggregate.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Use eventCategoryIds rather than eventCategories.

    Supply endTime to the TimelineChunk constructor.

    Accomodate the differing number of chunks to aggregate for
    different aggregation levels.

    Fix bug that caused different eventCategoryIds to be aggregated
    together.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:

    Conform to the change from category to categoryId.

    Pass accumulatorEndTime to extractTimelineChunkAndReset().

    getOrAdd the eventType to get the id.

    Pass the id to getOrAddSampleKind().

collector/src/main/java/com/ning/arecibo/collector/process/EventsUtils.java:

    Delete getSampleKindFromEventAttribute().

collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java:

    Conform to the changed args and return values of dao methods that deal with sampleKindIds.

    Every place we used to accept a query parameter that specified
    sampleKind, now accept eventCategory,sampleKind.

    Add a new endpoint, getHostSamplesUsingIds(), and cause all the
    existing sample endpoints to translate host names to host ids, and
    eventCategory,sampleKinds into sampleIds, and call the new method.

    Change writeJsonForAllChunks to accept ids rather than strings for
    hosts and sample kinds.

    Add @VisibleForTesting method findSampleKindIdsForHosts.

    Add private utility methods to translate lists of hosts and
    eventCategory,sampleKinds into ids.

collector/src/main/resources/collector.sql:

    Add table event_categories.

    Add event_category_id column to sample_kinds.

    Add end_time column to timeline_chunks.

collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:

    Replace event_category with event_category_id.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:

    Add methods for event categories.

    For sample kind methods, conform to the use of eventCategoryId.

collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:

    Synchronize get methods, since the insert methods are synchonized.

    Add support for event category methods.

    Conform to the other TimelineDAO method changes.

collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:
collector/src/test/java/com/ning/arecibo/collector/TestTimelineChunkAndTimes.java:
collector/src/test/java/com/ning/arecibo/collector/TestTimelineHostEventAccumulator.java:
collector/src/test/java/com/ning/arecibo/collector/persistent/TestDefaultTimelineDAO.java:
collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineAggregator.java:
collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineEventHandler.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestCollectorEventProcessor.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestInMemoryCollectorEventProcessor.java:
collector/src/test/java/com/ning/arecibo/collector/process/TestSamplesReplayer.java:
collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java:

    Conform to sample kind and event category changes.

util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:

    Add support for event category methods, and add caches and
    monitored getters for event categories.

    Conform to the other TimelineDAO method changes.

util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:

    Conform to the TimelineDAO method changes.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunk.java:

    Support endTime.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAccumulator.java:

    Conform to the addition of TimelineChunk.endTime.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkAndTimes.java:

    Replace eventCategory with eventCategoryId, and add endTime to
    TimelineChunk constructor call.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:

    Bind endTime.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkMapper.java:

    Add endTime to TimelineChunk constructor call.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineHostEventAccumulator.java:

    Replace use of eventCategory with eventCategoryId.

    When logging sampleKind, also long eventCategoryId.

util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimes.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesBinder.java:
util/src/main/java/com/ning/arecibo/util/timeline/TimelineTimesMapper.java:

    Replace use of eventCategory with eventCategoryId.

util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:

    Add eventCategory methods.

    Pass eventCategoryId into the sampleKinds get and add methods.

util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg:

    Add eventCategory templates.

    Pass eventCategoryId into the sampleKinds get and add templates.

util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCoder.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestSampleCompression.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAndTimesToJson.java:
util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineTimes.java:

    Pass eventCategoryId instead of eventCategory.

util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAccumulator.java:

    Pass eventCategoryId instead of eventCategory.

    Add endTime arg to extractTimelineChunkAndReset().
Fix braino in TimelineAggregator that caused timeline chunks to be dropped if they weren't the first for a given sample kind
Merge branch 'dev-dstryker' into collector-ng
Merge branch 'collector-ng' of github.com:ning/Arecibo into collector-ng
Fix definitional issue in aggregation loop: should loop while < aggregationLevel, not while < aggregationLevel - 1
Revert "Fix definitional issue in aggregation loop: should loop while < aggregationLevel, not while < aggregationLevel - 1"

The right way to fix this is a gepo property change to increase the max aggregation level, not changing the loop.

This reverts commit dc15930b3d4f9c1a17a00176894d4623e31928e9.
This checkin improves the indexes used by timeline aggregation.

collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:

    Change the default timeline length from 10 minutes to 60 minutes.

    Add parameter saying whether aggregation is enabled.

collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:

    Change interpretation of aggregation levels to be more
    self-consistent.

    Fix bug: don't try to aggregate if there are no chunks.

collector/src/main/resources/collector.sql:

    Improve the indexes used for aggregation on timeline_times and
    timeline_chunks.

collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:

    When getting sample chunks to aggregate, order by start_time, not
    timeline_times_id.

util/src/main/resources/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.sql.stg

    Whitespace fix.
util: fix getEventCategoryIdAndSampleKind query

Add missing SQL and mapper for CategoryIdAndSampleKind.

I've also added a unit test for it in TestDefaultTimelineDAO.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Merge branch 'collector-ng' of github.com:ning/Arecibo into collector-ng
util: introduce CategoryAndSampleKinds

CategoryAndSampleKinds captures all sample kinds for a given category.
This new POJO is used by the dashbaord.

Update the collector client library to return it for all sample kinds
calls.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
dashboard: initial prototype

A new page at http://127.0.0.1:8080/static/arecibo.html
will prompt the user for the type of data to graph.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
collector: update /host_samples endpoint for the dashboard

Before, the endpoint was returning:

[ {
  "hostId" : 8,
  "sampleKindId" : 2900,
  "samplesAsCSV" : "..."
} ]

The id are useless for the dashboard. Instead, return:

[ {
  "hostName" : "box.company.com",
  "eventCategory" : "ActiveMQBroker_localhost",
  "sampleKind" : "TotalConsumerCount",
  "samples" : "..."
} ]

Note that we still pass CSV samples to the dashboard. We should change it to pass
the compressed timeline times and data ("compact view") to speed up data retrieval.
Most of the wiring is there and the decompression methods are shared (util submodule).

I've also updated the collector client to use the /host_samples endpoint exclusively.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
collector-client-support: fix typo in DefaultCollectorClient

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
collector-client-support: fix getHostSamplesAsStream call

Add missing category_and_sample_kind parameters.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
dashboard: iteration on landing page

Use local storage to save the start and end times.
We should also consider turning on persistence for the two checkbox
trees.

Integrate with the graphing page by making the Graph button redirect
with the right query parameters.

Hide more icons in the checkbox trees.

Extract common code in js/utils.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
collector-client-support: fix another typo in DefaultCollectorClient

A call to .getSamples() was missing when merging the samples.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
dashboard: add graph.html page

The graph.html page takes care of the rendering.

The url structure is identical to the host_samples resource
which should give us access to the raw data as well as
permalinks/bookmarks for free.

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Merge remote-tracking branch 'origin/collector-ng' into metrics.base-pom-0.1.0-update

Conflicts:
	collector-client-support/src/main/java/com/ning/arecibo/collector/rest/DefaultCollectorClient.java
	collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java
	collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java
	dashboard/src/main/java/com/ning/arecibo/dashboard/resources/CollectorResource.java
	util/src/main/java/com/ning/arecibo/util/timeline/TimelineHostEventAccumulator.java
	util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkAndTimesToJson.java

Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>