Fixed the OpenTSDB bug that selected older metrics over newer ones among duplicates. (https://rbcommons.com/s/pepperdata/r/5381)
The bug prevented gagg from updating metrics with new values. The fix was using
HBase timestamp when tsd selects cells to drop. In addition, the newly created
compacted cell gets the timestamp of all the participating KeyValue records.

The bug described at https://pepperdata.atlassian.net/browse/PA-466 There are
race conditions between Gagg insertion and TSDB compaction.

Scenario A:
1. Gagg aggregates metrics globally and inserts into TSDB.    2. Gagg aggregates
another set of metrics with in the hour and inserts    into TSDB again.    3.
Then, TSDB compacts two cells of Gagg metric of the hour into a new    cell and
deletes the old cells.    4. Gagg observes metrics from late comers.    5. Gagg
aggregates metrics again for the same time range and inserts    into TSDB. They
are basically duplicated data points.    6. TSDB now sees two cells for the same
hour, so it tries to compact    cells.    7. While compacting, TSDB sees
duplicated data points and throws away    the new ones.    8. Then, TSDB deletes
the compacted Gagg cells.    9. As a result, users never see the fresh
aggregation of Gagg.

TSDB logs Guenther reported:
>>     I see lots of this with gagg.
I think some hosts are coming late and the gagg is trying to update datapoints.
>>     2014-11-19 06:16:42,888 WARN  [New I/O worker #4]
CompactionQueue: Found duplicate data: cell=Cell([86, -21], [68, -86, 64, 0]),
delta=1390000, prev cell=Cell([86, -21], [64, -128, 0, 0]), last_delta=1390000,
in row key=[0, 1, -19, 84, 108, 7, -80, 0, 0, 16, 0, 8, -11], row size=4, cell
count=841 -- Skipping the new one.

Scenario B:
The different order in 5, 6, and 7 steps of Scenario A
1. Gagg aggregates metrics globally and inserts
into TSDB.    2. Gagg aggregates another set of metrics with in the hour and
inserts    into TSDB again.    3. Then, TSDB compacts two cells of Gagg metric
of the hour into a new    cell and deletes the old cells.    4. Gagg observes
metrics from late comers.    5. At the same time, TSDB now sees two cells for
the same hour, so it    tries to compact cells.    6. Gagg aggregates metrics
again for the same time range and inserts    into TSDB. They are basically
duplicated data points.    7. After compaction, TSDB deletes the compacted Gagg
cells, which    deletes the last data Gagg inserted right before. TSDB never
sees the new    data.    8. As a result, users never see the fresh aggregation
of Gagg.

Scenario C:
It's also worth pointing out that these bugs don't occur
solely due to race between write and compaction. The bug can happen for reads
after writing duplicate data points. The same compaction code is exercised on
read path to consolidate data points when scanning rows. That's why Guenther hit
scenario A bug even after disabling compaction.

The fix for the scenario A was using HBase timestamp to select fresh data points.
In addition, we gave the biggest timestamp of participating cells (KeyValue records)
to the result cell of compaction so that we could prevent, in a later compaction,
discarding the new data points injected during the current compaction.
The fix for the scenario B was deleting data points before the participating cells
so that any fresher data points won't be deleted.
The fix for the Scenario A and B also fixs the bug of the scenario C.