Issue 1:
The goal of this issue is to agree on basic design principles and expectations from this refactoring.

Goals

Speed up critical places in query processing by skipping bound checks, which ByteBuffer always performs when making any read or write. Bound checks could be hoisted outside the hot loops manually or enabled via assertions/static final boolean flag, that is effectively free with some precautions.

Get rid or 2 GB limitation, use long indexes. See #3743.

Don't make a lot of Object copies when a Memory object has to be "sliced", "converted to read-only", etc. See #3716 (comment).

+ Eliminate current Endian craziness in Druid by standardizing on native-endian. #3892 (comment)

Design

Based on DataSketches's Memory.

It is moved to Druid source tree, not used as a dependency. See #3892 (comment)

Memory object is immutable. "position and limit", if needed, are passed along with Memory as two primitive longs.

Upon creation a mutable Memory object has a cached immutable "view" object (which implements read methods and throws exception from write methods), this object is always returned when "as read only" is needed.

Explicit close() or free() on Memory is possible, but not strictly required, there is a sun.misc.Cleaner safety net, as well as in DirectByteBuffer.

While transition from ByteBuffer to Memory is in progress (not expected to be done all at once, but subsystem by subsystem, class by class see #3892 (comment)), and also when we need to interop with external libraries which require ByteBuffer, conversion from Memory to ByteBuffer and vice versa is possible. Likely it requires to

Use DirectByteBuffer-compatible format of Cleaner inside Memory

Be able to access private DirectByteBuffer constructors via MagicAccessorImpl.

+ "Global" (immutable) Memory's bounds are checked optionally via assertions/guarded by static final boolean flag, "local" position and limit are check explicitly manually, with helper methods or versions of read and write methods of Memory, which accept read/write position and "local" limits. #3892 (comment)

Objections, additions, corrections, questions are welcome.

@leerho @cheddar @weijietong @akashdw @himanshug @fjy @niketh

Issue 2:
@michaelschiff @gianm

Druid version 0.8.2

Segment Granularity: HOUR

My problem is very similar with this closed issue #145,

Around 40 minutes after an hour, I added a longSum metric and restarted the node.

when merging index,will throw NPE

java.lang.NullPointerException
at io.druid.query.aggregation.LongSumAggregator.combineValues(LongSumAggregator.java:39) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.query.aggregation.LongSumAggregatorFactory.combine(LongSumAggregatorFactory.java:78) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger$RowboatMergeFunction.apply(IndexMerger.java:1260) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger$RowboatMergeFunction.apply(IndexMerger.java:1236) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.collections.CombiningIterator.next(CombiningIterator.java:78) ~[druid-common-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:747) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger.merge(IndexMerger.java:351) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:227) ~[druid-processing-0.8.2.jar:0.8.2]
at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:216) ~[druid-processing-0.8.2.jar:0.8.2]

So I think this issue hasn't been resolved.

look into the following code of IndexMerger line number from 657 to 688.

boats.add(
new MMappedIndexRowIterable(
Iterables.transform(
indexes.get(i).getRows(),
new Function<Rowboat, Rowboat>()
{
@Override
public Rowboat apply(@Nullable Rowboat input)
{
int[][] newDims = new int[mergedDimensions.size()][];
int j = 0;
for (int[] dim : input.getDims()) {
newDims[dimLookup[j]] = dim;
j++;
}

Object[] newMetrics = new Object[mergedMetrics.size()];
j = 0;
for (Object met : input.getMetrics()) {
newMetrics[metricLookup[j]] = met;
j++;
}

return new Rowboat(
input.getTimestamp(),
newDims,
newMetrics,
input.getRowNum()
);
}
}
)

Build newMetrics base on the metric of input row. the previous indexes don't have new addition metric, So in the newMetrics,the value of new addition metric is null

for (Object met : input.getMetrics()) {
newMetrics[metricLookup[j]] = met;
j++;
}

modify the code of LongSumAggregator ,first check for not null,it work well.

static long combineValues(Object lhs, Object rhs) {
if(lhs == null){
lhs = 0L;
}
if(rhs == null){
rhs = 0L;
}
return ((Number) lhs).longValue() + ((Number) rhs).longValue();
}

I found that pull requst #2257 can solve NPE when adding metric.

But I add following unit test on Build 0.9.1-rc1,still throw NPE

@Test
public void testAddMetrics() throws IOException
{
IncrementalIndex index1 = IncrementalIndexTest.createIndex(new AggregatorFactory[]{
new LongSumAggregatorFactory("A", "A")
});
closer.closeLater(index1);
long timestamp = System.currentTimeMillis();
index1.add(
new MapBasedInputRow(
timestamp,
Arrays.asList("dim1", "dim2"),
ImmutableMap.<String, Object>of("dim1", "1", "dim2", "2","A",5)
)
);

IncrementalIndex index2 = IncrementalIndexTest.createIndex(new AggregatorFactory[]{
new LongSumAggregatorFactory("A", "A"),
new LongSumAggregatorFactory("C", "C")
});

index2.add(
new MapBasedInputRow(
timestamp,
Arrays.asList("dim1", "dim2"),
ImmutableMap.<String, Object>of("dim1", "1", "dim2", "2","A",5,"C",6)
)
);

closer.closeLater(index2);

Interval interval = new Interval(0, new DateTime().getMillis());
RoaringBitmapFactory factory = new RoaringBitmapFactory();
ArrayList<IndexableAdapter> toMerge = Lists.<IndexableAdapter>newArrayList(
new IncrementalIndexAdapter(interval, index1, factory),
new IncrementalIndexAdapter(interval, index2, factory)

);

final File tmpDirMerged = temporaryFolder.newFolder();

File merged = INDEX_MERGER.merge(
toMerge,
new AggregatorFactory[]{
new LongSumAggregatorFactory("A", "A"),
new LongSumAggregatorFactory("C", "C")
},
tmpDirMerged,
indexSpec
);
final QueryableIndexStorageAdapter adapter = new QueryableIndexStorageAdapter(closer.closeLater(INDEX_IO.loadIndex(merged)));
Assert.assertEquals(ImmutableSet.of("A","C"), ImmutableSet.copyOf(adapter.getAvailableMetrics()));

}

can reproduce NPE