[BZ 1135629] do not load the full index into memory during data aggregation

This commit also includes some clean up, removing more unused classes and
adding some javadocs.