amplab/graphx — pull request #75 by ankurdave (2013-11-23 11:10:26+00:00)
Title: Simplify GraphImpl internals

GraphImpl and VertexSetRDD were previously split up into multiple RDDs to improve data reuse. However, this required many zip joins and hurt readability. This PR consolidates the internal data structures, making the following major changes:

* VertexSetRDD is backed by a single RDD of VertexPartitions instead of an index and a values RDD. Users can still extract a reusable index from this RDD using `VertexSetRDD.index`.
* `RDD.zipPartitions` is given a `preservesPartitioning` option similar to mapPartitions. This was not previously necessary because only the index RDD in VertexSetRDD needed an explicit partitioner, and it could be produced without calling zipPartitions. This is a change to Spark core and is submitted upstream as apache/incubator-spark#198.
* Common operations on VertexSetRDDs are extracted into the `mapVertexPartitions` and `zipVertexPartitions` methods.

Minor changes in the PR:
* For consistency with the rest of the codebase, the type parameters in VertexSetRDD are renamed using the "VD" convention.
* The `VertexSetRDD.aggregate` method is generalized from `AggregationMsg[VD]` to `Product2[Vid, VD]` to reduce code duplication.
* In VertexPartition, the BitSet representing the valid indices in the vertex attribute array is renamed from "bs" to "mask" to reflect its role as a mask over the vertex array.
* Vid2Pid is renamed to VertexPlacement to reflect the move to a pid-to-vid mapping within each vertex partition.
* VTableReplicatedValues is renamed to VTableReplicated to reflect the fact that it now contains an index and the replicated values.
* `localVidMap` is removed as a first-class RDD in GraphImpl. Instead, it is generated and immediately used to create VTableReplicated.

Resolves #71.