amplab/graphx — pull request #137 by ankurdave (2014-04-14 19:23:51+00:00)
Title: Unify GraphImpl RDDs + other graph load optimizations

This is work in progress. The PR also includes hotfixes apache/spark#367 and apache/spark#368. The main change is in 3c7a6bc6c6c5fc2b6ae85d7d757135621c38318b.

This commit makes the following changes:

1. *Unify RDDs to avoid zipPartitions.* A graph used to be four RDDs: vertices, edges, routing table, and triplet view. This commit merges them down to two: vertices (with routing table), and edges (with replicated vertices).

2. *Avoid duplicate shuffle in graph building.* We used to do two shuffles when building a graph: one to extract routing information from the edges and move it to the vertices, and another to find nonexistent vertices referred to by edges. With this commit, the latter is done as a side effect of the former.

3. *Avoid no-op shuffle when joins are fully eliminated.* This is a side effect of unifying the edges and the triplet view.

4. *Join elimination for mapTriplets.*

5. *Ship only the needed vertex attributes when upgrading the triplet view.* If the triplet view already contains source attributes, and we now need both attributes, only ship destination attributes rather than re-shipping both. This is done in `ReplicatedVertexView#upgrade`.