graphlab-code/graphlab — pull request #103 by realstolz (2013-11-25 07:16:53+00:00)
Title: fix bug and perf issue when parallel loading multiple files

Fix bug in oblivious and improve the performance of random/grid/pds in loading phase.

The performance of oblivious is very poor (even slower than forbidden parallel loading),
since there is a bottleneck (lock) in add_edge().

There is two possible solution (but both are complex)
1. parallel dht in add_edge
2. move heuristic operation from add_edge() to finalize()

Evaluation:
Input: LiveJournal (split to 48 pieces)
Cluster: 6 node (24-core 64G RAM 1G Ethernet)
#N	orig	now	partition
1	79.71	8.48	grid
2	8.48	4.40	grid
3	14.27	2.98	pds
4	9.09	2.37	grid
5	failed	37.67	obli
6	5.11	1.77	grid