tinkerpop/blueprints — pull request #308 by vadasg (2012-09-29 13:27:10+00:00)
Title: bypass BatchGraph cache for repeated out vertices when using presorted input data (pull request moved from master branch to batchloading)

This was previously a pull request on the master branch.

Edge Input data are often presorted or can easily be sorted before batch loading using BatchGraph. The current cache implementation in BatchGraph significantly speeds up loading presorted data, but for large cache sizes (or if the cache is turned off), a further performance gain can be had by bypassing the cache when loading a series of edges that have the same outgoing vertices. The cache is bypassed for only the outgoing vertex lookup, and only after it is a repeat of the previous outgoing vertex in the previous edge added.

I've added a few lines of code to BatchGraph that implement this. In a simple test loading of 20 million presorted edges (with an average of 10 edges per vertex), this gave an overall performance boost of 3%, with most of the gains coming from the end of the loading process. The performance gain for the last million edges loaded was over 5%; it should give even more of a boost when loading much bigger data sets, or sets that have high numbers of edges per vertex. I observed no performance penalty when loading data that are not presorted.