boundlessgeo/GeoGit — pull request #511 by groldan (2013-11-28 17:36:59+00:00)
Title: Performance and database size improvements

There are several patches in this pull request that together bring substantial performance improvements specially to the import of large datasets and commit operation.
- There's a new "prescribed storage order" for tree nodes, based on the FNV-1a hash algorithm instead of SHA-1, which makes building trees much faster.
NOTE this change makes 0.6+ repositories incompatible with previous versions, but the wins are worth it.
- The default (BDB JE) staging database operates in non transactional mode by default, with a 512MB write back log to sync to disk.
- Both the object and staging BDB JE databases use a single writer thread for bulk operations, while encoding the blobs in the calling thread
- ImportOp uses up to nCores/2 to fetch subsets of the whole dataset and import them in parallel. The shapefile datastore in GeoTools 9 does not support paging so the performance improvement is more noticeable with JDBC based data sources (PostGIS, Oracle, etc).
- When importing, the RevTree nodes are first serialized to a custom set of files in the system's temporary directory and the created by inserting the nodes in the tree builder in a sorted way according to the node storage order, making the tree building much faster and producing far less dangling intermediate trees.