speedata/gogit — pull request #4 by rakoo (2014-12-13 13:11:46+00:00)
Title: Make gogit work better with huge repos

`gogit` works really well with "standard" git repos, but I've tried to use gogit with [bup](https://bup.github.io/) repos where everything is orders of magnitude bigger. On my example repo, I have 822MiB worth of `.idx` alone in 357 files, which amounts to 30 million files for a grand total of a 178GiB repo.

My example program is somewhat similar to what `git rev-list` does: take a branch commit, output it's total size (commit + trees + blobs) and recurse for each parent. This involves a lot of searches in every index (the total number of elements touched is almost 9 million, ie I have other objects not related to this branch). I'd be happy to share this program if need be.

`gogit` currently builds a map for each index files, and this kills my process because it eats far too much memory (maps have a huge overhead). I've made a first attempt at copying the file (or parts of it) in memory and manually search (since git indexes are well organized, searching amounts to a targeted binary search so maps are actually unneeded). It barely worked (almost eat 1GiB of RAM) and took 8 hours. I also tried a naive approach where files are read on-the-fly. While it doesn't use as much RAM, it's still horrible performance-wise (lots of `file.ReadAt` in cpu profiling)

The current PR goes beyond that and replaces the manual copy of files with an `mmap` of the file. Since almost no memory management is done in the application, `pprof` reported half a MiB of RAM used by this application, and since files are completely cached, it behaves very cleanly in regard to disks: the result is that the same program only takes 40 minutes instead of 8 hours.

I have only one potential problem with this PR, it's that it involves an external dependency (and a system-dependent one), but it's such a boost in performance that I think it's worth it.