chriseldredge/NuGet.Lucene — pull request #1 by PaulStovell (2013-03-22 06:39:57+00:00)
Title: Reduce memory usage during indexing

Previously, a folder with a thousand 4mb packages would consume over 600MB of memory during indexing. That's because the implementation relied on ZipPackage, which likes to load everything into memory. Then, it loads all file bytes into memory again just to calculate a file hash, which can just as easily be calculated from a stream (I suspect this came from NuGet.Server code).

I'm bypassing these parts of NuGet by loading the metadata from the ZIP manually via System.IO.Packaging. We never actually load the files into memory, so during indexing we only use 40MB or so. Indexing still takes about the same amount of time (mostly due to calculating the hashes, I think) but at least the memory usage is better.