Removed BDB and streamlined code for Lucene. Lower memory overhead now
(250M over 2.7million files rather than 1.5GB). Slightly faster (20
minutes) too.