indexing the whole archive was getting into oom problems