Added code for doing in-memory preprocessing on a large corpus with multiple-threads