Added comprehensive corpora processing. Various code clean-ups and improvements