utility script for indexing all of the corpora
improvements to end-of-document detection in clean.py