Added document indexing to the tokenizer and some comments