adding multi doc tests for corpus and tokenizer edge cases