added test cases from JE
fixing off-by-one error in read_tokens_mixed