added some checking and helper method to TextTokenStream