Refactor of nextToken method. Not tested on D&G Indexer yet.