Got retokenization working and updated unit tests.