tokeniser uses state table