rewrite tokenize state 4 and 5 used with double array hash.