new tokenizer reuses attributes - rwrite concept filter
a bit to work with that and add test cases for some cases
where token possibly part of a prefix need to be pushed
back in the queue