some work on token processing