Reinterface

new tokenizer interface may be nicer in practice.