better tokenization, allow arbitrary rules