update assignment/rule tokenizer to use the new scheme based on token_list[]