Port more Python tokenizer updates to C.