I can do that, but this is the interface I plan to expose to Block#parse when implemented in C.  It will get the tokenizer struct from the object, then call tokenizer_next with it.  Since that function takes a token struct as output, everything in here would get used.