Implement tokenization in a C extension.