Add states for the tree builder
Allow passing RUSTFLAGS on the command line

Also switch to := (immediate) make variable assignment.
Add missing Makefile dependency
Invoke the test runner at top level for external tests
Benchmark the tokenizer
Refactor tokenizer to prepare for per-state loops
Don't re-dispatch on tokenizer state when it hasn't changed

Speeds up tokenization of webapps.html by 15%, and of plain text by 25%.