Add directive attributes to tokenizer.

See #140.
See #139.