Added a description of the pre-tokenization code.