The parse-engine then instructs the tokenizer to initiate the tokenization phase. Tokenization is an incremental process, and can interrupt when the scanner is blocked awaiting network data. The tokenizer controls and coordinates the tokenization of the input stream into a collection of CTokens. This continues until EOF occurs on the input stream, or an unrecoverable error occurs. After the tokenization phase completes, the parses enters the token iteration phase which validates the document and causes a content model to be constructed.
Token iteration proceeds until an unrecoverable error occurs, or the parser has visited each token.
The tokens are collected into related groups of information according to the rules provided by the nsDTD class. The DTD controls the order in which tokens can appear in relation to each other. At well defined times during this process, the parser notifies the content sink about the parse context, instructing the sink to construct the document according to the state of parser. Once tokenization and iteration have concluded, the objects in the parse system are destroyed to conserve memory. This can be useful in cases where preprocessing needs to occur, or where transforms from foreign document types into HTML should occur.