Latte: class MacroTokenizer renamed to MacroTokens (bc break?)
Tokenizer: key 'line' in token changed to 'offset' (BC break)
Tokenizer: keys 'value', 'type' and 'offset' changed to numeric keys and added constant Tokenizer::VALUE, Tokenizer::TYPE and Tokenizer::OFFSET (BC break, you should use these constants)
Tokenizer: tokenize() always captures offsets, so it returns array of arrays, even in simple mode (BC break)
Tokenizer: divided into Tokenizer and TokenIterator with completely new interface and tests (yes, BC break, se stim smir)