HippoPHP/Tokenizer — pull request #43 by rr- (2014-11-30 09:29:43+00:00)
Title: Removed useless state from Tokenizer

1. `Tokenizer::tokenize($buffer)` acts like a static method and returns iterator for tokens.
2. We have `Tokenizer::getTokenList()` that returns the token list that happened to be produced in last call to `::tokenize()`.
3. `Token::getTokenList()` has two issues:
1. Its name is incorrect, because it refers to result saved from `Tokenizer::getTokenList()`, so it should be at least `::getLastBuiltTokenList()`
2. Now that it name reflects what it does, don't you think it's just too silly? What is this method good for anyway? Caching? In our case, we already cache it in lazy initializers of HippoPHP's `CheckContext`.

I changed `tokenize()` signature to a static method, and removed obscure state manipulation. I updated `::tokenize()` doc comment to reflect correct return type.