squizlabs/PHP_CodeSniffer — pull request #208 by bpearson (2014-05-01 04:28:09+00:00)
Title: Separating the tokeniser from CodeSniffer

Take Two! Here is the tokeniser changes again, but this time against the fixer branch. Same deal:
I have done the code shuffle so the tokeniser can be separated from CodeSniffer core (and I would like this functionality outside of CodeSniffer), so far the files, you need Tokenizer.php, the Tokenizers directory and Tokens.php. To call the Tokeniser, you just need:
$tokenizer = new PHP_CodeSniffer_Tokenizers_PHP();
$tokenizer->setVerbose(TRUE|FALSE);
$tokenizer->setTabWidth(INT);
$tokenizer->setEncoding(ENCODING);