squizlabs/PHP_CodeSniffer — pull request #207 by bpearson (2014-04-30 10:49:28+00:00)
Title: Separating the tokeniser from CodeSniffer

I have done the code shuffle so the tokeniser can be separated from CodeSniffer core (and I would like this functionality outside of CodeSniffer), so far the files, you need Tokenizer.php, the Tokenizers directory and Tokens.php. To call the Tokeniser, you just need:
$tokenizer = new PHP_CodeSniffer_Tokenizers_PHP();
$tokenizer->setVerbose(TRUE|FALSE);
$tokenizer->setTabWidth(INT);