Decoupled tokenizer from detection strategy

This opens up possibilities to swap the default token_get_all()-
based tokenizer for another one, or to add tokenizers for other
programming languages.