Cleaned and updated version of token-processor support

This is the cleaned up version of pull request https://github.com/scikit-learn/scikit-learn/pull/1537 to fix #1156.