I can't think of a way to test tokenization in isolation without making it public in some way, and the unit test is specifically for that private method.  All I did here was move the private method call to a helper method.