shogun-toolbox/shogun — pull request #1209 by van51 (2013-07-04 16:17:30+00:00)
Title: Changed CDelimiterTokenizer behavior on consecutive delimiters

Now it won't skip consecutive delimiters, but if it reaches one it will return the same indices, start and end, on the SGVector.