Same algorithm from lucene's StandardTokenizer but in C++.