luceneplusplus/LucenePlusPlus — pull request #82 by vslavik (2014-11-20 13:16:41+00:00)
Title: Fix crash in StandardTokenizer if wchar_t is 32bit

StandardTokenizerImpl::getNextToken() uses a map to classify characters;
the map is 16bit, because this code was ported from Java and Java's
strings use UTF-16. But Lucene++ uses wchar_t, which may be 32bit on
some platforms (Unix, OS X), resulting in crashes when encountering
characters above U+FFFF.

Fix the crash by classifying anything outside of BMP as a character.
This is hardly ideal, but it prevents the crash and gets the job done
reasonably well (as StandardTokenizer is intended for European languages
in the first place). A proper fix would rework the map to correspond to
UTF-32 rather than UTF-16, or would port Unicode tokenizer from Lucene
3.1+.

Fixes #57.

(It would be great if this could be merged as at least temporary fix for the crash described in #57. I realize it’s not perfect, but it *does* fix an otherwise unavoidable crash and at worst results in some poor tokenization (in situations that Lucene 3.0’s StandardTokenizer wasn’t meant for in the first place).)