groonga/groonga — pull request #191 by naoa (2014-08-14 07:20:39+00:00)
Title: tokenize: support mode

Ngram tokenizer splits with different rules by ``GRN_TOKEN_GET``/``GRN_TOKEN_ADD`` mode.
I want to debug the different modes in the tokenize command.