groonga/groonga — pull request #193 by naoa (2014-08-15 06:02:58+00:00)
Title: push grn_token_mode to ctx and  add tokenizer token skip/skip with position for tokenizer

I want to get grn_token_mode in the tokenizer.
And, I want to skip the token and position.
Please merge.

[[groonga-dev,02637](http://sourceforge.jp/projects/groonga/lists/archive/dev/2014-August/002639.html)]