groonga/groonga — pull request #209 by kou (2014-10-02 14:53:02+00:00)
Title: Support token filters

It is a big change. :<

This change introduces new proc: `GRN_PROC_TOKEN_FILTER`. Key supported tables can have zero or more token filters. Token filters are applied for each token.

Tokenize sequence:

Normalize -> Tokenize -> (For each token: Apply token filters) -> Index/Search

This change includes some changes of internal core:

* Export `grn_token_mode`
* See 32935ce875b325c72ae99c72bdf97455303b6604 for details.
* Introduce `GRN_TOKEN_DONE_SKIP` that shows the last token is skipped.
* If the first `grn_token_next()` call returns `GRN_TOKEN_DONE_SKIP`, it means that there are no valid tokens. The string should be ignored. `grn_ii_select` ignores a search by the string.
* See e681cca8f838a91c92008f3d6136dd563197aa42 for details.

This approach saves token filters into spec not header of each table. (Tokenizer and normalizer are saved into header.) Because token filters are just needed when table is opened. It isn't needed to notify token filters change to other processes. So spec is reasonable save location.

This change adds the following features:

* Add `--token_filters` to `table_create` command.
* Add stop_word token filter. It uses the same approach of https://github.com/naoa/groonga-tokenizer-yangram .

TODO:

* Improve token filter plugin API
* Define callback types instead of using `grn_proc_func`