meta-toolkit/meta — pull request #30 by skystrife (2014-03-05 06:00:03+00:00)
Title: Add generic analyzers to index generation process

This is a major refactoring of what used to be called "tokenizers" into "analyzers.

Analyzers convert documents into (sparse) vectors of counts. A particular analyzer may (or may not) use a "token stream" to generate tokens to operate on. (`ngram_word_analyzer` does this, and is likely the most commonly used analyzer). "Token streams" or "filter chains" consist of a tokenizer followed by zero or more filters that add and remove tokens from the stream created by the tokenizer.

The defaults for token generation now consist of a tokenizer powered by the ICU library, a powerful, cross-platform implementation of many Unicode utilities (following closely with the Unicode standard). This tokenizer generates sentence boundaries and segments words according to the Unicode standard. This method is very robust and should be sufficient for most languages. This change, accompanied by a lightweight wrapper around ICU (meta-utf) makes MeTA Unicode-aware. All strings that MeTA works on internally are assumed to be in UTF-8: corpora may be optionally converted from their native format to this encoding if they are not already UTF-8 encoded (good examples are ceeaus, which is SHIFT_JIS, and 20newsgroups, which is LATIN-1).

This PR also introduces some simple factories to help contain the sprawling of analyzers and filters. Instead of having a single monolithic factory method, we now have a factory class (singleton) that creates analyzers and filters for us. Clients wishing to add functionality will register their classes with this factory and optionally provide a specialization of the factory method used to create their given object type (analyzer, filter, or tokenizer).