update tokenizer unit tests
remove dual perceptron from classifier unit tests
add config file checking to create_config in inverted_index_test
Merge branch 'unit-test' into develop
remove unused files
doxygen work on meta::index
Eliminate strict-aliasing violations.
Rename existing "tokenizers" to "analyzers".
Merge branch 'develop' into analyzers
Add tokenizers as the first step of the analyzer filterchain.

Idea is to go from a tokenizer through a set of filters, with the analyzer
using the output of the final filter as its tokens for analysis.
Add english_normalizer filter.

This filter is likely to go immediately after a whitespace_tokenizer, and
tries to normalize all whitespace and splits words from punctuation
characters.
add ASSERT_EQUALS function in unit_test.h
use ASSERT_EQUAL where possible
add ASSERT_LESS, ASSERT_GREATER and ASSERT_BINOP
use new ASSERTs where possible
Add sentence boundary filter.
Add length filter.
move test into include and src
Merge branch 'unit-test' into develop
Add "Doc" as sentence end exception.