fixing tokenize problems

Added new parameters for tokenization and adapted the query parsers for
it.