foursquare/twofishes — pull request #31 by jparise (2014-04-15 01:04:19+00:00)
Title: Preserve original token punctation.

Previously, the query parser would aggressively normalize the query
string, and all of the original punctuation would be lost. There are
cases where we want to preserve word-level punctuation characters,
however. For example: "carl's jr."

Now, normalize() only removes loose punctuation (i.e. punctuation
characters that aren't part of words). Then, when building the list
of tokens for name matching, we strip any remaining punctuation. This
lets us retain the original punctuation in originalTokens.

This adds a small additional cost to each call to parseQueryTokens()
(which is re-invoked for each retry attempt), but because the
expectation is that each call could provide a unique (or modified)
list of new tokens, it should be a fair tradeoff.

This change also fixes a bug where the config.maxTokens would be
applied to the entire original token list and not just the parsable
token list.