olivernn/lunr.js — pull request #69 by robrigo (2014-01-29 21:39:36+00:00)
Title: Preserve tags during tokenization

The regex substitutions on each token would truncate the open or closing
bracket of a tag, as they aren't discluded by \W. This caused issues for me
when I was trying to get my custom htmlStripper() pipeline function to work,
which strips out HTML tags to provide more accurate results, because it
wouldn't be able to match partial HTML tags. I propose that these be added
to core so anyone who could be indexing tokens that include tags won't have
to dig into the tokenizer and patch it themselves.