Handle hashtags mention and stock symbols

Also add a test for parsing MAC files
treat URLs as single words
Changes to shingling regex and normalization
move normalization code to a separate method
convert to lowercase after HTML un-escaping
add some comments
move BNMI function to main module, add tests