added functionality to remove odd characters
changed so mid word punctuation (such as apostraphe or dash) is no longer word break for tokenization