Detect and skip duplicate tweets

Repeated spidering has caused a built-up of repeated tweets where we've
relaunched the spider for different users.
Allow for certain types of tokens to be ignored when converting text to a vector