Improvements to crawling (should always gracefully exit).  Experiments with gensim and NLP.