add legacy url to index
trim some fields on scrape
extract scraper out of seedSearch
improve error handling during spidering