edit spiders in order to incrementally crawl