make spider crawl in parallel with parsing threads