moved threading handling into CrawlerBase
save from laptop