improved the crawler using workers