implemented multi threading for increased crawling performance