Added full crawling along with download of contents