Made changes to crawler. Added celery for scaling the crawl process.