Added checks to parse 'robots.txt' for the host to identify permission levels for the crawler in that website and to cache those results