Added an option to turn off http caching while crawling.