Issue 92: Created ShallowWebCrawler and DeepWebCrawler that are optimized for either deep crawls (store on disk) or shallow crawler (store in memory)
Revert "Added IsDeepCrawlEnabled, not used by crawled yet"

This reverts commit 9e2a11c1250b0106ccc39cdf4f65e54449450dd5.