fixed bug when robots was read from cache.
limit crawling to a depth of 1 (ex. /a/b/).