Added a DisallowedPages field to limit the spider.