added support for 'allowdomains' which will allow the scraper to follow non-root domains
formatting