Added ability to exclude URLs from crawl