Add an option to enable/disable crawling of pages

In certain scenarios you just need a list of URLs to be crawled, so I have added a flag which disables crawling and only relies on `options.urls`.