added support for global crawl - crawl outside url_root