adding more rules to non-crawlable links, follow redirects and fix sub-directoryurls