handle follow redirects in crawl for links to listed hosts (including via proxy)
ignore pid lock file