add a robots.txt parser, update web_scan to follow redirects, and re-add the masscan task