don't follow subdomains. clean up crawler test.