Code cleanup.
Use given HTTP auth data for robots.txt fetching.