Fixed bug in the intranet crawler where credential was not correctly set when fetching robots.txt file.