GETting robots.txt sometimes takes more than 3 seconds.