Make the reading of robots.txt asynchronous