Take care of gzip when requesting the robots.txt