Use robots.txt in Net::HTTP test