Add a robots.txt handling to request.py