Add a robots.txt handling to request.py

Like it says. Just adds a very simple robots.txt.

I'm running into troubles as I poke at adding twitter cards, because it thinks it's not allowed based on our current "robots.txt" that doesn't have real content.