Other than pep8 nits lgtm and runs with righteous speed against stage.

    py.test -v --baseurl=http://download.allizom.org -k test_robotstxt_exists
    ============================= test session starts ==============================
    platform darwin -- Python 2.7.2 -- pytest-2.2.4 -- /Users/mbrandt/.virtualenvs/23f9b98114a9622d/bin/python
    http://download.allizom.org
    collected 277 items 
    
    tests/test_redirects.py:145: TestRedirects.test_robotstxt_exists PASSED