robots and sitemap take precedence over globals

This is essentially hoop-jumping for the redirector tests, which may or may not
be semantically correct. Does it make sense to have a sitemap for a site with
no URLs (because there would be no mappings), or should it just use the global
status? Once the redirector's gone we could free ourselves of this particular
shackle.

The specific failing tests were related to www.jobcentreplus.gov.uk/sitemap.xml
from test_static_assets.