datagovuk/dgu_d7 — pull request #125 by rossjones (2013-10-15 15:12:29+00:00)
Title: Updated robots.txt to include some more CKAN pages to not be indexed.

We really don't want /qa/linkchecker to be indexed as it will just end up generating emails for me, and the resource_cache should *never* be indexed as we don't want them to turn up in a search engine as they are often (deliberately) out of date pages.