Allow DFID finder to be crawled by robots