Moved Robot check below scraped uris check