Added nagios check for robots.txt (HTTP code 200 && no 'Disallow /' line)