Robots checker function complete with exception of site not found