added some tracking of whether a url has been examined and was found to be OK to use