modding to use urllib2 for greater flexibility
minor
now crawls and adds to db in one
exceptions
record page being indexed