GALAXY 745 Fixes for the case of indefinate crawls for a non existant
site

Also resolve the case where a server did not provide a content-length
header, the lib would get in a tangle and cause jobs to fail, this was
caused by a bug in Typheous, Had to work around that bug in Typheous
http request lib that made the headers return a complete copy of the
headers hash  when the header that was asked for was not present.