Fix urlparser bug

Found a page, www.airscorp.com, which did not return a content type.
Now if there isn't a content type, the bot will try to parse as if
it was an html or xml page.