internetarchive/heritrix3 — pull request #78 by nlevitt (2014-07-18 04:12:35+00:00)
Title: treat content as html and extract links if it looks like html, even if m...

...ime type belies that

Some sites like http://85broadscornell.webs.com/ have no content-type header