updated crawler that fixes bug that makes it pickup other content types than HTML, see #289