updated this to work with 1.9

ALso includes skipping of non-html stuff, like pdf or csv or images.