better handling of bad HTML and redirects