A complete 404 checker for URLs that also handles soft-404s