get urls from meta tags too
add 5xx and 4xx to reports