add active resource validation, and robots.txt validation
fix to sitemap parsing