some improved resource handling.  Added robots.txt and sitemap.xml