Now ensures that robots.txt agent matching gives priority to the most
specific match (as opposed to the first match).  Sitemaps detected in
robots.txt are also preserved for sitemap resolving.