getnikola/nikola — pull request #1250 by Aeyoun (2014-04-19 01:58:43+00:00)
Title: New ROBOTS_EXCLUSIONS option for sitemap exclusions and new robots.txt generation

List of files relative to the server root (!) that will be asked to be excluded from indexing and other robotic spidering. * is supported. Will only be effective if SITE_URL points to server root. The list is used to exclude resources fro /robots.txt and /sitemap.xml, and to inform search engines about /sitemapindex.xml in a standard way.

The robots plugin can be merged into the sitemap plugin if its desirable to keep the plugin count low.