concrete5/concrete5 — pull request #1703 by ahukkanen (2014-04-04 11:42:22+00:00)
Title: Update to robots.txt

Added tools URLs that are requested through "index.php" to the robots.txt.

This is because links to tools urls generated with Loader::helper('concrete/urls')->getToolsURL('tool_name') contain the dispatcher filename if "URL_REWRITING_ALL" is not set to "true".

This causes google to index these tools URLs that contain "index.php".