DKF-638: added "generateRobotsFile" configuration setting to Sitecore Si...

Our project has a large multi-site SiteCore installation (79 sites and counting...) where each site are completely different financial and legal entities and therefore required separate "robots.txt" files per site, rather than one containing references to all the different sitemaps. With this new configuration setting we can choose to generate "robots.txt" files somewhere else, we do it using a custom HttpRequestProcessor (source not included).