guardian/robots.txt — pull request #2 by rrees (2013-10-23 12:12:26+00:00)
Title: Discussion: Block all discussion prefixes from being crawled

Discussion threads are currently available under the url structure:

www.theguardian.com/discussion/

This is causing issues for the bot, partly because this content doesn't exist on desktop and partly because we need to provide a sensible search context for this.

Therefore this changes robots.txt to hide the whole prefix from the bot for now.