molgenis/molgenis — pull request #66 by joerivandervelde (2012-10-12 13:08:41+00:00)
Title: Blocking webspiders in frontcontroller and big merge

Web spiders (crawlers, robots..) are now blocked due to problems with WormQTL being taken down by such programs. They are also blocked with a new 'robots.txt' file in molgenis_apps but that is not always refreshed or respected. Later on we shoulde make this fine-grained instead of a global block: e.g. allow the homepage and formscreens to be spidered, but not big data downloads.