molgenis/molgenis_apps — pull request #245 by joerivandervelde (2012-11-16 15:20:43+00:00)
Title: Deleted robots.txt so apps get spidered again and added blocking to WormQTL

Deleted robots.txt
Added Ant function to create robots.txt if needed
Using this function in the WormQTL build script
WormQTL now uses the option to block webcrawlers

--> this is because spidering this app will cause it to slow down for a long time
maybe we should disallow only certain API's which serve out big content so the site gets nicely spidered without being slow