Disallow web crawlers in robots.txt. Fixes #197
Merge pull request #207 from ministryofjustice/robots.txt

Disallow web crawlers in robots.txt. Fixes #197