Add better errors

Lets use the same error handling as the other applications for consistency
Merge pull request #44 from alphagov/add-better-errors

Add better errors
Add DFID to robots exclusions
Handle IDF type documents