Agent.allowed_error_codes - list of error response codes which should not raise an error to facilitate easier testing of error pages.
Move robots logic to avoid repetition.
Merge pull request #248 from lrowe/allowed_error_codes

Agent.allowed_error_codes