Issue #2181. Added a robots.txt
Make the reading of robots.txt asynchronous
Added more endpoints to robots.txt and a test
fix the CSP error in the helpers.isFunction test
Merge pull request #4051 from mozilla/issue-4050-csp-failure

fix the CSP error in the helpers.isFunction test

fixes #4050
Simplify the robots.txt logic, use the same logic as humans.txt

* Add Somsubhra Bairi to the CONTRIBUTORS list
Merge branch 'robots-txt-cached' into dev