Issue #2181. Added a robots.txt
Make the reading of robots.txt asynchronous
Added more endpoints to robots.txt and a test
Simplify the robots.txt logic, use the same logic as humans.txt

* Add Somsubhra Bairi to the CONTRIBUTORS list
Merge branch 'robots-txt-cached' into dev