Added more endpoints to robots.txt and a test