core/apiRouter: add /robots.txt route to disallow crawling