Update generated robots.txt

Looks like the default robots.txt need to be updated. Added `/api` and removed some old routes.