Only disallow the API in the robots.txt