updates to robots.txt to exclude all api calls
removing pdf parsing entirely, simplifications to import