Added robots.txt
Disabled resources on dev environment and updated readme to explain