added robots.txt and blocked all robots so dev can proceed without crawlers getting bad data.