added robots file to candidate web
added robots file to candidate web