adding default robots, humans and crossdomain