Revert "allow robots"

This reverts commit a47d58c0ac1e61762cf2dc98c0914dddaf38584f.