bug 866873 - robots.txt accordingly

Disengage robots by default.

NOTE: If we want Googlebot to spider our production site we need to override this default setting to `True` in prod's `settings/local.py`