added support for disallowing robots via config