fix for #1443

The urlpatterns in mysite/urls.py will have a pattern for robots.txt.
This pattern will have the Disallow directive set to / if settings.DEBUG
is True. Else, it will be set to an empty string. We need a test for
this. I would like to do it if you can help me with it. I referenced
this code
https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/urls.py
while writing this fix.