Update robots.txt file on production deploys

Issue 3581: Update robots.txt file on production deploys

https://code.google.com/p/otwarchive/issues/detail?id=3581