Allow basic swapping out of robots.txt file

For #1554 

My cursory research didn't give me a good sense of what wants to go into the public `robots.txt` file, as far as a list of public items. This will at least swap in a private `robots.txt` (which disallows robots from hitting all routes) when ALLOW_ROBOTS is set to something falsy ('false', 'f', 'no', 'n', or '0')

Any updates for public content can just go into the `robots/public_robots.text` file without changing anything else, though.