sebsauvage/ZeroBin — pull request #20 by greggles (2012-08-02 15:26:20+00:00)
Title: Add robots.txt to request search engines not index content

I read the [wiki](http://sebsauvage.net/wiki/doku.php?id=php:zerobin) which mentions that search engines are prevented from reading content because they usually don't execute javascript. It seems that crawlers may some day start executing javascript. It's also not great, from a privacy perspective, to have the URLs indexed since that might lead to discovery of the URLs by a user who then clicks the link using a browser with javascript and they then see the actual content.

There's also the idea that having search engine and other crawlers on a site are a slight performance pain. Disallowing them should reduce the load on the server even if it's a small amount.