Add a robots.txt file

This robots file allows complete access to webcrawlers except to the
/search/ folder.