Docs: add docs on `robots.txt`

Added some basic documentation on the `robots.txt` file, in `doc/misc.md`.