enhance documentation to cover robot search path and --robot-paths

@stdweird/@wpoely86/@geimer: please review (focus on the content, I'll make sure the references and formatting is correct)
I would make this title more indented or something. It shouldn't be on the same level as the `Default robot search path` title IMHO.