Docs: added info about robots.txt to extend.md