[robots] Add support to robots.txt files.