.gitignore updated, robots.txt added