Modified gitignore and added robots.txt.