Setting base urls and adding robots.txt