also generates robots.txt file now
generates robots.txt file now and gives index.html a priority of 1