added robots.txt creation & option for new index