js updates and robots.txt additions