robots.txt is no longer required