Test first if the robot can index or not the page and then if the met "robots" is coherent