Added the robots exclusions.