Add support for web crawler advices aka meta robot.