[core] extract function to check noindex for robots directive