[core] extract robots: noindex check to meta plugins