move robots .txt checker
fix issue wuth cookie if not present