added redirects in htaccess and started ignoring robots.txt