robots.txt improvements
deal with bad incoming store links