Further clean up of rules, and added robots.txt.