Add more cleanup functions
Missing new lines
Add rules to robots.txt

Disabled, need testing