Robots.txt and cleaned up unneeded files and directories