Robots.txt: Added "allow" and fixed a few issues