Ignoring Robots.txt restores connectivity