Removed Robots.txt. Tested multithreading. It works!