only can make robots.txt if disallowing some directories, otherwise it's better to have none