In order to create a robots.txt file without much effort, there are now several tools that you can use. One of these tools is the robots.txt generator from ryte.