They are certainly not the traffic producers that you have been told they are. Search engines are important, but they should only be a small part of your marketing efforts. You need to be able to submit to them, then forget them. Your web site is in competition, for a top listing, with every one of them. If there is one thing I have learned about robots, it is that there is absolutely no pattern to them.
Most robots are stupid and wander randomly.
Then they come back a week later, ask for the same thing and then go away, again. This happens over and over again for months. I have never figured it out. If they wanted to see if the website was really a web site, they could just Ping it. This would be much faster and much more efficient.