I'll see if I can copy a piece of a file in here for you to see. I have seen on several of the add urls for search engines that they now say they respect this file. Another reason to use it is to keep a spider out of the cgi bin area because spiders can get caught there and keep running. Yes, this is how the SEs find the pages on your site. It could have overloaded his site.
Frames are another question.
But frames do make it harder for the SE spiders to find your information. Yes, they prefer non frames. What I can do is put some of this information on a page and give you an address to look at later. To do as an extensive a robots file for various spiders you have to have all the information. I assume the maker of the software would have updates.