But, just because the possibility exists doesn't mean you should take on this task unprepared. A Robot is nothing more than an executable program. It can be in the form of a script or a binary file. It first makes a connection to a web server and requests a document be sent, much the same way a web browser works. The difference is in the automa tion provided by the robot.
Many robots often parse the html for links, and recursively follow the selected links.
Sometimes these reasons were robo t specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. The very nature of the World Wide Web is diversity and this very diversity makes robot writing both important and increasingly difficult. There is no one right h tml. They can be written in many ways and in many formats. My suggestion is to get the spec sheet for html and practice, practice, practice, making your robot robust.