By virtue of being essentially a browser, it knows how to speak HTTP to servers and how to dissect URLs. This is quite helpful on a jaunt through Cyberspace. Generally, if a language has network connectivity and parsing abilities, it is a good candidate for robot creation. With programming ease, however, comes much overhead. As your robot becomes more complex, you may outgrow your chosen language.
Most robots are simple wanderers, surfing links only to discover resources in webspace.
Other, more complex, robots construct elaborate keyword databases of links and present them to their users. WebCrawler is an excellent example of a complex robot. Robots can be used for other purposes such as link validation and site-mirroring. WT is a simple robot whose only purpose in life is link discovery. It traverses web sites and keeps a database of the links pointed to by these pages.