This results in a lower level of service to others using the system. Certain web browsers, however, can cause rapid fire, by retrieving all images on a page at the same time. Robots used in large-scale search engines usually do not put forth such a strain in a system. Not only do they run on large scale servers, they are written better, and do not involve a rapid fire to the system they are monitoring. A majority of robots are well designed, professionally operated, cause no problems, and provide a valuable service in the absence of widely deployed better solutions.
Robots can decide on where to visit in a number of ways.
In general they start from a historical list of URLs, such as a list of documents, server lists, or a list of popular sites on the web. Most large search engines allow users to enter their own URLs manually, which will then be visited by the robot. Search engines have become more user-friendly over the years. Originally, search strings were limited to using keywords, and the system would search for pages containing each word, and index them to the user. Boolean searching has been added to most search engines, and many have advanced functions which let you go into better detail what you are searching for.