The types of data that are stored in the database, and the method in which that data is search provide some of the keen distinctions between different web search engines. The nature of these databases and the behavior of search engines are changing all the time. Search engines get thier information from robots.
Robots are small computer programs that know how to do one particular task or set of tasks.
Most search engnes use robots , or variations of them known as 'spiders,' to roam the web, and gather information about what the robot finds. Typically, a robot starts at the top level of a website, or it's homepage. The robot sends information about that page to the search engine's database. Next, it finds all the links on that homepage and follows them. The robot will mill through the entire website, until it has found each page, always sending information about the pages it finds.