Web crawlers are automated programs that trawl the Internet for new and accessible resources, adding their discoveries to a cataloged database accessible by Internet users. The Web crawler then compares the words entered in the Search Box with that in millions of webpages, gathers a list of webpages that best match the search terms, adds all possible related resources and, finally, organises all that information on the Search Results page for your easy viewing. The value of Web crawlers lies in that you can find information on your requested topic within a website not specifically dedicated to that topic. For example, the Web crawler might discover information on the Melbourne International Film Festival in a website about arts events in Australia. Directories only contain submitted information, arranged topically.
Directory editors select submissions, determining placement and organisation of the directory.
Directories list only the specified information you request and are a better bet for ferreting specific information. Rather than trying to remember the addresses of the various search engines, it s easier to bookmark these resources. Search engines are likely to attract repeat usage and bookmarking them allows you to return easily at the click of a mouse. The Internet houses so much information that it cannot be indexed like a telephone directory. Too much information is being added to the Internet every minute from all over the world.