Web crawler

A web crawler or spider is a computer program that automatically fetches the contents of a web page. The program then analyses the content, for example to index it by certain search terms. Search engines commonly use web crawlers.

Related pages
 HTTrack – a web crawler released in 1998

References

Category:Web browsers