We are using the materials we collect to create a public library for researchers, historians, and scholars. The Archive collaborates with institutions including the Library of Congress and the Smithsonian . Web Robots A Web-crawling robot is software that automatically collects Web pages from publicly accessible Web servers. It examines each page for links to other pages that it can collect. In turn, if it finds more links on those pages, it follows those too.
Crawling is how most search engines collect Web pages for indexing.
We have plans to augment those crawls with other data sources. Alexa's robot currently gathers more than 100 gigabytes of publicly available information a day, often from thousands of different sites at a time. See Alexa for more information about the technology . Unlike people, who can follow any link when they browse the Web, robots do not, because Webmasters and authors use certain standards to control access to their sites by robots. Donations to the Internet Collections Alexa Internet donates its crawls of publicly accessible Web pages to the Internet Archive.