List of Web archiving initiatives
Web archiving initiatives

List of Web archiving initiatives

Section Title: Web archiving initiatives
Table Section Text: None

Name	Country	Creation Year	Technologies	Number of Employees	Comments

Full-time	Part-time

EU web archive	European Union	2013	Heritrix, Wayback	1	The EU web archive contains the main websites of the EU institutions, which are hosted on the europa.eu domain and subdomains.

Australia's Web Archive	Australia	1996	PANDORA Digital Archiving System (PANDAS), Heritrix, Bamboo, NLA Trove, HTTrack, Webrecorder, outbackCDX.	4	>10	The National Library of Australia leads the 'PANDORA' component of the Australian Web Archive which takes a selective approach and is a collaborative program of 10 agencies providing curatorial input.

PROMISE project	Belgium	2017	The PROMISE project is a two-year project that will explore the policy-related, legal, technical and scientific issues related to archiving the Belgian web.

PageFreezer.com	Worldwide	2009	PageFreezer's Deep Web Crawler, Hadoop, Cassandra, Elastic Search	40	Enterprise class SaaS solution for website & social media archiving. Provides automatic collection, replay, full-text search and data export of websites, blogs, social media and enterprise collaboration platforms for eDiscovery and regulatory compliance with FDA, FINRA, FSA, SEC, Federal Rules of Evidence, FOIA and records management laws.

WebPreserver.com	Worldwide	2015	WebPreserver	Chrome webbrowser plugin and web-based service to collect authenticated, legally-admissible webpages & social media pages for eDiscovery. Web snapshots can be exported in EDRM-XML, WARC, PDF and native HTML.

OoCities — GeoCities Archive / GeoCities Mirror	Germany

Web@rchive Austria	Austria	2008	NetarchiveSuite, Heritrix, OpenWayback	1

Deutsche Nationalbibliothek	Germany	2012	Tools of oia GmbH	3	The crawling for the selective web archive is done by the German company oia GmbH. The access is restricted to the reading rooms of the German National Library.

DILIMAG (Digital Literature Magazines)	Austria	2007	WebCurator	2	One technician, one for collecting and metadata.

Bibliothèque et Archives nationales du Québec (BAnQ)	Canada	2012	Heritrix, Wayback.	4	2 librarians, 2 developers

Web Archiving Program at Library and Archives Canada	Canada	2005	Heritrix and Wayback.	8	Web archiving in Canada is a legislated activity that is conducted for digital preservation purposes under section 8 (2) of the Library and Archives of Canada Act.

Web Information Collection and Preservation - WICP (Chinese Web Archive)	China	2003	Heritrix, Wayback and NutchWAX.

Croatian Web Archive (Hrvatski arhiv weba - HAW)	Croatia	2004	Crawl: DAMP software, Heritrix Access: Wayback, Lucene	2	2	The Croatian Web Archive (HAW) is a collection of content harvested from the Internet. In 2004 the Archive started as a concept of selective capturing of web resources.

Webarchiv (National Library of the Czech Republic)	Czech Republic	2000	Heritrix, Wayback and Seeder.	5	2	Czech web archive (Webarchiv) maintained by National Library of the Czech Republic focuses on archiving the Czech national web. Acquisition policy consists of three lines: selective harvests (collection of resources based on selection criteria), topic collections (focused on significant topics in the area of the Czech web) and comprehensive harvests (automatic harvests of content on the national domain).

Netarkivet	Denmark	2005	NetarchiveSuite, Heritrix, Free text search using Solr, and Wayback. Developing search frontend and playback engine SolrWayback.	~ 23 = 7.5 FTE	singapore

Estonian Web Archive	Estonia	2010	Heritrix, Squidwarc, Wayback.	4	Since 2006 the Legal Deposit Law allows the National Library of Estonia to collect Estonian websites as legal deposit copies. Harvested content was permitted to make publicly available.

Finnish Web Archive	Finland	2008	Heritrix, Solr, Wayback.	2	>2	Maintained by the National Library of Finland. Annually, all *.fi domains are harvested, as well as web servers located in Finland.

BnF - BnF Web Legal Deposit	France	2006	Heritrix, Wayback, NutchWAX, NetarchiveSuite, BCWeb.	10

Ina (Institut National de l'Audiovisuel)	France	2009	Crawl: PhagoSite, Crocket based on Firefox, Fantomas based on PhantomJS / Access: Vortex / Search: Dowser based on Elasticsearch	7	Staff of 80 documentalists taking part in nominating sites and QA

E-diaspora (Télécom ParisTech, FMSH)	France	2010	Crawl: PhagoSite	1	30 researchers taking part in nominating sites

Internet memory Foundation	France, Netherlands	2004	IM large scale crawler, Heritrix, IM Access software. Storage of Web Content: Hbase	Crawls monitoring, developers & infrastructure, manager & administration.

Internet Memory Research (ATN service)	France,	2011	IM large scale crawler, Heritrix, IM Access software. Storage of Web Content: Hbase	Crawls monitoring (QA, crawl engineering, project management), developers & infrastructure, manager & administration

Bibliotheksservice-Zentrum Baden-Württemberg	Germany	2003	Archive-It service	0.5	Websites of about 20 cities, municipalities, districts and associated corporations, and state libraries are collected by BSZ in commission within various Archive-It collections.

Web archive of the German Bundestag	Germany	2005

Hungarian Web Archiving Initiative	Hungary	2017	Heritrix, Wayback	2	4	In April 2017 the National Széchényi Library has started a web archiving pilot project as a part of its comprehensive IT infrastructure development programme.

Iceland	Iceland	2004	Heritrix, OpenWayback

National Library of Ireland Web Archive	Ireland	2011	Archive-it service	1	0.5 FTE	The National Library of Ireland selectively archives Irish websites of scholarly, cultural and political importance through its NLI Selective Web Archive.

Israel Web Archive	Israel	2011	Heritrix, Web curator tool, Wayback, Rosetta	1	>3	National Library of Israel collecting '.IL' domains, 1 Project Manager part time, 1 Technical Leader full time, 1 librarian part time, 1 IT Infrastructure part time

Web Archiving Project (WARP, The National Diet Library)	Japan	2002	Heritrix, Wayback, Solr. Previously: Wget, Accela BizSearch	9	1	Launched in 2002 as a pilot project, WARP (Web Archiving Project) has been in full-scale operation since 2006. Started the web archiving of official institution sites based on the legislation from April 2010.

National Library of Korea - OASIS (Online Archiving & Searching Internet Sources)	Korea	2001	Own system based on Oracle DBMS and specialized search engine (IRS) that performs data management and search function.	3	11

Bibliothèque nationale de Luxembourg	Luxembourg	2015	Heritrix, Wayback	2	The National library of Luxembourg conducts biannual broad crawls for the .lu domain as well as selective and event-based crawls. The websites that are harvested in the Web-Archive enrich the patrimonial collections of the National library, which allows for the preservation of digital publications for future generations.

Koninklijke Bibliotheek	Netherlands	2007	Heritrix, Web Curator Tool, Wayback, KB e-Depot system	~10	1 crawl engineer, and 9 collection specialists, all part-time (equivalent to around 2 full-time). The KB selectively collects Dutch sites of research and cultural value.

National Library of Latvia	Latvia	2005	Web Curator Tool and Wayback	1	Currently only storing for preservation, access to public in development (ETA June 2012). The Latvian term for web harvesting is "rasmošana".

New Zealand Web Archive	New Zealand	1999	Web Curator Tool, Heritrix3, Webrecorder, OpenWayback,OutbackCDX, Rosetta	4	>10	National domain harvests have been run since 2008, and annually since 2015 in collaboration with the Internet Archive. Selective harvesting is undertaken by the National Library of New Zealand using the Web Curator Tool.

The National Library of Norway	Norway

Arquivo.pt	Portugal	2007	In-house development, Heritrix, Wayback, NutchWAX, Pywb, Apache Solr	7	1	Arquivo.pt is a research infrastructure that preserves information gathered from the web since 1996 and provides a public search service over this collection.

Web archive of Cacak	Serbia	2009	HTTrack	1

Web Archive Singapore	Singapore	2006	Wayback, Heritrix, Solr	3	The Web Archive Singapore is managed by the National Library Board, Singapore (NLB). NLB conducts domain and selective archiving of websites with a focus on Singapore content.

Digital Resources (University Library in Bratislava)	Slovak Republic	2015	Heritrix 3.2.0, Wayback 2.2.0, Solr 5.2.1, Invenio, Custom Curator Tool	4	1	The University Library in Bratislava (ULIB) performed the first experiments of webharvesting in 2008-2009. In 2015 ULIB carried into operation a platform for web- and e-Born archiving (during the implementation of the national project „Digital resources“, that was supported by the European regional development fund) - https://www.webdepozit.sk/en/).

Slovenian Web Archive	Slovenia	2007	Heritrix, Wayback	1

Archivo de la Web Española	Spain	2009	NetarchiveSuite, OpenWayback, Solr	3+supervisor	2	Maintained by the National Library of Spain with the collaboration of regional libraries. Takes a mixed approach of selective and broad harvests.

PADICAT: The Web Archive of Catalonia	Spain	2005	Heritrix, Wayback, WERA, NutchWAX, Web Curator and CAT.	4	PADICAT is the open access Web Archive of Catalonia, created by the Biblioteca de Catalunya: the public institution responsible for collecting, preserving and distributing the bibliographic heritage of Catalonia, in Spain.

ONDARENET - Basque Digital Heritage Archive	Spain	2008	Heritrix, Wayback, NutchWAX and Web Curator.	1

Sweden (Kulturarw3)	Sweden	1996	Heritrix. Own system for storage, maintenance and access	1.25	Paus in operation November 2009 - May 2011.

Aleph Archives	Switzerland, United States	2010	Distributed crawler, ArchiView access plugin, High performance search engine, Near real time indexing, Web Monitoring tools	7	Enterprise-grade Web archiving platform for online heritage (content, brands) preservation and eDiscovery aimed to corporates, institutions, legal and government industries seeking to preserve their Web content regardless of their types (websites, wikis, social media, forums...).

Web Archiving Bucket	Switzerland, United States, Canada	2012	WARC Software Development Kit, Cobalt, Holon web server	The “Web Archiving Bucket” is an initiative launched by Aleph Archives, to preserve data and provide libraries and organizations with free-to-use web archiving tools and components.

Web Archive Switzerland	Switzerland	2008	Heritrix, Wayback, Webrecorder	5	1 crawl engineer, 3 persons for quality assurance (sharing less than 1 full time), 1 coordinator. The curators, who do the selection, are partner libraries all over Switzerland.

NTU Web Archiving System, NTUWAS	Taiwan	2007	Lucene	3

Web Archive Taiwan	Taiwan	2007

The UK Web Archive	United Kingdom	2004	Heritrix, Web Curator Tool, Wayback, Solr for searching.

UK Government Web Archive (UKGWA)	United Kingdom	2003	MirrorWeb	4	0	The UK National Archives' UK Government Web Archive (UKGWA) is a fully open web archive. It includes approx.

UK Parliament Web Archive	United Kingdom	2009	MirrorWeb	1	2	The UK Parliament Web Archive captures, preserves, and make accessible UK Parliament information published on the web. The web archive includes websites and social media dating from 2009 to the present.

MirrorWeb	Worldwide	2012	Heritrix, PYWB, custom social media archiving tools.	8	MirrorWeb provide website and social media archiving service for regulated industries and public sector. They run two public archives; the UK Government Web Archive and the UK Parliament Web Archive.

Internet Archive (provides Archive-it service)	United States	1996	Heritrix, Wayback, NutchWAX and other tools developed by the Internet Archive	150	Internet Archive's Wayback Machine is the largest and oldest web archive in the world, dating back to 1996. Internet Archive also provide various web archiving services, including Archive-IT, Save Page Now, and domain level contract crawls.

Reed Tech Archives	United States	2010	TrueArchive™ Technology	Reed Tech Archives provides support for Information Governance, Litigation Protection, Compliance, e-Discovery and Social Media Management. Solution offers both an automated approach or manual capture.

Stanford University Libraries	United States	2007	Heritrix, HTTrack, Wayback, CDL Web Archiving Service, Internet Archive Archive-It	2	5	Stanford University Libraries has been engaged in web archiving projects since 2007 and started establishing a web archiving program in 2013.

Columbia University Libraries	United States	2009	Archive-it service	2	>1	The Columbia University Libraries (CUL) web resources collection program archives selected websites in thematic areas corresponding to existing CUL collection strengths, websites produced by affiliates of Columbia University, and websites from organizations or individuals whose papers or records are held in CUL's physical archives.

Cornell University Library	United States	2011	Archive-it service	.5	>1

North Carolina State Government Web Site Archives	United States	2005	Archive-it service	3

Latin American Web Archiving Project	United States	2005	Archive-it service

Web Archiving Project for the Pacific Islands	United States	Archive-it service	4

Library of Congress Web Archives	United States	2000	Heritrix, Wayback, and the DigiBoard, an in-house curatorial/permissions tool	6	80	The part time workers spend a few hours per month (on average) selecting content for the collections.

Harvard Library: the Web Archive Collection Service (WAX)	United States	2006	Heritrix, Wayback, NutchWAX and WAXi, an in-house curatorial interface.	>6	3 part time on IT support. External curators within 3 units but do not know their size.

Web Archiving Service from California Digital Library (WAS service)	United States	2005	Heritix, Wayback, NutchWAX	4	>1	The number of hours that curators devote to the service is very variable.

Bentley Historical Library (University of Michigan) Web Archives	United States	2000	HTTrack, Teleport Pro, WAS service (2010-)	2

University of Texas at San Antonio Web Archives	United States	2009	Archive-It	3	The number of hours varies dependent upon how the crawls are scheduled.

qumram	Switzerland	2010	qumram Web Archiving / Web Information Governance Software Suite	Commercial web archiving / web information governance software suite. Provides both remote harvesting as well as transactional web archiving.

SAPERION	Germany	2011	SAPERION ECM Web Content Archive	Commercial enterprise content management suite specializes on regulatory compliance. The product provides both harvesting as well as transactional web archiving based on the integration of qumram's Chronos Web Archiving Software Suite.

Bibliotheca Alexandrina's Internet Archive	Egypt	2002	Heritrix, OpenWayback, WARCrefs	3	Current crawling interests: Egypt beyond January 25, Arab League ccTLDs Deduplication: using WARCrefs tool to deduplicate web archive contents in BA cluster OpenWayback: handling big data indexing by using ZipNumCluster to locate a certain URI in compressed CDX files

AUEB Web Archive	Greece	2010	Heritrix, Wayback and NutchWAX.	1	1	This project is part of the function of the University Library.

World Bank Web Archives	United States	2007	HTTrack crawler, Oracle RDBMS, Google Search Appliance	0	3

Russian National Digital Archive	Russia	2010	wpull, grab-site, HTTrack crawler, ad-hoc scripts developed for social media archiving. Experimenting: Heritrix, Wayback	About 5000 government websites collected (May 2018) using wpull and provided as archives for downloading.

Archive Team	Worldwide	2009	wpull, ad hoc scripts	1	~100	Volunteer group. They partially archived GeoCities, Yahoo! Videos, Google Video and others.

WikiTeam	Worldwide	2011	ad hoc scripts	0	0	Volunteers group. Over 20,000 wikis preserved.

University of North Texas CyberCemetery	United States	1995	Heritrix, Wayback; formerly HTTrack	2	The CyberCemetery is an archive of government websites that have ceased operation (usually websites of defunct government agencies and commissions that have issued a final report).

Archive.is	Worldwide	2012	Apache Accumulo, HDFS, ad hoc scripts	1	1	Saves external links from community web-sites (wikis, forums, blogs, ...). Can save snapshots of Web 2.0 pages.

Tamiment Library and Robert F. Wagner Labor Archives at New York University	United States	2007	WAS Service	1	1	Archives websites related to New York City and National Labor and Left Movements. Projects include: Alternative Mass Media / News; Anarchism; Animal Rights; Arts and Cultural Left; Civil Rights and Civil Liberties; Communism, Socialism, Trotskyism; Economic and Social Justice (Including Occupy Wall Street); Education and Student Movements; Electoral Politics and Parties / Political Action (U.S.

Preservica	Worldwide	2012	Heritrix, Preservica core product, Wayback	Cloud-based heterogeneous archiving service that allows ingest from multiple sources (including web archiving ingest via Heritrix). Ability to migrate content within WARC files and render in Wayback.

Central State Electronic Archives of Ukraine	Ukraine	2007	HTTrack, Wget	2	Archives interested in keeping websites and creating the thematic collections of such websites, Is presently in storage the Archives collections of websites which includes the topic of presidential elections in Ukraine from 2010 until today, about the Chornobyl disaster, the local elections, of Euro 2012 in Ukraine, UNESCO World Heritage sites in Ukraine, the 200th anniversary of the birth of Taras Shevchenko.

York University Libraries, York University Digital. Library	Canada	2012	Heritrix, Wget, Islandora, OpenWayback	1	0

New York Art Resources Consortium (NYARC)	United States	2012	Archive-It service	1	~3	Collaboration among Frick Art Reference Library, Brooklyn Museum Library & Archives, and Museum of Modern Art (MoMA) Library to archive specialist art historical web resources.

Netherlands Institute for Sound and Vision (Sound and Vision) web archive	Netherlands	2011	Heritrix, Elasticsearch for full-text index, Drupal for front-end	~7	Sound and Vision has been involved in web archiving projects since 2008, starting with the EU research project LiWA. After a couple of pilots, web archiving projects were scaled up in 2014.

Rhizome (organization)	United States	1999	ArtBase, Webrecorder, Oldweb.Today	3	1	Rhizome operates a digital preservation program, led by Dragan Espenschied, which is focused on the creation of free, open source software tools to decentralize web archiving and software preservation practices and ensure access to its collections of born-digital art.

University of Texas at Austin Libraries, Human Rights Documentation Initiative	United States	2009	Archive-It service	1	1	The University of Texas Libraries’ Human Rights Documentation Initiative (HRDI) captures the websites of human rights organizations in order to provide secure access to human rights documentation in the event that these often-fragile sites are taken down.

Kentucky Department for Libraries and Archives	United States	2009	Archive-it, Wayback	>1	0	This collection includes captures of websites for Kentucky state agencies in the Executive, Legislative, and Judicial Branches. Stand-alone websites for boards, councils, committees, quasi-governmental agencies, and agency programs are also archived.

University of California, San Francisco Library	United States	2007	Archive-it, Wayback, CDL WAS Service	>1	0	This collection documents the web presences of UCSF, as well as the larger health science focuses of AIDS history; anesthesiology; biotechnology and biomedical research; tobacco control and regulation; neuroscience; and computational medicine.

Ivy Plus Libraries	United States	2013	Archive-It	1	1	The Ivy Plus Libraries Web Collecting Program is a collaborative collection development effort to build curated, thematic collections of freely available, but at-risk, web content in order to support research at participating Libraries and beyond.

Archive.St	United States	2017	Archive.st custom programming provided by US Support LLC	>1	0	Archive.st provides free online web archiving in the form of a .JPG and HTML archive.

Sentence(s)

The UKGWA is one of the web archiving initiatives fully open.

final_sentence: The UKGWA is one of the web archiving initiatives fully open.