Distributed Digital Preservation in the Cloud

The LOCKSS system is a leading technology in the field of Distributed Digital Preservation. Libraries run LOCKSS boxes to collect and preserve content published on the Web in PC servers with local disk storage. They form nodes in a network that continually audits their content and repairs any damage. Libraries wondered whether they could use cloud storage for their LOCKSS boxes instead of local disks. We review the possible configurations, evaluate their technical feasibility, assess their economic feasibility, report on an experiment in which we ran a production LOCKSS box in Amazon's cloud, and describe some simulations of future costs of cloud and local storage. We conclude that current cloud storage services are not cost-competitive with local hardware for long-term storage, including for LOCKSS boxes. © David S. H. Rosenthal 2012 2 Distributed Digital Preservation in the Cloud Introduction The LOCKSS Program at the Stanford University Libraries pioneered the concepts of Distributed Digital Preservation about 14 years ago. by building a peer-to-peer ntwork of LOCKSS boxes in which libraries could collect and preserve content published on the Web. About 150 libraries currently run LOCKSS boxes preserving e-journals and e-books in the Global LOCKSS Network (GLN), and databases, government documents, special collections and other content in Private LOCKSS Networks (PLNs). These boxes are typically modest PC servers with substantial amounts of local disk. Some of these libraries asked whether they could use ``affordable cloud storage'' for their LOCKSS boxes. We first describe the relevant features of typical cloud services, then discuss the various possible technical architectures by which a LOCKSS box could use such a service, ruling some out for performance reasons. Applying the typical service's charging model to the remaining architectures rules others out for economic reasons. We then describe an experiment in which we implemented the most cost-effective architecture and ran a production LOCKSS box preserving GLN content in Amazon's cloud service for several months. We report on the pricing history of cloud storage, using it and the costs from the experiment to compare the projected future costs of storing content preserved by LOCKSS boxes in cloud and local storage. We conclude with discussion of future work, some of which is already under way. Features of Cloud Technology Amazon, as a typical cloud service, provides the following components relevant here: • A compute service in which a customer can run a number of virtual machine instances. These instances have the normal resources of a physical PC but their (virtual) disk storage is limited in size and is evanescent, vanishing without trace when the instance stops for any reason. Amazon calls this service Elastic Cloud Computing (EC2). • An object storage service in which a customer can store arbitrary-sized byte strings. These objects persist, are named by URLs, and are accessed via an API using HTTP PUT, GET and HEAD requests. Depending on the service, the object store may be more or less reliable. Amazon calls this service Simple Storage Service (S3), and offers two levels of reliability. The standard level is designed for 11 nines durability, and Reduced Reliability Storage (S3-RRS) is designed for 4 nines durability. Lots Of Copies Keep Stuff Safe, a trademark of Stanford University. This is true for instances backed by S3, not by EBS. But EBS is not reliable enough; see Experimental Results below. 8th International Digital Curation Conference January 2013 David S. H. Rosenthal 3 • A block storage service, in which a customer can store a file system that can be mounted by virtual machine instances running in the compute service. These file systems persist across virtual machine restarts but are designed for performance rather than extreme reliability. Facilities are normally provided by which snapshots of the file systems can be reliably preserved as objects in the object storage service. Amazon calls this service Elastic Block Storage (EBS).