Reliable and Secure Distributed Storage using Erasure Codes

Large-scale enterprise data is increasingly organized in the form of distributed storage. The three important issues that arise when moving to distributed storage are reliability, security and concurrency. We introduce a collection of methods drawn from the theory of erasure-correcting codes to design reliable and secure distributed storage that can perform correctly and efficiently even when concurrently accessed by multiple uncoordinated clients. Our methods are general and are capable of wide usage in a variety of distributed storage applications. I. DISTRIBUTED STORAGE STORAGE represents the lifeblood of the modern enterprise. The information economy is driven by fast computing and plentiful storage, and the concomitant explosion in day-to-day data processing motivates enterprises to organize and consolidate their storage to facilitate easy access and costeffective management. For scalability, ease of management, cost-effectiveness, and higher performance, enterprise storage is increasingly organized as distributed storage, usually in the form of Storage Area Networks (SANs) or Network-Attached Storage (NAS). SANs are storage solutions built around block-addressed storage units located on a dedicated high-speed network, usually 100 Megabytes/second Fibre Channel. NAS, on the other hand, exports a file-based interface to its clients, and is implemented by attaching specialized file servers, called “filers”, to a TCP/IP network, usually Ethernet. The introduction of iSCSI [1], a new standard for SANs, has precipitated a state of upheaval in the storage networking market: iSCSI is a network storage solution that exports a block interface similar to traditional SANs but is implemented by layering the SCSI protocol over the TCP/IP protocol with the networking technology of choice being Gigabit Ethernet. Proponents of iSCSI claim that iSCSI’s ease of management combined with the low-cost of Ethernet technology and the interoperability of the IP protocol, offer compelling advantages over Fibre Channel-based SANs; its detractors point to its lower performance due to TCP/IP protocol overheads and its lack of readiness to meet current business needs, and predict that currently popular Fibre Channel-based technology will prevail in the foreseeable future. From the interest in iSCSI technology displayed by leading industry vendors [2,3] and standards bodies, both in itself and in This work was supported in part by NSF Grant CCR-TC-0208975 combination with Fibre channel it appears likely that heterogeneous block-addressed storage networks accessed through multiple protocols and interconnects will play a prominent role in high-performance distributed storage applications. At the other end of the spectrum, much activity is happening at the wide-area level too, albeit in research rather than industry: emerging work on peer-to-peer file sharing and overlay networks [4, 5] has devised new ways to build large scale co-operative distributed file systems over heterogeneous widearea networks. While this work has enabled new applications like global-scale group collaboration, file sharing and storage, it poses new questions about creating a secure and reliable data storage and access facility over insecure and unreliable providers. A. Issues in Distributed Storage Design An important issue with moving from server-attached storage to distributed storage built from many storage units, is faulttolerance. The earliest work [6] that proposed distributing storage over multiple disks recognized the increased failure rate of such systems built out of many units that can fail independently, and from then on, fault-tolerant storage systems have been studied in great detail and scope [7–13]. The usual approach to building a reliable system out of unreliable components is redundancy: in a fault-tolerant storage system, data is stored redundantly over multiple disks; should one fail, all data on it is recovered from the others. The traditional approaches towards redundant storage are mirroring, where every block is replicated in a mirror block on another disk, and parity schemes, where a single parity block is designated for a group of data blocks. Erasure code-based schemes are a generalization of these earlier schemes, in which blocks are protected by an additional parity blocks using an maximum distance separable (MDS) erasure correcting code [14]. These codes have the property that any of the encoded blocks suffice to recover the original data blocks. A storage system built using these codes can recover from simultaneous disk failures and hence, for a given storage overhead, erasure code-based schemes provide optimal protection against data loss from one or more disk failures. Since mirroring and parity may be considered trivial and erasure correcting schemes, the ideas in this paper applies to these schemes too. In erasure code-based schemes, parity blocks must be calculated as a function of data blocks by a procedure called encod-