Resource Reclamation in Distributed Hash Tables

Distributed hash tables (DHTs) are increasingly being proposed as the core communication and storage layer for distributed services in the Internet. This trend spans many traditional distributed applications, including file, storage, and archival systems [2,11,13,15], content delivery systems [3,6,8], databases [9], mail servers [7], messaging systems [10], and distributed naming services [12, 16]. DHTs map keys in a large, virtual ID space to associated values stored and managed by individual nodes in an overlay network. Since DHTs span many unreliable nodes in a wide-area environment, network and node failures are a common occurrence. To cope with such failures, previous work has explored mechanisms and policies for using redundancy to mask failures as users and applications allocate and update data in the DHT. In this paper, we discuss the problem of storage reclamation in DHTs that use redundancy to provide high data availability. Two general approaches to managing data redundancy in DHTs have emerged, eager repair and lazy repair. For data stored in the DHT under a particular key, the eager approach maintains redundant data on n successors of the node that “owns” the key [4, 5, 15]. Each node in the system runs a background stabilization process that synchronizes its data with its n successors. This stabilization process conveniently handles both redundancy maintenance and reclamation simultaneously. Nodes propagate redundant data to a successor if the successor does not have it. Similarly, nodes implicitly propagate deletions to their successors so that successors can determine when they can remove the redundant data they store. As a result, the time it takes for deletions to propagate from the owner to all successors is typically n rounds of the stabilization process. Since rounds are short, eager repair will quickly delete all redundant versions of data stored at the owner node and its successors. Note, though, that eager repair implicitly assumes that successors will have the capacity to store redundant data. Compared with eager repair, lazy repair uses additional redundancy to mask temporary node departures [2]. Eager repair immediately reacts to node arrivals and departures by propagating redundant data to new successors. Since node churn makes departures and arrivals very common, lazy repair uses additional redundancy to delay the propagation of redundant data to new nodes until many nodes storing the redundant data have simultaneously departed. As a result, lazy repair reduces the communication overhead of maintaining redundant data for availability in the face of high node churn by decoupling redundancy maintenance from node departures. Lazy repair uses a level of indirection to store redundant data. When storing data at a key’s owning node, lazy repair creates and stores redundant versions of the data on random nodes in the system. It records and tracks the nodes storing redundant data in a metadata structure stored at the key’s owner. To ensure that the metadata is also highly available, lazy repair uses eager repair just for the metadata. To delete data in a system using lazy repair, an application will send a request to delete data associated with a key to the key’s owning node. The owner will then mark the data as deleted and subsequently delete all redundant versions of this data on storage nodes. Once storage nodes delete the redundant