# Towards a Highly Available Model for Processing Service Requests Based on Distributed Hash Tables
## 2 Materials and Methods  
This section describes the improved novel solution for a highly available distributed service for processing its clients' requests, which maintains the same interface to its clients, while the changes to make it highly available are unnoticeable to the clients. It focuses on introducing as little new communication as possible, in order to maintain the Single System Image (SSI), despite the novel decentralization method, based on _DHTs_ and _highly available data_. In terms of generic components that should be used to decentralize any distributed service, we enumerate: (1) a _load-balancing dispatcher_, which transparently distributes the load among nodes; (2) a DHT peer-to-peer network based on Chord [9]; (3) a reliable decentralized distributed database, also based on DHTs.  
The physical testbed on which the solution was deployed  
The improved solution for high availability was deployed and tested in our team's datacenter, designed for high performance computing on large-scale distributed systems. It contains more than 3500 CPU cores and 12 TB of dedicated RAM, together with 30,000 GPU cores and 220 TB of distributed storage, interconnected by GluInfiniband. The CPU core ensemble comprises hybrid architectures such as Intel Xeon, AMD Opteron, Power7 and CellBE, interconnected by means of very-high-speed networks, such as 10 Gigabit and Gigabit Ethernet. The various node architectures and the high-speed interconnections among them, together with the extended storage, make this infrastructure a good candidate for validating new findings regarding highly available decentralized distributed processing of clients' requests, with minimum communication delays. All of these are features envisaged by the improved software architecture for high availability, presented in the following subsections.  
The novel contributions of this paper and the design of the experimental system are mainly driven by the features of the hash function within the Chord DHT. Note that in Chord all peers are equal, and the destination of a message is identified by collaboration among the peer nodes, with high probability in no more than \(O(log_{2}(No_{pores}))\) steps, according to the dimensions and organization of the per node routing finger tables [9].  
The hash function of the DHT is essential in this work's vision for an improved model of obtaining a _highly available service_, since it guarantees, with high probability, both: (1) the uniqueness of the hash ID, which has been obtained by computing a hash of the data; (2) the large distance, i.e., difference, within the space of the hash IDs of the hash values that correspond to very similar pieces of data. This second property means that related pieces of data are stored or handled on different nodes, a property which is good at obtaining a _balanced distributed system_. In other words, pieces of similar data are stored, with a high probability, on totally independent nodes, a thing that is very useful when storing similar and thus correlated data, which often can be derived from one another, thus yielding _fault tolerance_, too. Note that this exact property is also important when replicating within a world-wide distributed Chord DHT: it is very likely for a peer that is responsible for some data, and its clock-wise successor peer on the ring, which backs that same piece of data, to find themselves very distant, geographically, thus _physically independent_ from one another. This allows the distributed ensemble of nodes to behave as a _fault tolerant_ system, thus offering a _persistent_ data storage. This last property is essential to a reliable storage system based on DHTs, such as a _decentralized highly available database_[7; 16; 17], which can reliably store the data needed by any request processing service. Moreover, for a more \"dispersed\" storage of the data in the DHT network, one can successively apply more hash functions [37].