# Towards a Highly Available Model for Processing Service Requests Based on Distributed Hash Tables
## 2 Materials and Methods
### System Scalability Based on the Intrinsic Properties of the Chord DHT
From Figure 5, too, note that the maximum routing time needed to send any message via the Chord DHT distributed routing infrastructure was no larger than 8 ms, which represents a reasonable overhead for the benefit of increased scalability. The obtained results are according to the authors' expectations, since, when measuring the maximum inter-node communication delay within our cluster, it was of no more than 2 ms per node pair. The almost stair shape of the plotted curve in this work's experimental results is in tight correlation with the shape of the integer part of the logarithm of the number of DHT deployed nodes, i.e., \([log_{2}(No_{pers})]\). ### Practical Evaluating of the System's Fault Tolerance

This section evaluates the fault tolerance of the system by means of experimental validation. The following experimental steps were performed: a number of peers ranging from 0 to 128 was deployed, clustered in three different groups of nodes of our datacenter. For each of the DHT dimensions ranging from 1 to 128, we considered both: (1) the case when a new peer _joined_ the network, and have evaluated the time until that peer or the data keys it must handle could be reached, by routing via the DHT; (2) the case when a peer _left_ the network, and evaluated the time until its backup peer took over for the IDs that belonged to the failing peer, as was also described in the replication mechanism of Chord [9]. Note that, as already explained in Section 2, the keys that a peer \(p\) is responsible for are also Chord IDs. Moreover, each peer is responsible for the Chord IDs that are

\[Finger_{k}\approx ID_{p}+2^{k},\]3 The extra time needed by the system to reply to the client, via the DHT. situated between its predecessor and himself, i.e., the keys that a peer \(p\) is responsible for are: \(ChordID_{predecessor(p)}<ChordID_{key(p)}\leq ChordID_{p}\). Note, from Figure 6, that the time intervals needed for the DHT infrastructure to stabilize, i.e., to be entirely aware of the topology changes, are clustered around three centers. Remember that the DHT had been deployed in three areas of our datacenter, on heterogeneous types of nodes. It has already been mentioned in this section, that the inter-node communication delay within a single area is very low, i.e., 0.30 ms, while the communication delay between each pair of different areas of the cluster is of between 1-2 ms. If three areas in the datacenter are considered, besides the pairs of nodes that belong to the same area and thus communicate extremely fast, we can form only three other types of pairs \((peer_{source},peer_{destination})\), i.e., \(\mathrm{C}_{2}^{3}\) types of pairs, where the area for each source peer is different from the area of each destination peer. These three areas in the plot, that can be seen in Figure 6, correspond to the total stabilization time needed by peers of one datacenter area to learn about the infrastructure changes in each other datacenter area. The infrastructure changes involve, with equal probability, both new peers joining the DHT and failing peers, leaving the DHT. The explanation for the slight spikes in the obtained plots is the fact that not all peers have been deployed within the same rack, and the communication delay within a rack is of about 300 microseconds, while the inter-rack communication delays experienced in our experiments is between 1-2 ms.

#### 3.2.1 The Improvement in Fault Tolerance

The proposed new model based on DHT enables the entire system, running on a collection of peers, to become aware rather fast of the changes in the peer topology, i.e., either peer join or peer fail/leave, which is extremely difficult without a DHT. From Figure 6, in the worst case scenario, the model needs no more than 2 s for _every_ peer to update their DHT routing tables, thus becoming aware of the new topology.