A Distributed Fault-Detection and Recovery Protocol for Reliable Multicast Collaborative Communications Walid Mostafa Mukesh Singhal Department of Computer and Information Science The Ohio State University Columbus , OH 43210 email : -LRB- mostafa , singhal -RRB- @cis . ohio-state . edu Phone : 614-292-5839 Fax : 614-292-2991 Abstract Reliable multicast transport protocols support dissemination communication and their throughputs are either limited by the sender or by intermediate nodes that consolidate acknowledgements and retransmissions . RMSP is a portable reliable multicast protocol that supports collaborative communication at the session-layer level and provides higher throughput by maintaining two transport connections per node irrespective of the number of nodes . However , the throughput only of RMSP may degrade if one or more nodes or the centralized connection manager that monitors them fails . In this report , we propose a fully distributed fault-tolerant protocol , DFT-RMSP , that detects and isolates faulty nodes and reinstates the multicast collaborative communications . We derive analytic expressions for throughput degradation and analyze its sensitivity to node-failure rates , network delay , and multicast group size . We show that DFT-RMSP provides less downtime and achieves graceful throughput degradation under high failure rates compared with RMSP . simultaneous node failures . 1 Introduction The continuous growth in applications that require communications among a group of hosts or simultaneous dissemination of data to multiple sites on the Internet has led to considerable interests in multicast communication -LRB- e.g. , video conferencing -LSB- Jac94 -RSB- and distributed whiteboards -LSB- McC92 -RSB- -RRB- . Multicast protocols provide a spectrum of services to cater for the needs of a wide array of multicast applications . Such protocols support multicast communications in two modes : dissemination and collaboration . Dissemination is a 1xR communication scheme where a single sender multicasts data to a group of R receivers -LSB- Che92 -RSB- . Collaboration is a RxR communication scheme where each node is actively involved in multicasting data to and receiving data from all other -LRB- R ? 1 -RRB- nodes -LSB- MS97a -RSB- . Fault-tolerance of multicast protocols addresses both reliable delivery of messages and maintaining multicast communication despite node failures . Reliable message delivery is accomplished using acknowledgements and message retransmissions and is classified into sender-initiated , receiverinitiated , and hierarchical approaches . Sender-initiated protocols -LSB- CM84 -RSB- require a sender to maintain state information for each receiver using positive acknowledgements , ACKs . Receiver-initiated protocols -LSB- HSC95 -RSB- rely on receivers to detect and request retransmission of missing packets using negative acknowledgements -LRB- NAKs -RRB- . Hierarchical protocols , e.g. , -LSB- YGS95 -RSB- and -LSB- FJM +95 -RSB- -RRB- , arrange nodes into a tree-like configuration to consolidate acknowledgements and localize retransmissions at a subtree level . Node failure detection and isolation is essential to maintain multicast communications , specially for nodes that assume a particular role besides receiving multicast traffic . For example , the failure of the master node in MTP -LSB- BZ93 -RSB- , a domain manager node in TMTP -LSB- YGS95 -RSB- , or a designated receiver in -LSB- LP96 -RSB- will terminate the multicast connection . Therefore , some protocols provide mechanisms to detect , isolate , and replace faulty nodes . A common mechanism for detecting node failure is a heartbeat . A heartbeat is an interval of time that is relevant to the function of a transport protocol . A node can either periodically send a heartbeat message to indicate its status or it can be polled to send a heartbeat message by another node in a master-slave mode . The latter approach uses more bandwidth for full-duplex heartbeats but has the flexibility to employ variable and/or selective heartbeat periods . The length of a heartbeat interval is impacted by the longest round-trip delay of communicating nodes . Longer heartbeat intervals delay the availability of information while shorter heartbeat intervals incur higher bandwidth utilization and require more processing . Braudes and Zabele introduced in -LSB- BZ93 -RSB- a Multicast Group Authority , MGA , that arranges nodes into a tree and requires a parent to poll its children for heartbeats to check their connectivity . The owner of the transmit token in the MTP protocol -LSB- AFM92 -RSB- must transmit at least one message in every heartbeat or the master node assumes that it failed and generates a new transmission token . A source node in TRM -LSB- SBD96 -RSB- send heartbeats periodically to the multicast group to declare its state when there is no data to transmit . A connection manager in RMSP -LSB- MS97a -RSB- polls all nodes in a multicast group every heartbeat period and rearranges the connection around faulty nodes . Fault-detection using a centralized manager , e.g. , RMSP -LSB- MS97a -RSB- , does not scale well as the number of nodes in a multicast group increase . This is because the manager has to send more polling messages and process more responses during a heartbeat period . Consequently , the heartbeat period must be long enough to accommodate message exchange with all nodes . Longer heartbeat periods imply delayed detection and isolation of faulty nodes and slower recovery of a multicast connection . Tree-based fault-detection protocols -LSB- BZ93 -RSB- achieve better scalability but incur nonuniform overhead per node , e.g. , non-leaf nodes are responsible for their children while leaf nodes have no responsibility . RMSP is a reliable scalable multicast protocol that uses a session-level layer to provide multicast management and uses existing reliable transport protocols , e.g. , TCP , to provide reliable message delivery . RMSP achieves efficiency by multiplexing session-level messages into transport-level packets . However , RMSP uses a connection manager and a heartbeat mechanism to detect and isolate faulty nodes and recover multicast communications . Thus , the fault-tolerance mechanism of RMSP is subject to poor scalability and longer heartbeat intervals as discussed earlier . In this report , we present DFT-RMSP ; a distributed fault-detection , isolation , and recovery protocol variation to the RMSP protocol -LSB- MS97a -RSB- . Our objective is improve the scalability of fault-detection and isolation and increase the overall throughput of multicast communication . We accomplish this by distributing fault-tolerance responsibilities among all nodes . In DFT-RMSP , a node is responsible for detecting and isolating the failure of only one node irrespective of the multicast group size . Section 2 presents an overview of the RMSP protocol . Section 3 describes the distributed fault detection and isolation layer of DFT-RMSP . In Section 4 , we develop analytic models of both centralized and distributed fault-detection and isolation and derive throughput-degradation expressions for each approach . Section 5 provides a performance comparison between the centralized and distributed fault-tolerance approaches . Summary and concluding remarks are presented in Section 6 . 2 RMSP Overview RMSP -LSB- MS97a -RSB- is a reliable multicast session protocol for collaborative continuous-feed applications . It supports NxN group multicast communications at the session layer and runs on the top of existing reliable transport protocols -LRB- e.g. , TCP -RRB- . It does not assume multicast support at the network layer . RMSP is portable as a network-layer-independent library that is suitable for both LAN and WAN environments . It multiplexes messages from different nodes into transport packets to reduce the transport-layer processing requirements per message . It requires only two transport-level connections per node , irrespective of the multicast group size , to reduce the processing requirements of acknowledgements and retransmissions per node . Analytical studies -LSB- MS97b -RSB- show that the protocol yields higher throughput than general sender-initiated and receiver-initiated reliable multicast transport protocols -LSB- PSK94 -RSB- . RMSP maintains all multicast communication information at the session level . It has a data structure for every session in the multicast group to maintain state information -LRB- e.g. , last message received , last message sent , last message acknowledged , etc. -RRB- . The transport layer has no knowledge of the multicast group . RMSP assumes a connection manager that arranges the multicast group local node session C D BA TCP Application IP RMSP Network B C DA TCP Application IP RMSP Network A D B CA TCP Application IP RMSP Network D B A C D TCP Application IP RMSP Network B C remote node session Figure 1 : RMSP Multiplexed Session Layer nodes into a virtual ring where each node establishes a transport-level connection with a predecessor node and a successor node . We use the notation x ! y to represent a a directed data path from node x to node y. Figure 1 shows a RMSP virtual ring with four nodes ; A , B , C , and D . The transport paths A ! B ; B ! C ; C ! D ; and D ! A constitute the NxN connection or virtual ring . A multicast session message flows through the virtual ring . The sender treats the message as a positive acknowledgement when it returns back . The connection manager maintains the topology of the multicast connection and communicates any update to all nodes . RMSP multiplexes several session messages in a transport packet to reduce the transport layer overhead per message . Each session message has a header identifying which session it belongs to , the message 's relative sequence number within that session , and the length of that message . The RMSP session layer on a node demultiplexes incoming transport-level packets into session messages , processes them , and then multiplexes them into transport-level packets . The transport layer is responsible for delivering transport-level packets reliably and in-sequence between any two nodes on the virtual ring . Fault-tolerance of multicast communication is maintained by the connection manager . In addition to the two data transport ports , each node has one maintenance port for communication with the connection manager as illustrated in Figure 2 . Upon connection establishment , the connection manager determines a heartbeat interval that is long enough to accommodate all nodes in the multicast group . During a heartbeat period , the connection manager sends a heartbeat polling message to each node and expects a heartbeat response message back . Thus , the heartbeat period must be long enough to account for the longest round-trip delay of a heartbeat message exchange . maintenance port CM C D A B C B A D RMSP DFT-RMSP CM connection oemanager data port data path heartbeat message heartbeat response Figure 2 : Centralized vs. Distributed Fault-Tolerant RMSP A connection manager detects a faulty node if it does not receive a heartbeat response from this node by the next heartbeat interval . The connection manager isolates the faulty node by instructing its predecessor and successor nodes on the virtual rings to drop their connections with the faulty node . Multicast communication is recovered when the predecessor and successor nodes establish a transport connection among themselves and acknowledge the connection manager . If a failed node a recovers , it sends a reconnection request to the connection manager . The connection manager determines where does this node fit on the virtual ring . In other words , it identifies two adjacent nodes , x and y , that should serve as the predecessor and successor of node a , respectivey . The connection manager sends this info to the three nodes , x , a , and y. Nodes x and y drop their data and maintenance connections , establish new ones with node a , and send acknowledgements to the connection manager . The connection manager communicates the toplogy change to all multicast group through the maintenance connections . 3 Distributed Fault-Tolerant RMSP RMSP employs a centralized connection manager to monitor all nodes in a collaborative multicast connection as discussed in Section 2 . The connection manager maintains a unicast maintenance transport connection with every node to poll and receive heartbeat messages . Thus , as the group size increases , more maintenance connections are needed and higher processing requirements are incurred by the connection manager . This results in longer heartbeat intervals leading to larger delays to failure detection , longer connection downtime , and degraded overall throughput . In this section , we present DFT-RMSP ; a distributed approach to node-failure detection , isolation , and recovery . Unlike RMSP , DFT-RMSP does not use the connection manager to monitor the nodes of a multicast group . Instead , it distributes this responsibility among all nodes . The approach is symmetric , uniform , and scalable ; each node monitors only one node and is monitored by only one node , irrespective of the number of nodes in the multicast group . DFT-RMSP uses a distributed heartbeat mechanism to detect failed nodes independently on the virtual ring . Each node establishes two maintenance connections ; one with its predecessor and one with its successor on the virtual ring -LRB- that is with respect to multicast traffic flow -RRB- . A pair of adjacent nodes exchange heartbeat messages over the maintenance connection between them in a master-slave mode . If a slave does not respond within a timeut period , the master drops the transport connections with it and recursively attempts to establish both data and maintenance connections with those nodes on the successor side of the failed node . For a given pair of nodes , DFT-RMSP defines the master and slave nodes of a maintenance connection as a node and its successor with respect to the multicast data path . Thus , node x is the master of the maintenance connection between nodes x and y if x ! y is a segment of the multicast data path . We use the notation -LRB- x -RRB- y -RRB- to represent a maintenance connection where x is the master node and y is the slave node . Figure 2 illustrates an example of a DFT-RMSP virtual ring with four nodes : A ; B ; C ; and D. Therefore , the maintenance connections of this ring are : -LRB- A -RRB- B -RRB- ; -LRB- B -RRB- C -RRB- ; -LRB- C -RRB- D -RRB- ; and -LRB- D -RRB- A -RRB- . For a given maintenance connection -LRB- e.g. , -LRB- A -RRB- B -RRB- -RRB- , the directed dashed arrow represent a heartbeat polling message from the master node to the slave node and the directed dotted arrow represent a heartbeat response from the slave node to the master node . During a heartbeat interval , the master node of a maintenance connection polls its slave node for a heartbeat message . If the slave node does not respond by the next heartbeat interval , the master assumes the slave has failed and drops both the data and the maintenance connections with the failed slave node . The master then attempts to connect to the successor of its failed slave . For example , if slave node B , of the maintenance connection -LRB- A -RRB- B -RRB- , does not respond to a heartbeat polling message from its master node A by the following heartbeat interval , node A assumes that node B failed and attempts to establish a connection with node C , the successor of B. t failed nodegood node 1 1 1 1 1 1 2 2 2 3 3 4 a o p i j h g f e d c b k l m n q r s Figure 3 : Distributed Loss-Detection/Isolation Figure 3 presents a snapshot of a virtual ring for a given heartbeat interval where few nodes have failed but are neither detected nor isolated . For example , nodes a and i are the masters of the -LRB- a -RRB- q -RRB- and -LRB- i -RRB- h -RRB- maintenance connections , respectively . During the given heartbeat interval , node a polls its slave , node q and independently , node i polls its slave , node h , for heartbeat . Since nodes q and i are down , they do not respond to their masters . By the next heartbeat interval , both a and i realize that their successor nodes are down and attempt to connect with nodes r and g , respectively . Node i succeeds in establishing transport connections with node g and multicasts the topology change to the group . Node a attempts to connect with one node every heartbeat interval passing by the failed nodes q ; r ; s until it finally connects with node t by the fourth heartbeat . The heartbeat interval under RMSP is longer than that of DFT-RMSP and has a larger variance for several reasons . First , the connection manager RMSP may be close to some nodes but very far from others . The heartbeat interval must be larger than the round-trip delay to such a distant node . Second , the heartbeat interval must be long enough to allow the RMSP connection manager to poll every node in the group and process all responses . A node that uses DFT-RMSP needs to process two heartbeat exchanges , one with its slave and one with its master , irrespective of the number of nodes . DFT-RMSP offers flexible and independent heartbeat intervals . In other words , there is no requirement that all masters use the same heartbeat interval or that the heartbeat interval per maintenance connection is fixed all the time . This is because a pair of adjacent nodes may exist on the same LAN or be far apart on a WAN . DFT-RMSP is scalable with respect to node-failure detection and isolation because because a node has only two maintenance connections and monitors only one other node irrespective og the number of nodes in the multicast group . DFT-RMSP is also scalable with respect to communication recovery because a node uses the multicast data-pipe to update the group with any failure it detects , using only one multicast message . 4 Down-Time Analysis Our objective in this section is to develop analytic models and derive expressions for multicast communications down-time for both the RMSP and DFT-RMSP protocols . We compare these two protocols in Section 5 and analyze the sensitivity of their down-time with respect to number of nodes , node failure rates , and network delays . Our approach to the analysis of down-time for these protocols is to determine the steadystate probability of having active multicast communications . We accomplish this by : a -RRB- defining a multicast-connection state , b -RRB- developing a state transition diagram , c -RRB- computing the state transition rates , d -RRB- reducing the diagram to a birth-death model , and e -RRB- solving the birth-death model to determine the steady-state probabilities . 4.1 State Transition Diagram We assume that all N nodes in a multicast group are independent and do not fail simultaneously . We also assume that a node fails with rate > = and recovers with rate ? . A node may be in one of three states : i -RRB- active , ii -RRB- undetected -LRB- faulty but not yet detected -RRB- , or iii -RRB- isolated -LRB- faulty , detected , and isolated -RRB- . We define the state of a multicast connection with a tuple -LRB- i ; j ; k -RRB- where i , j , and k represent the number of nodes in the undetected , active , and isolated states . The fundamental equation of the multicast-connection state is : i + j + k = N : -LRB- 1 -RRB- i , j , k i-1 , j +1 , k i , j +1 , k-1 i +1 , j,k-1 g i +1 , j-1 , k i,j-1 , k +1 i-1 , j , k +1 -LCB- -LRB- i +1 , j,k-1 -RRB- , -LRB- i , j , k -RRB- -RCB- -LCB- -LRB- i,j-1 , k +1 -RRB- , -LRB- i , j , k -RRB- -RCB- -LCB- -LRB- i , j , k -RRB- , -LRB- i-1 , j , k +1 -RRB- -RCB- -LCB- -LRB- i-1 , j +1 , k -RRB- , -LRB- i , j , k -RRB- -RCB- -LCB- -LRB- i , j , k -RRB- , -LRB- i , j +1 , k-1 -RRB- -RCB- a a -LCB- -LRB- i , j , k -RRB- , -LRB- i +1 , j-1 , k -RRB- -RCB- b b g Figure 4 : Transitions from/to a General State Figure 4 shows possible state transitions from state -LRB- i ; j ; k -RRB- . Note that the sum of a tuple 's parameters of any state is N . It is important to observe that a multicast-connection state transition occurs when exactly one node changes state . This is an essential property for a later Markov process analysis . Therefore , there are at most three possible transitions from a general state -LRB- i ; j ; k -RRB- that are triggered by a change in the state of a node : ffl a node fails , e.g. , -LRB- i ; j ; k -RRB- to -LRB- i + 1 ; j ? 1 ; k -RRB- , with transition rate ff -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i + 1 ; j ? 1 ; k -RRB- -RSB- -LRB- represented by a solid line -RRB- ffl a failed node is detected and isolated , e.g. , -LRB- i ; j ; k -RRB- to -LRB- i ? 1 ; j ; k ? 1 -RRB- , with transition rate -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k ? 1 -RRB- -RSB- -LRB- represented by a dotted line -RRB- ffl an isolated node has been recovered and reconnected , e.g. , -LRB- i ; j ; k -RRB- to -LRB- i ; j + 1 ; k ? 1 -RRB- , with transition rate -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ; j + 1 ; k ? 1 -RRB- -RSB- -LRB- represented by a dashed line -RRB- & reconnects 4l 3l 3l l l l l 3 1 2 4 4,0,0 3,1,0 3,0,1 2,0,2 2,1,12,2,0 1,1,2 1,2,1 1,3,0 1,0,3 0,0,4 0,1,30,2,20,3,10,4,0 2l 2l 2l detect & isolate faulty node node fails node recovers Figure 5 : 4-Node State Transition Model The failure rates on the transition arcs are independent of the multicast protocol . The failure rate from state -LRB- i ; j ; k -RRB- to -LRB- i + 1 ; j ? 1 ; k -RRB- is j > = since any one of j active nodes can fail . However , both failure-detection rates and recovery rates depend on the protocol and are analyzed in Section 4.3 . Figure 5 illustrates a transition diagram of a multicast group with four nodes . 4.2 Birth-Death Model One approach to solve a general N-node state transition diagram is to use the Chapman-Kolomogorov equation -LSB- Kle75 -RSB- . However , we are not interested in the steady-state behavior of all the different states of the multicast connection . Rather , we are only interested in those states where the multicast connection is up and active . Alternatively , we are interested in all -LRB- 0 ; j ; k -RRB- states where < = j ; k < = N and i + j = N . In fact , it is sufficient to obtain the sum of the steady-state probabilities for these -LRB- 0 ; j ; k -RRB- and not the individual ones . We now reduce the multicast-connection state transition diagram to a birth-death model . First , we observe in Figure 5 that states are grouped in layers or clusters and each state in a cluster has the same first parameter , i . Such a cluster is referred to as cluster i as it represents all the states that has i faulty but undetected nodes . The cluster where i = 0 , i.e. , no undetected faulty nodes , is of particular interest because it groups all the states where the multicast connection is active . Second , we observe that transitions from any state in cluster i either go to cluster i , cluster -LRB- i ? 1 -RRB- , or cluster -LRB- i + 1 -RRB- . Third , we observe that the number of states in cluster i is -LRB- N ? i + 1 -RRB- . This is because with i faulty nodes , the number of active nodes remaining in the multicast connection is either ; 1 ; 2 ; ? ? ? ; or -LRB- N ? i -RRB- , i.e. , -LRB- N ? i + 1 -RRB- distinct states . 1 N-1 N ... 2 m l 1 l l 2 l N-2 N-1 l mmm N m N-1 m 3 2 m 1 Figure 6 : n-Node Birth-Death Model The birth-death model that we choose is an -LRB- N + 1 -RRB- - state model where state i represents an aggregation of the states in cluster i. Figure 6 illustrates this model . The transition rate > = i from state i to state -LRB- i + 1 -RRB- is the sum of all transitions from any state in cluster i to any state in cluster -LRB- i + 1 -RRB- . The transition rate ? i is defned similarly . The transition rates within a cluster i are not required if we are only interested in the steady-state probability of this cluster . The steady state solution of this model can be found in -LSB- Kle75 -RSB- . The steady state probability Pn is the probability that system is in state n , 1 < = n < = N , and is given by Pn = oenP0 , where oen = > = 0 > = 1 > = 2 ? ? ? > = n ? 2 > = n ? 1 ? 1 ? 2 ? 3 ? ? ? ? n ? 1 ? n = n ? 1 Y i = 0 > = i ? i +1 : -LRB- 2 -RRB- We are only interested in the steady state probability P0 of the multicast connection being active . P0 must satisfy PNi = 0 Pi = 1 . Hence , P0 = 1 1 + PNn = 1 oen : -LRB- 3 -RRB- We use notation C , centralized , and D , distributed , as superscripts of any variable or expression to designate RMSP and DFT-RMSP protocols , respectively . 4.3 Transition Rates We start by deriving expressions for failure transition rates , > = i , < = i < = -LRB- N ? 1 -RRB- . As we described in Section 4.1 , failure rates from one state to another do not depend on the protocol . Thus , > = Ci = > = Di = > = i . The transition rate , > = i , from state i to state -LRB- i + 1 -RRB- , < = i < = -LRB- N ? 1 -RRB- , can be expressed as > = i = N?i X j = 0 ff -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- ; k = N ? i ? j : -LRB- 4 -RRB- However , the failure rate when j nodes are active is j > = . Therefore , we have ff -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- = j > = : -LRB- 5 -RRB- Rewriting Equation -LRB- 4 -RRB- we get > = i = N?i X j = 0 j > = = > = 2 -LRB- N ? i -RRB- -LRB- N ? i + 1 -RRB- : -LRB- 6 -RRB- We now derive expressions for failure detection and isolation rates , ? i , 1 < = i < = N . Recall that ? i is the sum of all transitions from state i to state -LRB- i ? 1 -RRB- and represents the rate of detecting and isolating exactly one faulty node out of i faulty nodes . Therefore , ? i can be expressed as ? i = N?i X j = 0 -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- ; k = N ? i ? j : -LRB- 7 -RRB- Transition rate -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- , which represents the rate of detecting and isolating exactly one faulty node out of i faulty nodes when j nodes are active , is what distinguishes DFT - RMSP from RMSP . This is because at this transition , DFT-RMSP has j active nodes that have the task of detecting and isolating one of i faulty nodes while , in case of RMSP , only the connection manager has this task . RMSP Detection/Isolation Rates To derive transition rate C -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- , we divide the number of faulty nodes , i , by the expected time that a connection manager takes to detect and isolate i faulty nodes , TC . Thus , C -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- = i = TC : -LRB- 8 -RRB- Figure 7 illustrates message exchanges and the associated processing requirements for a connection manager , CM , and three adjacent nodes , -LRB- i ? 1 -RRB- , i , and -LRB- i + 1 -RRB- , during three consecutive heartbeat intervals , each of duration Hc . We use this example to show a heartbeat message exchange between CM and i and the expected time for CM to detect and isolate node i after it fails and reinstate the multicast connection . In the first heartbeat interval , CM sends a heartbeat polling message to node i . The mean processing requirement at CM to send this message is xh . The message encounters a network delay with mean Tc . Node i receives the polling message with mean processing requirement yh and sends a response message to CM with mean processing requirement xh . The response message encounters a network delay with mean Tc . CM receives the response and updates node i status with mean processing requirements yh . In the second heartbeat interval , CM sends a heartbeat polling message to node i. However , node i failed sometime after responding to the first heartbeat polling message and before receiving the second one . The length of such period of time , TCb , as depicted in Figure 7 is given by TCb = Hc ? -LRB- yh + xh -RRB- : -LRB- 9 -RRB- This is because node i has one full heartbeat interval , Hc , to fail minus the processing time that it used to process the previous heartbeat , -LRB- yh + xh -RRB- . By the third heartbeat interval , CM detects that node i has failed because it has not received a heartbeat response from it . Therefore , the expected time , TCf , for CM to detect that node i has failed can be expressed as TCf = TCb = 2 + -LRB- Hc ? -LRB- xh + Tc -RRB- -RRB- = 3 2Hc ? 3 2xh ? 1 2yh ? Tc : -LRB- 10 -RRB- -LRB- xh + Tc -RRB- is subtracted from Hc because it is accounted for in Equation -LRB- 9 -RRB- . Upon detecting the failure of node i , CM processes a timeout and sends two reconfiguration requests to the predecessor and successor of node i ; that is , nodes -LRB- i ? 1 -RRB- and -LRB- i + 1 -RRB- , respectively . The respective mean processing requirements for these three tasks are xt , xr , and xr . Upon receipt of the reconfiguration requests from CM , nodes -LRB- i ? 1 -RRB- and -LRB- i + 1 -RRB- drop their connections with node i , establish a connection between themselves , and acknowledge CM with mean processing requirements yr , -LRB- xr + ya -RRB- , and xa , respectively . CM processes an acknowledgement from each node with mean processing requirements ya . We note that the mean network delay between nodes -LRB- i ? 1 -RRB- and -LRB- i + 1 -RRB- is Td . Therefore , the mean time TC to detect and isolate a faulty node is given by TC = TCf + 3xr + yr + xa + ya + Td + Tc : -LRB- 11 -RRB- When a faulty node recovers , it sends a reconnection request to CM . CM processes the request and determines two adjacent nodes that the recovering node should be inserted between . The mean processing requirements for CM to do this task is xv . Similar to fault-isolation processing , CM sends reconfiguration requests to the selected adjacent nodes . The two nodes establish connections with the recovering node and drop the connection between themselves . We assume that the impact on ongoing multicast is negligible , i.e , it is not interrupted . During a heartbeat interval , CM sends a configuration update message to each active node if one or more nodes are added or removed . If the mean processing requirements of various tasks equal ffi , we have xr = yr = xa = ya = xh = yh = xv = xu = ffi : -LRB- 12 -RRB- Using Equations -LRB- 10 -RRB- , -LRB- 11 -RRB- , and -LRB- 12 -RRB- , we rewrite TC as TC = 3 2Hc + Td ? 13 2 ffi : -LRB- 13 -RRB- We now derive a lower-bound expression for Hc , the heartbeat interval for RMSP . The connection manager performs the following tasks during a heartbeat interval while the multicast connection is in state -LRB- i ; j ; k -RRB- : ffl exchanges heartbeat messages with j active nodes with mean processing requirements j -LRB- xh + yh -RRB- and mean response time of -LRB- 2Tc + xh + yh -RRB- . ffl detects and isolates at most i faulty nodes with mean processing requirements i -LRB- xt +2 xr +2 ya -RRB- and mean response time of -LRB- 2Tc + Td -RRB- . ffl processes reconnection requests from at most k nodes with mean processing requirements k -LRB- xv + 2xr + 2ya -RRB- and mean response time of -LRB- 2Tc + Td -RRB- . ffl send topology update messages to j active nodes if necessary with mean processing requirements jxr . We observe that failure detection and reconnection requests incur higher processing requirements and have longer response time . Thus , the longest response time in a heartbeat interval occurs at the maximum number of undetected faulty nodes , N . Hence , the lower bound for a heartbeat interval must be long enough to detect and isolate N faulty nodes . Thus , we have Hc > = N -LRB- xt + 2xr + 2ya -RRB- + 2Tc + Td : -LRB- 14 -RRB- If the assumptions of Equation -LRB- 12 -RRB- hold , the lower bound on the heartbeat interval can be expressed as Hc > = 5ffiN + 2Tc + Td : -LRB- 15 -RRB- Substituting Equation -LRB- 15 -RRB- into Equation -LRB- 13 -RRB- , we get TC > = 15 2 ffiN + 3Tc + 5 2Td ? 13 2 ffi : -LRB- 16 -RRB- Finally , we compute the probability of an active RMSP multicast connection , that we derived in Equation -LRB- 3 -RRB- . First , We expand the numerator of oen using Equation -LRB- 6 -RRB- as follows n ? 1 Y i = 0 > = Ci = n ? 1 Y i = 0 > = 2 -LRB- n ? i -RRB- -LRB- n ? i + 1 -RRB- = -LRB- > = 2 -RRB- n n ! -LRB- n + 1 -RRB- ! : -LRB- 17 -RRB- Then , we expand the denominator of oen using Equations -LRB- 2 -RRB- and -LRB- 8 -RRB- nY i = 1 ? Ci = nY i = 1 n?i X j = 0 i = TC = nY i = 1 i -LRB- n ? i + 1 -RRB- = TC = n ! n ! = -LRB- TC -RRB- n : -LRB- 18 -RRB- Substituting Equations -LRB- 17 -RRB- and -LRB- 18 -RRB- into -LRB- 2 -RRB- oeCn = -LRB- n + 1 -RRB- -LRB- > = T C 2 -RRB- n : -LRB- 19 -RRB- Let ! = > = T C = 2 . Therefore , N X n = 1 oeCn = N X n = 1 -LRB- n + 1 -RRB- ! n = N X n = 1 d d !! -LRB- n +1 -RRB- = d d ! N X n = 1 ! -LRB- n +1 -RRB- = d d ! ! 2 -LRB- ! N ? 1 -RRB- ! ? 1 = N ! -LRB- N +1 -RRB- + 2 ! -LRB- ! N ? 1 -RRB- -LRB- ! ? 1 -RRB- ? ! 2 -LRB- ! N ? 1 -RRB- -LRB- ! ? 1 -RRB- 2 : -LRB- 20 -RRB- Using ! = > = T C = 2 and Equations -LRB- 3 -RRB- and -LRB- 20 -RRB- , we obtain the steady state probability that a RMSP multicast connection is active , PC0 = 1 = -LRB- 1 + PNn = 1 oeCn -RRB- . DFT-RMSP Detection/Isolation Rates We derive the failure detection and isolation rates for DFT-RMSP using similar analysis to that of RMSP . Let TD be the expected time for an active node to detect and isolate one faulty node . Figure 7 illustrates message exchanges and the associated processing requirements for DFT-RMSP from the perspective of node -LRB- i ? 1 -RRB- . Similar to the derivation of TC , we derive an expression for TD as follows : TDb = Hd ? -LRB- yh + xh -RRB- TDf = TDb = 2 + Hd ? -LRB- xh + Td -RRB- TD = TDf + 2xr + yr + ya + 2Td -LRB- 21 -RRB- and using the assumptions of Equation -LRB- 12 -RRB- , TD can be rewritten as follows : TD = 3 2Hd + Td + 2ffi : -LRB- 22 -RRB- h xh xt xr Hd Hd Hd yh yr xr xh yh ya T d T d T d T d node `` i '' failure detected T d xh xh xh xt xr xr xr yr xr yh yh T c T c T c Hc Hc Hc Hc xa ya T d yr xa ya T c T d T c T c ya ya x detected DFT-RMSP RMSP i +1 i node `` i '' may fail in this range node `` i '' failure isolated i-1 CM i-1 i +1 i node `` i '' may fail in this range node `` i '' failure isolated node `` i '' failure Figure 7 : Centralized vs. Distributed Heartbeat Processing Similar to the derivation of a lower-bound value to Hc , we can express the lower-bound value of DFT-RMSP heartbeat interval as Hd > = xt + 2xr + 2Td + yr + ya = 2Td + 5ffi : -LRB- 23 -RRB- However , it is not practical to use this lower-bound value of Hd because it means using all the resources of a node to do fault-tolerance activities . Thus , if OE is the percentage of a node 's resources dedicated to fault-tolerance , then the heartbeat interval must be longer than -LRB- 1 = OE -RRB- times the lowerbound value . For a multicast connection in state -LRB- i ; j ; k -RRB- , the mean transition rate to detect and isolate one faulty node can be expressed as D -LSB- -LRB- i ; j ; k -RRB- ; -LRB- i ? 1 ; j ; k + 1 -RRB- -RSB- = -LRB- j = TD if j < = i i = T D if j > i -LRB- 24 -RRB- We assume that the last node in a multicast group does not fail , i.e. , < j < = N . Thus , the birth-death model will only have N states -LRB- 0 ; 1 ; ? ? ? ; N ? 1 -RRB- . Therefore , the aggregate transition rate ? Di from cluster i to cluster -LRB- i ? 1 -RRB- can be expressed as ? Di = iX j = 1 j = TD + N?i X j = i +1 i = T D = -LRB- i -LRB- i +1 -RRB- 2TD if i < = N = 2 i -LRB- i +1 -RRB- 2TD + -LRB- N ? 2i -RRB- -LRB- N +1 -RRB- 2TD otherwise -LRB- 25 -RRB- We now have expressions for > = Di and ? Di in Equations -LRB- 6 -RRB- and -LRB- 25 -RRB- . Therefore , we compute oeDn = Qn ? 1 i = 0 > = Di ? Di +1 . We use this to compute the steady state probability PD0 that multicast communication is active for DFT-RMSP where PD0 = 1 = -LRB- 1 + QN ? 1 n = 1 oeDn . 5 A Performance Comparison In this section , we compare the steady state probabilities , PD0 and PC0 , of DFT-RMSP and RMSP , respectively . We refer to this probability as the multicast connection utilization . We examine the utilization sensitivity to node-failure rates , network delay , number of nodes , and length of heartbeat intervals . We set ffi to 1 in all computations . Figure 8 shows how DFT-RMSP multicast connection utilization , PD0 varies with the length of DFT-RMSP heartbeat interval Hd and with the number of nodes N when node-failure rate > = is constant . We set Hd to 1 , 10 , 20 , and 100 times the lower-bound value of the heartbeat interval . We observe that for low values of N , the utilization of DFT-RMSP increases slightly as N increases until it asymptotically reaches a constant value . This is because increasing the number of nodes yields higher rate of node-failure detection and isolation for DFT-RMSP . This is an important result because it shows that distributing fault-tolerance processing among all nodes reaches a constant multicast connection utilization irrespective of the number of nodes . We also observe that increasing the duration of the heartbeat interval decreases the utilization of DFT-RMSP . This is due to longer delays in detecting and isolating faulty nodes before reinstating multicast communication . We note that setting Hd to the lower-bound value of the heartbeat interval is not practical for DFT-RMSP because it dedicates the entire throughput of a node to fault-tolerance activities , i.e. , no resources for data multicasting . Setting Hc to 20 times the lowerbound value , for example , dedicates at most 5 % of a node 's processing resources to fault -LCB- tolerance activities . Figure 9 shows how RMSP multicast connection utilization PC0 varies with the length of RMSP heartbeat interval Hc and with the number of nodes N when node-failure rate > = is constant . For comparison , we show DFT-RMSP utilization PD0 when its heartbeat interval Hd is set to 20 times its lower-bound value . We set Hc to 1 , 2 , 5 , and 10 times the lower-bound value of the heartbeat interval . We observe that RMSP utilization decreases as the number of nodes increase . This is because the centralized connection manager for RMSP has more We also observe that the longer the heartbeat interval the lower the utilization of RMSP . This is due to longer delays in detecting and isolating faulty nodes . We note that RMSP does not have to use longer heartbeat intervals than its lower-bound value . This is because , unlike DFT-RMSP , fault-tolerance processing is performed at the connection manager with very little contribution from the multicast nodes . Figure 10 shows how multicast connection utilization PC0 and PC0 vary with node-failure rate > = when the number of nodes N is constant . RMSP uses a heartbeat interval that is set to its lower-bound value and DFT-RMSP uses a heartbeat interval that is 10 times its lower-bound value . We observe that the utilization of both protocols decreases as > = increases . This is due to a longer connection down-time required to detect and isolate an increasing number of faulty nodes . However , RMSP utilization decreases at a higher rate than DFT-RMSP utilization . This is because DFT-RMSP distributes the tasks of fault-tolerance among active nodes which lead to faster node-failure detection and isolation . In other word , that ratio of the utilization of multicast communications between DFT-RMSP and RMSP increases as node-failure rates increase . This is a fundamental result that shows the increased multicast throughput of DFT-RMSP compared to RMSP as nodes become more vulnerable to failure . Figure 11 shows how RMSP multicast connection utilization PC0 varies with the network delay Tc between the connection manager and a node in the multicast group . We set Tc to 1 , 10 , and 100 times Td , the network delay between to adjacent nodes on the virtual multicast ring . For comparison , we show DFT-RMSP utilization PD0 when its heartbeat interval Hd is set to 10 times its lower-bound value . We observe that RMSP throughput decreases linearly as the network delay increases . This is due to longer detection and isolation of faulty nodes . This important result shows how sensitive RMSP is to fluctuations in network delays between the connection manager and distant nodes in particular . Figure 12 shows how DFT-RMSP multicast connection utilization PC0 varies with the network delay Tc between the connection manager and a node in the multicast group . We set Td to 1 , 10 , and 100 given that ffi is set to 1 . As the network delay increases , the throughput of DFT-RMSP decreases , although at a slower rate than that of RMSP . This is because multiple nodes are involved in fault-detecetion and isolation . Figure 13 shows the steady state probability distribuition of states through 150 when the number of nodes is 150 . We observe that except for state 0 , DFT-RMSP has lower probability of being in state i compred with RMSP , i > 0 . We also observe that such probability decreases with higher rate for DFT-RMSP than RMSP as i increases . This is due to shorter fault-detection and isolation times by DFT-RMSP . 6 Summary and Concluding Remarks Node-failure detection and isolation using a centralized manager in multicast protocols suffers from poor scalabilty , icreased downtime , and decreased throughput . Distributing fault-tolerance tasks among more nodes improves the scalability of a protocol and decreases the time to detect failures . RMSP is a reliable multicast session protocol that uses a centralized connection manager to detect and isolat faulty nodes . In this report , we presented DFT-RMSP ; a distributed fault-detection , isolation , and recovery protocol variation for the RMSP protocol . DFT-RMSP does not use a conection manager , rather it distributes fault-tolerance responsibilities among all nodes where a node is responsible for detectin and isolating the failure of only one other node irrespective of the multicast group size . We developed analytic models to compute connection downtime and throughput degradation due to node failures . We compared the centralized fault-tolerance approach of RMSP with the distributed approach of DFT-RMSP and showed the decreased connection downtime and increased throughput using DFT-RMSP . We also analyzed the sesnsitivity of the down-time with respect to number of nodes , node failure rates , and network delays . References -LSB- AFM92 -RSB- S. Armstrong , A. Freier , and K. Marzullo . \ Multicast Transport Protocol '' . Internet Request for Comment -LRB- RFC 1301 -RRB- , February 1992 . -LSB- BZ93 -RSB- R. Braudes and S. Zabele . \ Requirements for Multicast Protocols '' . Internet Request for Comment -LRB- RFC 1458 -RRB- , May 1993 . -LSB- Che92 -RSB- D. Cheriton . \ Dissemination-Oriented Communication Systems '' . DSG Working Paper , 1992 . -LSB- CM84 -RSB- J. Chang and N. F. Maxemchuk . Reliable broadcast protocols . ACM Transactions on Computer Systems , 2 -LRB- 3 -RRB- :251 -LCB- 273 , August 1984 . -LSB- FJM +95 -RSB- S. Floyd , V. Jacobson , S. McCanne , C. Liu , and L. Zhang . \ A Reliable Multicast Framework for Light-weight Sessions and Application Level Framing '' . In Proceedings of ACM SIGCOMM ' 95 . ACM , August 1995 . -LSB- HSC95 -RSB- H. Holbrook , S. Singhal , and D. Cheriton . \ Log-Based Receiver-Reliable Multicast for Distributed Interactive Simulation '' . In ACM SIGCOMM Conference . ACM , August 1995 . -LSB- Jac94 -RSB- Van Jacobson . \ Multimedia Conferencing on the Internet '' . In SIGCOMM 1994 Tutorial Notes . ACM , August 1994 . -LSB- Kle75 -RSB- L. Kleinrock . Queueing Systems . John Wiley & Sons , 1975 . -LSB- LP96 -RSB- J. Lin and S. Paul . \ RMTP : A Reliable Multicast Transport Protocol '' . In INFOCOM ` 96 Conference . IEEE , March 1996 . -LSB- McC92 -RSB- S. McCanne . \ A Distributed Whiteboard for Network Conferencing '' . DSG Working Paper , 1992 . -LSB- MS97a -RSB- W. Mostafa and M. Singhal . \ A Reliable Multicast Session Protocol for Collaborative Continuous-Feed Applications '' . In 12th ACM Annual Symposium for Applied Computing -LSB- SAC ' 97 -RSB- . ACM , February 1997 . -LSB- MS97b -RSB- W. Mostafa and M. Singhal . \ Performance Analysis of a Reliable Multicast Session Protocol for Collaborative Continuous-Feed Applications '' . In Fifth International Symposium on Modeling , Analysis , and Simulation of Computer and Telecommunication Systems -LSB- MASCOTS ' 97 -RSB- . IEEE Computer Society , January 1997 . -LSB- PSK94 -RSB- S. Paul , K. Sabnani , and D. Kristol . \ Multicast Transport Protocols for High Speed Networks '' . In Proceedings of International Conference on Network Protocols , October 1994 . -LSB- SBD96 -RSB- B. Sabata , M. Brown , and B. Denny . \ Transport Protocol for Reliable Multicast : TRM '' . In Proccedings of the IASTED International Conference on Networks . IEEE , January 1996 . -LSB- YGS95 -RSB- R. Yavatkar , J. Griffioen , and M. Sudan . \ A Reliable Dissemination Protocol for Interactive Collaborative Applications '' . In ACM Multimedia Conference . ACM , 1995 . 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 50 100 150 200 250 300 Multicast Connection Utilization Number of Nodes -LRB- N -RRB- failure rate/node = 0.0001 RMSP Hd = min -LRB- Hd -RRB- Hd = 10 min -LRB- Hd -RRB- Hd = 20 min -LRB- Hd -RRB- Hd = 100 min -LRB- Hd -RRB- Figure 8 : DFT-RMSP Sensitivity to Heartbeat Interval 0.2 0.4 0.6 0.8 1 50 100 150 200 250 300 Multicast Connection Utilization Number of Nodes -LRB- N -RRB- failure rate/node = 0.0001 DFT-RMSP Hc = min -LRB- Hc -RRB- Hc = 2 min -LRB- Hc -RRB- Hc = 5 min -LRB- Hc -RRB- Hc = 10 min -LRB- Hc -RRB- Figure 9 : RMSP Sensitivity to Heartbeat Interval 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1e-05 0.0001 0.001 Multicast Connection Utilization Node Failure Rate N = 150 nodes RMSP DFT-RMSP Figure 10 : RMSP / DFT-RMSP Sensitivity to Failure Rates 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 50 100 150 200 250 300 Multicast Connection Utilization Number of Nodes -LRB- N -RRB- failure rate/node = 0.0001 DFT-RMSP Tc = Td Tc = 10 Td Tc = 100 Td Figure 11 : RMSP Sensitivity to Network Delay -LRB- Tc/Td ratio -RRB- 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 50 100 150 200 250 300 Multicast Connection Utilization Number of Nodes -LRB- N -RRB- failure rate/node = 0.0001 Td = 1 Td = 10 Td = 100 Figure 12 : DFT-RMSP Sensitivity to Network Delay 1e-300 1e-200 1e-100 1 20 40 60 80 100 120 140 Steady State Probability Steady State n -LRB- 0 < = n < = N -RRB- number of nodes = 150 , failure rate = 0.0001 RMSP DFT-RMSP Figure 13 : RMSP / DFT-RMSP Steady State Probability Distribution

O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

collaborative communication
reliable multicast

distibuted fault-tolerence
session protocols