An Efficient Checkpointing Method for Multicomputers with Wormhole Routing Kai Li ? Jeffrey F. Naughtony James S. Plank ? In International Journal of Parallel Programming , Vol . 20 , No. 3 , June , 1992 , pp 159-180 Abstract Efficient checkpointing and resumption of multicomputer applications is essential if multicomputers are to support time-sharing and the automatic resumption of jobs after a system failure . We present a checkpointing scheme that is transparent , imposes overhead only during checkpoints , requires minimal message logging , and allows for quick resumption of execution from a checkpointed image . Furthermore , the checkpointing algorithm allows each processor p to continue running the application being checkpointed except during the time that p is actively taking a local snapshot , and requires no global stop or freeze of the multicomputer . Since checkpointing multicomputer applications poses requirements different from those posed by checkpointing general distributed systems , existing distributed checkpointing schemes are inadequate for multicomputer checkpointing . Our checkpointing scheme makes use of special properties of wormhole routing networks to satisfy this new set of requirements . 1 Introduction Recently , multicomputer systems1 with large numbers of fast processors , high-speed interconnects , and concurrent I/O systems have become commercially available . While the raw hardware on these machines is powerful , multicomputers are still more difficult to use than uniprocessors and shared memory multiprocessors . Two important factors making multicomputers difficult or at least inconvenient to use are ? Department of Computer Science , Princeton University , Princeton , NJ 08544 . yDepartment of Computer Science , University of Wisconsin at Madison , Madison , WI 53706 1As defined by Athas and Seitz , a multicomputer is an ensemble of \ small programmable computers , called nodes , connected by a message-passing network -LSB- 2 -RSB- . '' ffl There is no facility for automatic resumption from failure points for long-running jobs . Because there is no resumption facility , if the multicomputer has to be taken down , perhaps because of a failure , perhaps for routine maintenance , all currently active jobs must be terminated . After the system is brought back up , these terminated jobs must be restarted from the beginning . This is a serious drawback , since it is precisely the long-running jobs that require the resources of a multicomputer . ffl There is no way to start a job , then suspend it because a higher priority job has arrived , then resume it after the higher priority job has terminated . This makes sharing a multicomputer difficult . Once a job has been allocated the multicomputer , or some portion of the multicomputer -LRB- e.g. , a subcube in a hypercube -RRB- , the job must run to completion or be aborted . To remove these difficulties requires a checkpoint/resume capability . Since a multicomputer can be viewed as a distributed system , schemes presented in the distributed checkpointing literature can be used for multicomputer checkpointing . However , multicomputers present a set of opportunities and requirements significantly different from those in general distributed systems . The special opportunities arise because multicomputers have a lower degree of node autonomy than do distributed systems , and because existing multicomputer interconnection networks use deterministic , deadlock-free routing algorithms . The predominant special requirement of multicomputer checkpointing is that both checkpointing and recovery must be fast . In fact , it is primarily this requirement that renders previous distributed system checkpointing algorithms insufficient for multicomputer checkpointing . Previous distributed checkpointing schemes fall into four broad categories : distributed database checkpointing schemes , pessimistic checkpointing schemes , optimistic checkpointing schemes , and Chandy-Lamport algorithms . The problem of taking distributed database checkpoints has been well-studied . However , distributed database checkpointing schemes , such as those presented in -LSB- 11 , 18 , 17 -RSB- , require that the checkpointed computation consist of a sequence of transactions . Since multicomputer applications are general purpose parallel programs , no such computational model can be assumed . Pessimistic schemes , such as those presented in -LSB- 3 , 20 -RSB- , require that every message be logged synchronously as it is sent or received . The advantage of pessimistic checkpointing is quick recovery | the synchronous logging of messages makes recovery from a checkpointed state trivial . However , the overhead of synchronously logging every message makes pessimistic schemes inappropriate for multicomputer application checkpointing . -LRB- The name \ pessimistic '' comes from the fact that these schemes optimize recovery at the expense of normal execution , implicitly assuming frequent failures . -RRB- Optimistic schemes , such as those presented in -LSB- 12 , 24 -RSB- , allow asynchronous logging of messages , and hence can be implemented to cause less overhead than the pessimistic schemes during normal operation . The name \ optimistic '' comes from the fact that these schemes optimize normal operation at the expense of recovery , implicitly assuming few failures . One of the strengths of the optimistic schemes is that they allow a large degree of autonomy among the nodes participating in the checkpointing . In particular , each node can choose to checkpoint its internal state and buffered messages at any time , independent of the other nodes . This autonomy does not come for free | it is a source of much of the complexity in the recovery algorithm . The optimistic schemes require rollback and need to replay their logs in order to discover and restore a consistent state . Rolling back and replaying can be time-consuming so that existing optimistic recovery methods are not appropriate for fast job swapping on multicomputers . The fourth class of checkpointing schemes -LSB- 23 , 14 , 13 , 1 , 5 -RSB- give up this node autonomy and attempt to strike a balance between the low overhead of optimistic schemes and the simple recovery of pessimistic schemes . In all of these algorithms , -LRB- which we call C algorithms as they stem from a paper by Chandy and Lamport -LSB- 4 -RSB- -RRB- , there are identifiable global snapshots , and each processor knows when a global snapshot begins and ends . This allows for quick recovery from checkpointed images , since there is no need for a complex distributed algorithm to determine from which checkpointed state each processor should resume . The CL algorithms view a distributed system as a set of processes with I O channels for communication . While their method seems to be the most natural for multicomputer checkpointing , it fails for the following reason : or the algorithm to work correctly , each processor must send a special marker message to all other processors to which it has a channel . If the network is fully connected , this means -LRB- n2 -RRB- messages . While the processors in current multicomputers are not physically fully connected , the \ wormhole '' or \ virtual cut-through '' routing networks used in most multicomputers enforce the illusion of full connectivity at the processor level . That is , if a message from processor p1 to processor p2 passes through the router at the node for processor p3 , there is no way for any of p1 , p2 , or p3 to detect this fact . As far as the three processors are concerned , the message was sent over a direct link between p1 and p2 . This virtual full connectivity means that an implementation of the Chandy and Lamport checkpointing algorithm on a multicomputer will have message complexity -LRB- n2 -RRB- , which will render the PPP1231mm2m3CCC231timem4 igure 1 : A sample distributed system . scheme unacceptably slow when n scales to large machines . An improvement to the Chandy and Lamport technique has been suggested in -LSB- 25 -RSB- , but it degrades to -LRB- n2 -RRB- messages as the interval between checkpoints grows larger . A main contribution of this paper is a new checkpointing algorithm , which stems from the CL algorithms , but is also based on the deadlock-free routing algorithms employed by multicomputers . We refer to this algorithm as the etwor weepin al orit , since it conceptually \ sweeps '' through the interconnection network , enabling processes to checkpoint without sending messages to all other processes . The etwork Sweeping algorithm requires far fewer than -LRB- n2 -RRB- messages | -LRB- n log n -RRB- for hypercube-connected multicomputers such as the Intel iPSC/2 and iPSC/8 0 , and -LRB- n -RRB- for mesh-connected multicomputers . These bounds hold in spite of the virtual full connectivity provided by the multicomputer interconnection networks . Checkpointing omencl ture The standard way of visualizing a distributed group of processes is to use horizontal lines to denote the relative progress of the processes over time . A message between processes is denoted by an arrow from the point at which the sending process sends the message to the point at which the receiving process receives it . A cut is represented by a dotted line that crosses each process 's time line exactly once . A message is said to cross the cut line from left to right if its sending point is to the left of the cut line , and its receiving point is to the right of the cut line . Crossing from right to left is defined similarly . A consistent cut is a cut in which no messages cross the cut from right to left . Other cuts are called inconsistent . igure 1 shows a distributed system with three processes , four messages , and three cuts . The first two cuts -LRB- 1 and 2 -RRB- are consistent as no messages cross them from right to left . The last -LRB- 3 -RRB- is inconsistent due to message 3 . Checkpoints are related to cuts as follows : A distributed checkpoint consists of process state and essa e state . The process state is created from individual local c ec points of each process , and the message state is a lo of messages from one process to another . iven a cut , one can construct a distributed checkpoint by having each process p take a local checkpoint at the point where the 's cut line crosses p 's time line , and by logging each message that crosses the cut line from left to right . A distributed checkpoint is restarted by starting each process from its local checkpoint , and then resending the messages in the log . It is well-known -LSB- 13 , 5 -RSB- that checkpoints can only be made from consistent cuts . Inconsistent cuts lead to the recording of spurious messages in the process state of the checkpoint . ost checkpointing algorithms concentrate on finding consistent cuts for the taking of local checkpoints , and on discovering which messages comprise the message log . A series of checkpoints divides the execution of the checkpointed program into distinct intervals that can be visualized as being the spaces between the checkpoints ' cuts . urther , checkpoints divide all messages into two types : Cross-cut messages , which start in one checkpointing interval , and end in another , and non-cross-cut messages , which are contained in a single checkpoint interval . or example , messages 1 , 2 and 3 in igure 1 are cross-cut messages ; 4 is a non-cross-cut message . inally , we make the assumption that there is one process per processor in the multicomputer . This makes it possible to ignore the distinction between processor and process , so for clarity of exposition we refer to messages passing between processors rather than processes . he Ch nd - mport Checkpointing Algorithm The original Chandy-Lamport checkpointing algorithm has all processors periodically define a cut and make a distributed checkpoint . Its main assumption is that the channels between processors enforce a I O ordering among messages . The algorithm works as follows : Some processor p0 starts the checkpoint by taking its local checkpoint , and then broadcasting a special ar er message C to all processors to which it has a channel . All processors p then perform the following : ffl If p receives C and has n't taken its local checkpoint yet , then it takes its local checkpoint , and broadcasts C to all processors to which it has a channel . ffl Afterwards , if p receives a non-marker message along a channel c , and it has not yet received C on c , then is a cross-cut message , and must be logged . ffl When all processors have received C on all their incoming channels , then there are no more cross-cut messages to be logged : The checkpoint is finished . This algorithm performs three major functions : irst , it ensures that the local checkpoints will define a consistent cut , as any message that would cross the cut from right to left would have to bypass an C message , which is illegal given the assumption of I O channels . Second , it correctly identifies cross-cut messages : These are the ones received by a processor p on channel c after p has checkpointed , but before p has received C on c. Third , it provides notification to each p that p will receive no more cross-cut messages : This is when p has received C on every incoming channel . As stated before , the problem with this algorithm when applied to multicomputers with wormhole routing is that at the process level , the processors are fully connected . -LRB- In the preceding terminology , every processor p has a channel to every other processor . -RRB- Thus , every processor must send C to every other processor , leading to -LRB- n2 -RRB- messages on multicomputers with n processors . Of all the variants of the Chandy-Lamport algorithm , only -LSB- 25 -RSB- attempts to reduce this number of messages . It optimizes the CL algorithm by having each processor remember to which processors it has sent messages in the previous checkpoint interval . Then it only sends C to those processors . -LRB- The algorithm uses additional acknowledgement and synchronization messages to ensure correctness without sending the full n2 messages . -RRB- This is indeed an optimization , however its worst case is still -LRB- n2 -RRB- messages , and it depends upon how many pairs of processors exchange messages between cuts . nless the patterns of communication in the application program are very restrictive , as the size of checkpoint intervals grows , the number of C messages that a processor must send will also grow . he etwork weeping Algorithm In this section , we present our checkpointing algorithm that greatly reduces the number of marker messages that a processor must send . We call this the etwor weepin algorithm as it uses properties of wormhole routing to \ sweep '' the interconnection network of a multicomputer , and checkpoint in less than -LRB- n2 -RRB- messages . To state a bound on the number of messages sent by the checkpointing algorithm , we define two functions : -LRB- n -RRB- = the number of physical communication links in the multicomputer pp -LRB- n -RRB- = the number of distinct processor pairs -LRB- pi ; p -RRB- such that pi sends p an application program message while the checkpoint is in progress . On a hypercube like the Intel iPSC/2 or iPSC/8 0 , -LRB- n -RRB- is -LRB- n log n -RRB- ; on the mesh-connected multicomputers -LRB- such as the Intel Paragon -RRB- , -LRB- n -RRB- is -LRB- n -RRB- . The function pp -LRB- n -RRB- depends upon the behavior of the application program . However , it is critical to note that pp -LRB- n -RRB- depends on the number of pairs of processors that communicate while the checkpoint is actuall in pro ress -LRB- i.e. during the time interval bounded by the first processor beginning its local checkpoint and last processor learning that the checkpoint is over -RRB- , rather than the number of pairs of processors that communicate between cuts as in -LSB- 25 -RSB- . As a side note , in -LSB- 25 -RSB- , the authors prove that their scheme achieves the theoretical lower bound on the number of marker messages that must be exchanged to take a checkpoint . While our scheme uses fewer messages than that lower bound , this is not a contradiction , since our scheme makes stronger assumptions about the interconnection network and routing scheme than those used in the proof in -LSB- 25 -RSB- . As stated before , the Chandy-Lamport algorithm achieves three functions -LRB- determining a consistent cut , identifying cross-cut messages , and guaranteeing that there are no more cross-cut messages -RRB- with just one type of marker message -LRB- C -RRB- . Our algorithm differs in that it uses three types of markers : 1 . The first type -LRB- which we still call C -RRB- is like the C of the CL algorithms as it is used to define a consistent cut and help define the cross-cut messages . It differs , however , as it need not be sent to all neighboring processors , and as it contains a one-bit sequence number . The C messages can be sent between arbitrary pairs of processors in the multicomputer . 2 . The second type of marker message -LRB- which we call , for \ etwork Sweeping '' -RRB- is used by a processor to determine when there are no more cross-cut messages for the processor to log . The messages are sent between pairs of processors connected by a physical link when it has been determined that no more cross-cut messages will be sent along that link . -LRB- We consider links to be unidirectional . We view bidirectional links as two unidirectional links -RRB- . We discuss how to make this determination below in section 5 . ntil then , the above definition suffices for our explanation . 3 . The third type of message -LRB- which we call for \ Stop essage '' -RRB- is used by all processors to determine when the checkpoint is over . The combination of the three types of messages achieves the same three functions as the CL algorithms above , using a smaller overall number of markers , yet still allowing the individual processors to perform useful work while the checkpoint is being taken . Our algorithm works as follows : To oversee the taking of the checkpoint , we assign one processor pc to be the coor inatin processor . This processor is responsible for starting the checkpoint , and for knowing when the checkpoint is over . In order to initiate global checkpoint , pc first takes a local checkpoint , then broadcasts C with sequence bit -LRB- mod 2 -RRB- to all other processors . All processors p then perform the following : ffl If p receives C or and has n't taken its local checkpoint yet , then p takes its local checkpoint . ffl After taking its local checkpoint , p may start sending messages . As stated above , it only sends to p0 if p and p0 are connected by a physical link . It does so when it is sure that no more cross-cut messages can be sent along that link . When p has sent on all its outgoing links , and has received on all its incoming links , then we say that p has nis e sweepin . ffl After taking its local checkpoint and until the checkpoint is over , if p sends a non-marker message to p0 and p has not sent C to p0 , then it must send C with a sequence bit of -LRB- mod 2 -RRB- to p0 before sending . ffl After taking its local checkpoint and until it is finished sweeping , if p receives a non-marker message from processor p0 , and p has not yet received C from p0 , then is a cross-cut message : logs . ffl When p has finished sweeping , then it knows that it will receive no more cross-cut messages . Therefore , p commits all of its checkpointing information -LRB- its local checkpoint and message log -RRB- to disk . Afterwards , it notifies pc by sending it . ffl When pc receives from all processors , checkpoint is over . It broadcasts back to all processors to let them know . After sending , pc is free to start the next checkpoint . ffl Before p finishes sweeping , if p receives a C message with a sequence bit of -LRB- -LRB- 1 -RRB- mod 2 -RRB- , then it ignores the C . Similarly , after it finishes sweeping p should ignore any C with a sequence bit of -LRB- mod 2 -RRB- . ffl When p receives from pc , then it knows that checkpoint is over . It stops sending C messages . 2 Pc P1 P2 igure 2 : A simple multicomputer . To give a bit of intuition about how this algorithm works , we present an example in which the multicomputer of igure 2 takes a checkpoint . Although this multicomputer is extremely simple , it is sufficient to illustrate our points . In this example , we assume that all processors have finished taking checkpoint 0 . We show how they take checkpoint 1 , where there is one cross-cut message and one non-cross-cut message . The events are shown graphically in igure 3 . We detail them below : 12345678910TimemCL MarkermNS MarkermC MessageNormal MessagePcP1P2mm ` Pc does n't log m ` P1 logs m igure 3 : Time line of xample 1 . 1 . At time 1 , pc decides that it wants to start checkpoint 1 . Thus it checkpoints itself , and broadcasts C with a sequence bit of 1 to the other two processors . After it checkpoints itself , pc ca n't send any more cross-cut messages . Therefore , its link to p1 wo n't receive any more cross-cut messages : pc sends to p1 . 2 . At time 2 , p2 has not yet received C . Its application program sends the message to p1 . 3 . At time 3 , p1 and p2 both receive C from pc . Thus they checkpoint themselves . ow , since it ca n't send any more cross-cut messages , p2 knows that its link to p1 will not receive any more cross-cut messages : It sends to p1 . 2 ne might note that it 's possible for to receive a L or message from checkpoint before receiving from checkpoint . If this happens , can simply use the L or to ag both the end of checkpoint and the beginning of checkpoint . It then ignores the when it arrives . here can be no confusion between the from checkpoint and the one from checkpoint : he former must be received by before finishes sweeping for checkpoint . he latter must be received afterwards . 4 . At time 4 , p1 receives from p2 . Since it has n't received C from p1 , it knows that is a cross-cut message : p1 logs . 5 . At time 5 , the two messages bound for p1 arrive . Since p1 has checkpointed , and since no cross-cut messages will come through its incoming links , p1 knows that no cross-cut messages will be sent through its outgoing links . Therefore , p1 sends out both of those links . Afterwards , it is finished sweeping . It notifies pc by sending it . . ext , at time , p2 's application program has resumed , and wants to send the message to pc . It must precede with C . 7 . At time 7 , pc receives the C and from p2 . Because of the C marker , pc knows that is not a cross-cut message . Therefore pc does n't log . 8 . At time 8 , pc and p2 both receive the messages from p1 . Thus they are finished sweeping . 2 notifies pc of this by sending to pc . . At time , pc receives the messages from p1 and p2 . Checkpoint 1 is over . c then broadcasts back to p1 and p2 to let them know . 10 . At time 10 , p1 and p2 receive from pc . Thus they know that checkpoint 1 is over . After this point , they stop sending C messages . Thus , the checkpoint is correctly taken , and the one cross-cut message is successfully logged . ote that the processors ' decisions about when to send messages depend on the underlying interconnection network . We formalize how the processors make these decisions in section 5 below . The total number of messages that this algorithm uses is as follows : -LRB- n -RRB- for the initial broadcast of C messages by pc pp -LRB- n -RRB- for the other C messages -LRB- -LRB- n -RRB- -RRB- for the messages -LRB- n -RRB- for the messages Since -LRB- n -RRB- must be at least n ? 1 for the multicomputer to be connected , the total number of messages can be simplified to -LRB- -LRB- n -RRB- pp -LRB- n -RRB- -RRB- . We later describe an optimization that gets rid of the pp -LRB- n -RRB- term , and thus checkpoints with only -LRB- -LRB- n -RRB- -RRB- extra messages . We provide the following lemmas to prove the algorithm 's correctness , and to further illustrate some of the details of the algorithm . ur etwor weepin al orit e nes a consistent cut . The only way this can be false is if there is a message crossing the checkpoint 's cut from right to left . In other words , it is sent by a processor p after p checkpoints , and received by p0 before p0 checkpoints . ow , assume that p has checkpointed itself , but that p0 has not . Since p is in the interval between checkpointing and receiving from pc -LRB- pc ca n't send until after p0 checkpoints -RRB- p must precede with C . When p0 receives C , it checkpoints itself . Therefore , when it receives , it will have checkpointed itself : can not cross the cut from right to left , and the cut is consistent . This next lemma is to provide justification for the sequence bit in the C messages . One might notice that a processor p can receive C messages well after receiving its final message from pc . This lemma shows that these C messages are properly ignored , even if another checkpoint is started soon after the current one finishes . o essa e can cross ore t an one cut . Suppose processor p sends the message to p0 , and crosses more than one cut . Since all cuts are consistent , this means that p sends in checkpoint interval , and p0 receives in interval where 1 . This also means that p0 receives it along a link a ter receiving for checkpoint 1 on that link . This is a contradiction from the definition of messages . Therefore it is not possible for such an to exist . The way this lemma ties in with the sequence bit is as follows : Suppose that processor p has just taken its local checkpoint for global checkpoint but p has n't finished sweeping . Then p will ignore C messages with sequence bits of -LRB- -LRB- 1 -RRB- mod 2 -RRB- . Since no message -LRB- including markers -RRB- can cross more than one cut , the C messages that p does n't ignore are those from checkpoint interval . These are exactly the ones that p needs to decide whether or not to log a non-marker message . Suppose now that p has finished sweeping for checkpoint , but has n't started checkpoint 1 . The only C messages that p cares about are those from interval 1 . Since p ignores C markers with bits of -LRB- mod 2 -RRB- and since no message can cross more than one cut , the C marker that p does n't ignore does indeed come from checkpointing interval 1 . The next two lemmas show that the etwork Sweeping algorithm correctly logs messages . ll cross-cut essa es are lo e . or a message to be a cross-cut message , it must be sent by p before p starts checkpoint , and received by p0 after p0 starts checkpoint . By the definition of the messages , must be received by p0 before p0 finishes sweeping . Thus , p0 will log if and only if it does n't receive a C message from p prior to receiving . As stated above , the only C messages that are n't ignored are those from checkpoint . Since p sends before starting checkpoint , p0 wo n't receive any such C message before receiving , and thus it will recognize as a cross-cut message . will be properly logged . o non-cross-cut essa es are lo e . on-cross-cut messages come in two varieties : Those sent by p after p starts checkpointing and before p stops checkpointing , and those sent by p after p stops the current checkpoint and before p starts the next checkpoint . In either case , the messages are not logged . If p sends before receiving from pc , then must be preceded by an C message . If is received by p0 before it is finished sweeping , then p0 knows from the receipt of the C marker that is a non-cross-cut message . If it receives after it is finished sweeping , then the C message does n't matter : p0 already knows that is a non-cross-cut message . In either case , p0 does n't log . If p sends after receiving from pc , then it is guaranteed that p0 has finished sweeping : Again , p0 will recognize as a non-cross-cut message , and will not log it . Therefore , these lemmas show that our etwork Sweeping algorithm is correct : It defines a consistent cut , and correctly identifies and logs the cross-cut messages . oreover , the lemmas provide some justification for various parts of the algorithm : The sequence bit , the markers and the messages are all necessary for the proofs . ending the Mess ges on Wormhole Routing etworks Above we state that a processor p sends a marker along a link when it determines that no more cross-cut messages will be sent along that link . However , we do n't state specifically how p makes this determination . The example in igure 2 was simple enough to provide an a oc solution , but here we consider the general case . We are aided by the fact that while the interconnects on wormhole-routing multicomputers provide the illusion of full connectivity , in reality each of the -LRB- n2 -RRB- paths between processors corresponds to a route through some subset of the physical links of the system . urthermore , with the exception of the original store-and-forward hypercube multicomputers , all of the multicomputer interconnection networks of which we are aware satisfy the following three conditions : 1 . Routing in the system is deterministic , using a deadlock-free routing algorithm as defined by ally and Seitz -LSB- -RSB- . 2 . Allocation of links along a route is non-preemptive ; that is , once a message is allocated a link along its route , no other message travels along the same link until the message is on the next link in the route -LRB- or until the message is at the destination processor -RRB- . 3 . ueueing for incoming messages in the system is strictly I O . We call an interconnection network that satisfies these conditions a well - e a e interconnection network . any existing and proposed multicomputers are well-behaved . Among these are Intel 's iPSC/2 , iPSC/8 0 , elta , Sigma and Paragon machines , Stanford 's ASH -LSB- 15 -RSB- , and IT 's - achine -LSB- 8 -RSB- . The first two of these use hypercube interconnections , while the last four are two and three-dimensional meshes . or well-behaved interconnection networks , we present a method for sending the messages . We prove that the method is guaranteed to start and terminate , and that it does what it promises : When is received on a link , no more cross-cut messages will be received on that link . Afterwards , we display how the method works for hypercubes and two-dimensional , toroidal meshes . Since a constant number of messages are sent per physical link of the interconnection network , the total number of messages is -LRB- -LRB- n -RRB- -RRB- . - ally and Seitz -LSB- -RSB- define an interconnection network to be a strongly connected , directed graph = -LRB- ; -RRB- , where is the set of nodes , and is the set of channels , or physical links that connect the nodes . A routing function : maps the current channel ci upon which a message resides , and its destination node n , to the next channel c that it will take on its way to n . Thus , is a non-adaptive , memoryless routing function . It 's non-adaptive because it always yields the same path from one node to another , and it 's memoryless because the function works from channel to channel , retaining no knowledge of the source node once it 's underway . To prove a routing function deadlock-free , they define a c annel epen enc rap = -LRB- ; -RRB- , where is the same set of channels described above , and is a set of edges determined by as follows : = -LRB- ci ; c -RRB- -LRB- ci ; n -RRB- = c for some n They show that is deadlock-free if and only if has no cycles . We use the results of ally and Seitz to define how to send the marker messages . It is assumed that each processor pi resides on node ni . oreover , if the interconnection network contains the channel c = -LRB- ni ; n -RRB- , then the path of a message from pi to p is routed through c. inally , when we say \ pi sends on channel c , '' we mean that if c = -LRB- ni ; n -RRB- , then pi sends to p , thereby using channel c . The method is quite simple : ffl Assume that pi has checkpointed itself . ffl or all outgoing channels c from pi , if there is no c such that -LRB- c ; c -RRB- , then pi sends the marker on channel c . ffl or all other outgoing channels c from pi , when markers have been received from all channels c such that -LRB- c ; c -RRB- , pi sends on channel c . The following lemmas prove the method 's correctness : ntil essa es a e een sent on all c annels t ere is alwa s so e processor t at a sen a essa e . Since is a graph with no cycles , we can number the channels from 1 to in such a way that if -LRB- ci ; c -RRB- , then . ow , since there is no ci such that -LRB- ci ; c1 -RRB- , a marker can instantly be sent on c1 . The proof proceeds by induction . Suppose that has been sent on channels c1 through c , but not on channel c 1 . Consider all ci such that -LRB- ci ; c 1 -RRB- . rom our numbering , 1 , so has been sent out all channels ci . As is deadlock-free , the markers are eventually received , and can be sent out c 1 . Therefore , there is always a channel that is ready to receive a marker . en is recei e on a c annel c it is uarantee t at no ore cross-cut essa es will e sent or recei e alon c. Suppose p sends a cross-cut message to p0 . That is , is sent before p has taken its local checkpoint , and received after p0 has taken its local checkpoint . urther , let c1 ; ... ; c be the route that takes ; in other words , -LRB- c 1 ; n -RRB- = c for 1 . We show that always travels along a channel ci before can be sent on ci . Since allocation of a route is non-preemptive , and channels are I O , will be put on channel c1 before p can send out another message on channel c1 . In particular , is put on c1 before p can put on c1 . We now show that is put on all channels c1 ; ... ; c before any processor can put there . This is done by induction . irst , as noted above , is put on c1 before can be put there . ext , suppose that for all , travels through c before a message can be put there . Before can be put there , must be able to allocate c 1 . Therefore is put on channel c 1 before is put on channel c . ow , since -LRB- c ; n -RRB- = c 1 , -LRB- c ; c 1 -RRB- , and will not be sent on c 1 until after it has been received on c . As stated above , wo n't be received on c until after has been put on c 1 . Thus , wo n't be put on c 1 until after has been put there . Therefore it is impossible for a cross-cut message to travel on a channel c after has been put there . Thus , when is received along a channel , it is guaranteed that no more cross-cut messages will be use that channel . Dimension 0 links Dimension 1 links Dimension 2 links 000 001 010 011 100 101 110 111 igure 4 : A 3-dimensional hypercube The Intel iPSC/2 and iPSC/8 -LSB- 1 -RSB- are multicomputers with hypercube interconnection networks that are well-behaved . ach process is associated with a node on a hypercube . In a - dimensional hypercube , nodes are numbered through 2 ? 1 . If the binary representation of nodes and differ by exactly their - th bit , then they are physically connected by a link that is said to be in dimension . As an example , igure 4 shows a 3-dimensional hypercube with its links . ach link is bidirectional , and thus can be divided into two unidirectional channels . The iPSC/2 and iPSC/8 both use the e-cu e routing function , which is deadlock-free . It specifies that the only channels used are those in the dimensions in that the binary representation of the source and destination nodes differ . These channels are used in increasing order of dimension . or example in igure 4 , a message from node 000 to node 111 will pass through the channels -LRB- 000 ; 001 -RRB- ; -LRB- 001 ; 011 -RRB- , then -LRB- 011 ; 111 -RRB- . To put this in the notation of ally and Seitz , let ci = -LRB- n ; n -RRB- . Then -LRB- ci ; n -RRB- = c , where c is the channel emanating from n in the lowest dimension in that the binary representations of n and n differ . or example , in igure 4 , -LRB- -LRB- 000 ; 001 -RRB- ; 111 -RRB- = -LRB- 001 ; 011 -RRB- . The method for sending markers therefore works as follows : ffl After processor p takes its local checkpoint , it sends on its outgoing channel c0 of dimension 0 . This is because c0 always starts a message route : There are no c such that -LRB- c ; c0 -RRB- . ffl sends on its outgoing channel c of dimension only after it has received on each incoming channel c of dimension . This is because for each of those c , -LRB- c ; c -RRB- . This method results in two markers sent per physical link . As each node has log n links incident to it , the total number of messages is n log n , which is -LRB- -LRB- n -RRB- -RRB- for the hypercube . As a second example for sending messages , we consider a toroidal mesh . The next generation multicomputers , including the Intel Paragon use toroidal mesh interconnection networks . In these interconnection networks , the nodes are laid out on a grid , and each node has two outgoing links : one to its right , and one going up . To be more formal , in a toroidal mesh , = -LRB- ; -RRB- , where : = ni and = -LRB- ni ; ni -RRB- = -LRB- 1 -RRB- mod -LRB- ni ; ni -RRB- = -LRB- 1 -RRB- mod As before , for the purposes of the deadlock-free routing function , we split each link into two channels . This time we call them i and low channels , for reasons discussed below . To be more clear , we give each channel a 4-digit identification number \ c , '' where = channel 's dimension : 1 if it is of the form -LRB- ni ; ni -RRB- , otherwise . = 1 if it 's a high channel , if it 's a low channel . = row number of its source node . c = column number of its source node . 00 10 20 01 11 21 02 12 22 0101 0102 01000100 0001 0002 00000000 0111 0112 01100110 0011 0012 00100010 0121 0122 01200120 0021 0022 00200020 1110 1120 11001100 1010 1020 10001000 1111 1121 11011101 1011 1021 10011001 1112 1122 11021102 1012 1022 10021002 igure 5 : A 3 3 toroidal mesh igure 5 shows an example of a 3 3 toroidal mesh . eadlock-free routing on the mesh is similar to the routing on the hypercube . The message first gets routed to the correct column , and then to the correct row . When routing to the correct column , low channels are used when the number of the current column is greater than the number of the desired column , and when routing to the correct row , low channels are used when the number of the current row is greater than the number of the desired row . We formally define in terms of the current node n c and the destination node ni as follows : -LRB- n c ; ni -RRB- = c00 c if c c01 c if c c10 c if c = and c11 c if c = and The definition of in terms of the current channel and destination node is slightly more complex , but follows directly . We omit it for the sake of clarity . The first thing to notice about this function is that channels with identification numbers of the form c00 , c01 C 1 , c100c , and c11 1 c never get used . Thus , they can be removed from the network . ext , a moment 's thought shows that in terms of the channel identification numbers , this function is strictly increasing . In other words , if -LRB- ci ; c -RRB- , then . Therefore , the channel 17 dependency graph has no cycles , and the routing function is deadlock-free . The method for sending the messages works in a manner similar to the hypercube example : i sends a message on outgoing channel c i only after it has received messages from each incoming channel c i where . Thus , unlike the hypercube method , where every processor can start sending messages after checkpointing , on a mesh only the processors pi1 can start by sending on their channel c00i1 . Like the hypercube algorithm , however , it sends approximately two messages per physical link , which results in 4n total messages . - - A - ary - cube is a general class of interconnection network topologies in which there are nodes . These nodes are laid out in dimensions , with sets of nodes in each dimension . Channels connect neighboring nodes that differ in one dimension . Both hypercubes and toroidal meshes are examples of - ary - cubes : A - dimensional hypercube is a binary - cube , and a - by - toroidal mesh is a - ary 2-cube . eneral - ary - cubes are important because ally has shown that multicomputers built using low-dimensional - ary - cubes as their interconnection networks give optimally low latency and high throughput -LSB- -RSB- . Accordingly , many new multicomputers , such as the Paragon , Stanford 's ASH -LSB- 15 -RSB- , the - achine -LSB- 8 -RSB- , the Ametek 2010 -LSB- 21 -RSB- and the osaic -LSB- 22 -RSB- , have been designed to be - ary 2 and 3-cubes . Since all of these machines use wormhole routing and are well-behaved , they too can be checkpointed with our etwork Sweeping algorithm . We do n't include the details here , but they follow as an extension of the mesh example above . inally , ally has proposed press Cu es as an optimization to standard - ary - cubes -LSB- 7 -RSB- . xpress cubes use extra e press c annels for messages to jump over a number of nodes in each dimension so that the node-latency of a message 's transmission is reduced . If the space allows , multiple express channels can be added hierarchically to each dimension to give more opportunity for greater throughput . He shows that express cubes both lower latency and increase throughput . Again , multicomputers built with express channels are well-behaved , and thus can be checkpointed with the etwork Sweeping algorithm . The extension here is simply to have messages sent along express channels as well as the normal channels . Reco er Recovery in our algorithm is the same as recovery in all the CL algorithms : ach processor reads its process state from its local checkpoint , and messages from the message log are replayed . This replaying can take one of two forms : ither each processor can resend each of its messages which was logged by the receiving processor , or each processor can rebuild its queue of messages from its own message log . The latter form is superior since it involves no extra communication between processors , and no reading of other processors ' message logs . in l An l sis nd ptimi tions Our algorithm for checkpointing a multicomputer gives an overhead of -LRB- -LRB- n -RRB- pp -LRB- n -RRB- -RRB- extra messages . As stated before , pp -LRB- n -RRB- is equal to the number of distinct pairs of processors that communicate during the checkpoint . In the worst case , this number is -LRB- n2 -RRB- , but this worst case is only realized if every processor sends an application program message to every other processor in the multicomputer while the checkpoint is in progress -LRB- this is as opposed to , or rather is a subset of -LSB- 25 -RSB- , where the worst case is realized when every processor sends a message to every other processor during an entire checkpointing interval -RRB- . As a potential optimization -LRB- similar to the one in -LSB- 14 -RSB- -RRB- , we can eliminate the need for extra C markers if we simply attach an extra bit to each normal message in the system -LRB- on some multicomputers , like the iPSC/2 , this bit comes for free , as there is an extra bit in message headers -RRB- . Then , for messages coming from even checkpointing intervals , we set this bit to and for those coming from odd intervals , we set it to 1 . ow there is no need for C messages after pc 's initial broadcast , because a processor can tell if it needs to checkpoint or if it needs to log an incoming message simply by looking at this extra bit . Thus with this method , there is no possibility of degradation to -LRB- n2 -RRB- markers instead , the number of extra messages is merely -LRB- -LRB- n -RRB- -RRB- , which is -LRB- n log n -RRB- for hypercubes and -LRB- n -RRB- for - ary 2 and 3-cubes . This optimization does not come for free , however . In some multicomputers , there might not be any free bits in message headers . This combined with the fact that every receive operation must check this extra bit means that the regular processing of messages will be slowed down even in the absence of checkpointing . Thus , the degree of optimization -LRB- or degradation -RRB- depends on how much the multicomputer is slowed down by this extra bit , as well as on the frequency of checkpointing and the behavior of the program being checkpointed . The performance of this algorithm should be sufficient to provide fault-tolerance for long-running programs without incurring a significant overhead . However , the latency of taking local checkpoints may be too large for implementing fine-grain time-sharing . This latency can be reduced by using a multicomputer 's address translation hardware to overlap the most time-consuming operations of the checkpoint with the running of the target program -LSB- 1 -RSB- . Such a method allows a processor to save its checkpointed address space to disk concurrently with the target program and with the sending of marker messages . If checkpoints are taken frequently , this latency can be further reduced by saving only the pages that have been changed since the last checkpoint -LSB- 10 -RSB- . Similar techniques can be used to decrease the latency of recovery . Conclusion We have designed a new distributed checkpointing algorithm , specialized for the demands and opportunities that arise in checkpointing programs on multicomputers with wormhole routing interconnection networks . The checkpointing algorithm is unique in the distributed checkpointing literature in that it is highly efficient both when taking checkpoints and when recovering from checkpointed images . The algorithm achieves its high performance by making use of assumptions not satisfied in general by distributed systems , but satisfied by the routing hardware in most of the existing and proposed multicomputers of which we are aware . We are currently implementing the algorithm on the Intel iPSC/2 hypercube . ai Li was supported in part by S grant CCR - 0208 3 , by ARPA and O R under contracts 00014 - 1 - -403 , and by Intel Supercomputer Systems ivision . effrey aughton was supported in part by S grant IRI-8 7 5 and by S Presidential oung Investigator Award . ames Plank was supported in part by AT T fellowship . The authors would like to thank ack ongarra for his help and support . References -LSB- 1 -RSB- ohan Ahuja . Repeated global snapshots in asynchronous distributed systems . Technical Report OS - CISRC-8 / 8 TR40 , Ohio State niversity , August 1 8 . -LSB- 2 -RSB- William C. Athas and Charles L. Seitz . ulticomputers : uessage-passing concurrent computers . Co puter , 21 -LRB- 8 -RRB- : 24 , August 1 88 . -LSB- 3 -RSB- A. Borg , . Baumbach , and S. lazer . A message system supporting fault tolerance . In rocee in s o t e C posiu on peratin ste rinciples , pages , Atlanta , eorgia , October 1 83 . -LSB- 4 -RSB- . ani Chandy and Leslie Lamport . istributed snapshots : etermining global states of distributed systems . C ransactions on Co puter ste s , 3 -LRB- 1 -RRB- :3 75 , ebruary 1 85 . -LSB- 5 -RSB- Carol Critchlow and im Taylor . The inhibition spectrum and the achievement of causal consistency . Technical Report TR 0-1101 , Cornell niversity , ebruary 1 0 . -LSB- -RSB- William . ally . Performance analysis of - ary n-cube interconnection networks . ransactions on Co puters , 3 -LRB- -RRB- :775 785 , une 1 0 . -LSB- 7 -RSB- William . ally . xpress cubes : Improving the performance of - ary n-cube interconnection networks . ransactions on Co puters , 40 -LRB- -RRB- :101 1023 , September 1 1 . -LSB- 8 -RSB- William . ally . ine-grain concurrent computing . esearc irections in Co puter cience n erspecti e , pages 127 154 , 1 1 . -LSB- -RSB- William . ally and Charles L. Seitz . eadlock-free message routing in multiprocessor interconnection networks . ransactions on Co puters , C-3 -LRB- 5 -RRB- :547 553 , ay 1 87 . -LSB- 10 -RSB- S. eldman and C. Brown . Igor : A system for program debugging via reversible execution . C otices or s op on arallel an istri ute e u in , 24 -LRB- 1 -RRB- :112 123 , an 1 8 . -LSB- 11 -RSB- im . ray et . al. . The recovery manager of the System R database manager . C Co putin ur e s , 13 -LRB- 2 -RRB- :223 242 , une 1 81 . -LSB- 12 -RSB- avid B. ohnson and Willy waenepoel . Recovery in distributed systems using optimistic message logging and checkpointing . Co puter ste s esearc at ice ni ersit nnual eport - , pages 83 102 , 1 8 . -LSB- 13 -RSB- Richard oo and Sam Toueg . Checkpointing and rollback-recovery for distributed systems . ransactions on o tware n ineerin , S -13 -LRB- 1 -RRB- :23 31 , anuary 1 87 . -LSB- 14 -RSB- Ten H. Lai and Tao H. ang . On distributed snapshots . n or ation rocessin etters , 25:153 158 , ay 1 87 . -LSB- 15 -RSB- aniel Lenoski , ames Laudon , ourosh harachorloo , Anoop upta , and ohn Hennessy . The directory-based cache coherence protocol for the dash multiprocessor . In e t nnual nternational posiu on Co puter rc itecture , pages 148 15 , Seattle , Washington , ay 1 0 . -LSB- 1 -RSB- ai Li , effrey . aughton , and ames S. Plank . Real-time , concurrent checkpoint for parallel programs . In econ C posiu on rinciples an ractice o arallel ro ra in , pages 7 88 , Seattle , Washington , ar 1 0 . -LSB- 17 -RSB- Barbara H. Liskov and Robert W. Schei er . uardians and actions : Linguistic support for robust , distributed programs . C ransactions on ro ra in an ua es an ste s , 5 -LRB- 3 -RRB- :381 404 , july 1 83 . -LSB- 18 -RSB- C. ohan and B. Lindsay . fficient commit protocols for the tree of processes model of distributed transactions . In rocee in s o t e C posiu o istri ute Co putin , pages 7 80 , ontreal , august 1 83 . -LSB- 1 -RSB- Steven . ugent . The iPSC/2 direct-connect communications technology . In rocee in s o t e ir percu e Con erence . AC , ovember 1 88 . -LSB- 20 -RSB- ichael L. Powell and avid L. Presotto . Publishing : A reliable broadcast communication mechanism . In rocee in s o t e C posiu on peratin ste rinciples , pages 100 10 , october 1 83 . -LSB- 21 -RSB- C. L. Seitz et al. . The architecture and programming of the Ametek series 2010 multicomputer . In rocee in s o t e ir percu e Con erence , pages 33 37 . AC , anuary 1 88 . -LSB- 22 -RSB- C. L. Seitz et al. . Submicron systems architecture project semiannual technical report . Technical Report TR-88-18 , Caltech Computer Science Tech . Rep. , ovember 1 88 . -LSB- 23 -RSB- adalene Spezialetti and Phil earns . fficient distributed snapshots . In rocee in s o e i t nternational Con erence on istri ute Co putin ste s , pages 382 388 , Cambridge , assachusetts , ay 1 8 . I Computer Society . -LSB- 24 -RSB- Robert . Strom and Shaula emini . Optimistic recovery in distributed systems . C ransactions on Co puter ste s , pages 204 22 , August 1 85 . -LSB- 25 -RSB- S. enkatesan . essage-optimal incremental snapshots . In rocee in s o e int nternational Con erence on istri ute Co putin ste s , pages 53 0 , ewport Beach , California , une 1 8 . I Computer Society .

O O B O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O B O O O O O B O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O B O O O O B O O O B O O O O O O O O O B O O O B O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O B O B I O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

checkpointing
multicomputers
wormhole routing