The Maintenance of Common Data in a Distributed System ? Baruch Awerbuch y M.I.T. Leonard J. Schulman z M.I.T. Abstract A basic task in distributed computation is the maintenance at each processor of the network , of a current and accurate copy of a common database . A primary example is the maintenance , for routing and other purposes , of a record of the current topology of the system . Such a database must be updated in the wake of locally generated changes to its contents . Due to previous disconnections of parts of the network , a maintenance protocol may need to update processors holding widely varying versions of the database . We provide a deterministic protocol for this problem , with only polylogarithmic overhead in both time and communication complexities . Previous deterministic solutions required polynomial overhead in at least one of these measures . ? Presented at the 32nd Annual Symposium on Foundations of Computer Science . Computing Reviews Category C. 2 -LRB- Computer-Communication Networks -RRB- . yDept . of Mathematics and Lab . for Computer Science , MIT , Cambridge , MA 02139 . Supported by Air Force Contract TNDGAFOSR-86-0078 , ARO contract DAAL03-86-K-0171 , NSF contract CCR8611442 , and a special grant from IBM . zDept . of Mathematics and Lab . for Computer Science , MIT . Supported by an ONR Graduate Fellowship . Current address : College of Computing , Georgia Institute of Technology , Atlanta GA 30332-0280 . 1 Introduction 1.1 Motivation Many tasks in distributed computing deal with concurrently maintaining the \ view '' of a common object in the separate sites of a distributed system . This object may be the topology of a communication network -LRB- in which case the view is a description of the underlying network graph -RRB- , or certain resources held at the system sites -LRB- in which case the view is an inventory listing the resources held at each site -RRB- , or a general database . These objects are subject to occasional changes -LRB- e.g. , a link fails , a resource unit is consumed or released , a database record is modified -RRB- . It is thus necessary to have an efficient mechanism for maintaining consistent and updated views of the object at the different sites . The most well known example of a task falling into this category is that of Topology Update , whose essence is updating at each node a local view of network topology , in response to topological changes -LRB- failures and recoveries of network links -RRB- that occur in the network . A Topology Update protocol must communicate such changes -LRB- which are initially known only locally -RRB- throughout the asynchronous network -LRB- and in spite of possible temporary network disconnections -RRB- . Topology Update protocols constitute a key component of the major commercial and governmental networks such as the ARPANET -LSB- MRR80 -RSB- , DECNET -LSB- Wec80 -RSB- , and SNA -LSB- BGJ +85 -RSB- , as well as in new high-speed networks such as IBM 's PARIS -LSB- CG88 , ACG +90 -RSB- and NSF/DARPA 's AURORA gigabit-rate testbed . The update protocols are used to handle management tasks such as routing and bandwidth reservations . As further motivation for the study of the Topology Update problem , it was shown in -LSB- ACK90 -RSB- that it is in fact the hardest instance of a database update problem . Note also that topology-dependent tasks such as Shortest Paths and Minimum Spanning Tree can be solved locally in a network in which current replicas of the topology are maintained at each node . -LRB- Of course individual problems may have more efficient problem-specific protocols -RRB- . Due to disconnections of the network , it may be impossible to maintain identical views of the common database at all the nodes , and some nodes or subnetworks may have outdated records . However , once connectivity is restored , say , by an edge recovery that creates a \ bridge '' connecting two previously disconnected subnetworks , the discrepancies must be eliminated . The endpoints of the \ bridge edge '' can transmit their views to each other , and consequently agree on the common version of the object . Next , each endpoint of the bridge can transmit this common version to all the nodes of its subnetwork . One naive way of achieving this transmission is simply to broadcast the information to all the nodes . This method , referred to as Full Broadcast , can be very wasteful in communication since it fails to take advantage of prior partial knowledge available in the system . For , processors need be informed of relatively few changes if they already hold a nearly correct view of the database . The Incremental Update strategy in -LSB- ACK90 , ACG +90 -RSB- exploits fully the prior knowledge available in the system , sending each processor only one message per error . While Incremental Update is clearly superior to Full Broadcast in terms of communication complexity , it can be significantly inferior in terms of time complexity . This is due to the fact that the Full Broadcast method takes advantage of message pipelining , while the Incremental Update method does not -LSB- ACK +91 -RSB- . The Broadcast with Partial Knowledge problem , formulated in -LSB- ACK +91 -RSB- , is that of updating the views of all the processors , while taking advantage of prior knowledge and of the possibility of pipelining | in order that both the message and time complexities of the protocol be minimized . A formal statement of the problem will be given below . A key result of -LSB- ACK90 -RSB- is that the Topology Update problem effectively reduces to the Broadcast with Partial Knowledge problem . More precisely , any protocol for Broadcast with Partial Knowledge yields a Topology Update protocol with no greater overheads in communication and time . 1.2 The Problem In the Broadcast with Partial Knowledge problem we consider an asynchronous communication network consisting of n + 1 processors , each holding an m-bit local input . One processor is distinguished as the source . The problem requires all the processors to write in their local output the value of the input at the source . Note that we allow the inputs of the various processors to differ in arbitrary ways . We make the neighbor-knowledge assumption : namely , that each processor knows the inputs of each of its network neighbors . This assumption is justified in the context of Topology Update because the cost of exchanging information among the two processors on a new \ bridge edge '' can be charged to the topology change , without effect on the asymptotic communication or time complexities for Topology Update . For detail see -LSB- ACK90 -RSB- . We need consider only the case in which the topology of the network is a tree , with the source node being its root . This fits into the framework of -LSB- ACK90 , ACG +90 -RSB- in which topology changes are broadcast over a spanning tree . Furthermore it will be sufficient to solve the problem on a chain , with the source at one of its ends . For , the broadcast problem on a spanning tree can be reduced to that on a chain by first computing a depth-first tour of the tree , and then running a chain algorithm on that depth-first tour ; the length of the chain is then at most twice the number of processors . -LRB- This implementation does not affect the stabilization time of the algrorithms , which will remain linear . -RRB- For simplicity we will assume -LRB- although this assumption can be dispensed with -RRB- that each processor knows its own position in the chain . This can be achieved by broadcasting a message from the source of the tree , with minimal overhead . Note that we assume the network is static . Entry or departure of links or nodes must be handled by an underlying topology protocol -LRB- as in -LSB- ACK90 -RSB- and -LSB- ACK +91 -RSB- -RRB- , and our protocol may need to be restarted on a new tree of processors . We will picture the global input configuration as a two-dimensional array of bits , with the source at the left , machine index increasing to the right , and each column vector representing a local input of a particular processor -LRB- see figure 1 -RRB- . In section 3 this picture will be the setting for a geometric analysis of the time complexity of our protocol . LocalData 0 1 0 00 0 1 0 02 1 0 0 13 0 1 1 141 1 1 0 1Machines Figure 1 : Example illustrating the Broadcast with Partial Knowledge problem . In this example , we have n +1 = 5 processors maintaining a joint view with m = 4 items . The source is processor . The incorrect entries of each processor are underlined . The local discrepancy is the number of underlined entries at a processor . The total discrepancy is ? = 7 . The input stored at each processor is the local representation of the database at this processor . The correct description of the database is held by the source ; the local descriptions may differ from the correct one , and from each other , as a result of changes in the database and in the network topology . Our goal is to spread the source 's view of the database throughout the network . 1.3 Complexity The communication complexity of a protocol is defined as the number of one-bit messages which it transmits . We use the standard notion of time for asynchronous systems , in which each link can at any time carry a single one-bit message , and convey it from one end of the link to the other in at most one unit of real time . In order to quantify the possibility of exploiting local knowledge , we use a measure that captures the level of \ correctness '' of the knowledge held by each processor -LSB- ACK +91 -RSB- . Let the local discrepancy of a given processor be the number of bits in which the local input at this processor differs from the source 's input . Define the total discrepancy ? -LRB- which is not assumed to be known to the processors -RRB- as the sum of the local discrepancies . The communication and time complexities of solutions will be expressed as functions of m , n and ? . 1.4 Previous Solutions We have already mentioned the Full Broadcast solution to the Broadcast with Partial Knowledge problem , which is wasteful in communication since it requires O -LRB- nm -RRB- messages without regard to ? ; but which , implemented with pipelining , terminates in the optimal O -LRB- n + m -RRB- time . In the Incremental Update -LSB- ACK90 -RSB- solution , machines are brought one at a time into complete agreement with the source 's database . Neighbor-knowledge is assumed and therefore a machine holding the correct database can correct the errors in its neighbor 's database with just one message per error . Thus the communication complexity of this solution , O -LRB- n + ? log m -RRB- , is optimal . However since the corrections are made sequentially , the time complexity is also O -LRB- n + ? log m -RRB- , which can be as bad as O -LRB- mn log m -RRB- . Observe that although some \ opportunistic '' parallelism may be possible in a particular run of this protocol -LRB- once a processor has received a correction message for any bit beyond position j , it can correct bit j at its successor if necessary -RRB- , there are many examples where little or no such parallelism occurs -LRB- e.g. errors only at every other processor -RRB- . Since the network is asynchronous , there is no way for a processor to determine that a bit is correct on the basis , say , that a sufficient amount of time has elapsed for receipt of a correction message . A nearly optimal randomized solution was given in -LSB- ACK +91 -RSB- . That algorithm does not assume \ neighbor knowledge '' , uses O -LRB- ? log m + n log m '' -RRB- communication and O -LRB- n log -LRB- 2 + ? n -RRB- + m + log 1 '' -RRB- time , and achieves error probability '' . By comparison , in the absence of neighbor knowledge there is an ? -LRB- n + m -RRB- time lower bound -LRB- even for randomized algorithms -RRB- ; and -LSB- ACK +91 -RSB- provides an ? -LRB- n + ? log mn ? -RRB- communication lower bound . 1.5 Our Solution We provide a deterministic solution to the Broadcast with Partial Knowledge problem which uses O -LRB- -LRB- n + ? -RRB- log m -RRB- communication and O -LRB- -LRB- n + minfm ; ? g -RRB- log3 m -RRB- time . The corresponding lower bounds -LRB- assuming neighbor knowledge , and that n > = 2 -RRB- are ? -LRB- n + ? -RRB- communication and ? -LRB- n + minfm ; ? g -RRB- time . The communication lower bound is obtained by placing the identical ? = n errors at every processor but the source -LRB- processor 0 -RRB- , and observing that every processor but the neighbor of the source -LRB- processor 1 -RRB- must learn the locations of these errors . -LRB- A standard counting argument provides the stated lower bound . -RRB- The time lower bound is obtained by placing the identical minfm ; ? g = 2 errors at processors 1 and 2 , and observing that the locations of the errors must be conveyed across the link between processors 1 and 2 . -LRB- The same counting argument is again employed . -RRB- It follows that the multiplicative overheads of our solution in communication and time are both polylogarithmic , in comparison with previous deterministic solutions for which at least one of these overheads was polynomial . See figure 2 . Reference Commun . Time Full Broadcast nm n + m Incr . Upd . -LSB- ACK90 -RSB- -LRB- n + ? -RRB- log m -LRB- n + ? -RRB- log m This paper -LRB- n + ? -RRB- log m -LRB- n + minfm ; ? g -RRB- log3 m Lower Bd . n + ? n + minfm ; ? g Figure 2 : Complexity of deterministic protocols -LRB- up to constant factors -RRB- . The intuition behind our solution is as follows . We place one \ process '' , a program that runs on the host processors , in charge of the entire protocol | that is , in charge of \ cleaning '' the entire array of bits . Then , in order to strive for good time complexity , we adopt a very weak version of pipelining , by allowing this process to work recursively and create \ child '' processes . Each of the child processes is in charge of cleaning a smaller subarray , and they are staggered relative to one another in their work . In order to clean their subarrays the children run the same protocol as the main process -LRB- with smaller parameters -RRB- . In particular , they can create children which are in charge of subarrays of their subarray , and so forth , down to descendents which are in charge of constant-size subarrays . Now , if each process were to immediately delegate all of its work to its children , the message complexity of the protocol would be very poor . We avoid this problem by allowing our processes to be lazy . For a while they will try , themselves , to clean the subarray that they are in charge of ; only if this starts to take a long time , will they create children , and delegate the work . In this manner the communication costs associated with the creation of children , will not actually be incurred by a process unless enough errors are available to pay for these costs . The time analysis is more involved . The time complexity of a protocol is proportional to the length of a longest chain of causally dependent messages in it -LRB- times the complexity of sending one message , which in our case will be O -LRB- logm -RRB- -RRB- . We will view the history of our protocol geometrically , laying out the processors as one axis , and the data bits as another axis , of a grid in the plane ; every message sent in our protocol will be at some location in this grid . Roughly speaking , every causally dependent chain of messages will induce a path in this grid ; it is the length of such paths that we will bound . In order to prevent these paths from being contorted and long , we would like the protocol to impose conditions that make them as monotone as possible : for , the length of a monotone path -LRB- in which each coordinate is nondecreasing when the other is increased -RRB- is bounded by the sum of the dimensions of the grid , or m + n . The causal paths in our protocol will not actually be monotone , but in a sense , they will be close to monotone : the protocol has a recursive structure , and each \ backtracking '' of the path will be amortized against a level of the recursion . It will be shown that the effect of these meanderings is only a multiplicative log2 m factor on the maximal length of a causal path . -LRB- Another view of this analysis is that it is measuring the \ fractal dimension '' of a causal path ; the length of the path will be shown to be -LRB- m + n -RRB- -LRB- 1 + o -LRB- 1 -RRB- -RRB- by demonstrating that its dimension is 1 . -RRB- We stress that , analysis aside , the protocol is simple and easily implemented . 2 Protocol 2.1 Preliminaries For simplicity we will assume that m is a power of 2 -LRB- otherwise just \ pad '' -RRB- . In the exposition we will also suppose that n is a multiple of m , and in this case we will label the machines M0 -LRB- the source -RRB- through Mn . However the protocol as described below will run correctly and within the stated time and communication bounds even if n is not a multiple of m , with the machines labelled Ma -LRB- the source -RRB- through Mn + a , for a = m -LRB- dn = me ? n = m -RRB- . Although communications are allowed in both directions in our model , our protocol will in fact use messages only in the direction away from the source -LRB- always to the right in our diagrams -RRB- . We will refer to the right-hand neighbor of a machine as its successor . The incorrect bits at each machine will be corrected one-by-one in order of increasing bit index , by messages from the machine 's left-hand neighbor . No machine will issue such a \ correction '' message for bit j until it knows that all its own bits 1 ; : : : ; j are correct ; thus the \ clean '' area of certified bits will have the monotone appearance of figure 3 . MachinesBitsClean Bits Figure 3 : A snapshot of the operation of the algorithm . Our algorithm will be administered by the processes . These are programs which progress from machine to machine , and \ run '' on their current host machine . Each process Q is responsible for cleaning some rectangle in the diagram , which we will refer to as the Q-rectangle : for example , the entire protocol will be started and maintained by the main process , which is responsible for cleaning the whole table . Each process Q will have two fixed parameters -LRB- see figure 4 -RRB- : -LRB- a -RRB- Its altitude yQ , and -LRB- b -RRB- Its height mQ . The rectangle that it is responsible for consists of the bits yQ ? mQ +1 through yQ , on the machines starting from Q 's host 's successor up through the next machine whose index is divisible by mQ . Thus the width of a Q-rectangle -LRB- number of processors -RRB- is at most equal to its height -LRB- number of bits -RRB- . In our diagrams a vertical line through a machine whose index is divisible by m will be called an m-column . Q-rectangleAltitudeHeightm - columnQm - columnQyQmQ Figure 4 : The rectangle of a process Q. 2.2 Outline A key observation to achieving a balance between the two extreme strategies is this : suppose a process starts correcting the errors in its Q-rectangle itself , thus -LRB- as in the incremental up - date algorithm -RRB- taking no risks with the message cost . If it progresses quickly | all is well . Conversely : slow progress is a sign that it is encountering many errors | enough , so that it can \ pipeline '' part of its task : it can generate a pair of child processes , delegate to each the cleanup of a subrectangle of Q , and pay for the messages which are required for them to travel from host to host . In this case , essentially by induction , the process 's quick progress will be guaranteed , until the children terminate . By the time it terminates , a process Q will have created either 0 , 2 or 4 children -LRB- see figure 5 -RRB- : either none of U ; L ; U ; L0 ; or one of the pairs U ; L and U ; L0 ; or all four of U ; L ; U ; L0 . The conditions for these events will be explained below . m / 2-columnUU 'LL 'm - columnQm - columnQQmQ Figure 5 : The rectangles of the children of a process Q . In addition to its height and altitude , which are fixed from its creation , a process also maintains an \ error counter '' variable ? -LRB- in the range ; : : : ; mQ -RRB- with which it counts the errors that it itself corrects . Every message we use will be labelled with an index between 1 and m , corresponding to its \ altitude '' in the diagrams . The messages will satisfy a \ local monotonicity '' condition : The messages sent by any machine to its successor during the protocol occur in order of nondecreasing altitude . We will use two kinds of messages in the algorithm : corrections , and process messages . In a correction message , a machine tells its successor to \ flip bit j '' , for some 1 < = j < = m. -LRB- Recall that thanks to the neighbor-knowledge assumption , a machine knows the current value of its neighbor 's j' th bit . -RRB- There will be exactly ? such messages during the algorithm . The \ altitude '' of such a message is j. Note that such a message uses O -LRB- log m -RRB- bits . The other class of messages , the process messages , are simply the way in which a process moves from its current host machine , to that machine 's successor . Such a message is of the form -LRB- y ; h ; ? ; b ; c -RRB- , representing the altitude y , height h , and error counter ? of the process ; a bit b representing which of two modes the process is in ; and a bit c indicating whether this is the main process -LRB- that is in charge of the entire chain -RRB- . Each of the first three variables is bounded by m so the process message uses O -LRB- log m -RRB- bits . The altitude of a process message is defined to be equal to the altitude y of the process it represents ; so these two uses of the term altitude , agree . At any moment , a process will be in one of two modes : open mode or split mode . In open mode , the process issues corrections itself ; in split mode , it delegates the cleanup to its children and tags along behind them . -LRB- See figure 6 -RRB- . The first thing a process does when it is created is to start an open mode . The entire protocol begins when the main process , with height m and altitude m , starts an open mode at the source machine M0 . UU 'LL ' Figure 6 : The operation of a process -LRB- correction messages marked by arrows -RRB- . 2.3 Open Mode When a process Q starts an open mode , it initializes its \ error counter '' ? to 0 , and then starts correcting every error it sees in its host 's successor , in order of increasing altitude . Once the successor 's vector is clean throughout the segment that the process is responsible for , the process moves on to the successor , and continues operating in open mode there . With each error that it corrects , it increments ? . The process -LRB- of height mQ -RRB- continues in this manner until either : it reaches an mQ-column , and terminates -LRB- unless it is the main process -RRB- ; or it reaches an mQ = 2-column which is not an mQ-column , in which case it resets ? to 0 , effectively beginning a new open mode ; or ? = mQ , in which case it immediately starts a split mode -LRB- to continue until the nearest mQ = 2-column -RRB- . The only case not covered above is that of the main process reaching an mQ-column . In this case it is to reset ? to and start a new open mode , unless it has reached the last machine in the chain in which case it terminates . 2.4 Split Mode In split mode , the process itself does not correct any errors . Rather , upon entering split mode , the process creates two child processes , each responsible , in each machine they encounter until the end of the split mode -LRB- where they terminate -RRB- , for cleaning half of the sector of bits that their parent process is responsible for . After creating the child processes , the parent process just tags along behind them until they terminate . If the parent is of height mQ and altitude yQ , then the height of each child is mQ = 2 , the altitude of the upper one is yQ , and the altitude of the lower one is yQ ? mQ = 2 . Both children terminate at the next mQ = 2-column . Thus each is responsible for a rectangle of width at most mQ = 2 . In accordance with the local monotonicity condition -LRB- section 2.2 -RRB- on the order of messages , all operations of the upper child will be staggered behind the lower child . All that remains to be specified is that the parent process itself tags along just behind the upper child . -LRB- Note that they are at the same altitude , so their order is not dictated by local monotonicity . It is convenient to think of a parent as having \ slightly '' higher altitude than its upper child . -RRB- Observe that when the parent proceeds from Mx to Mx +1 , the data at Mx +1 is already clean up to altitude yQ . When the child processes terminate at an mQ = 2-column then , if this is an mQ-column , the parent process terminates as well -LRB- unless it is the main process -RRB- ; while if it is not an mQ-column , the parent process resets ? to and starts a new open mode . If the parent process is the main process then , even upon reaching an mQ-column , it resets ? to and starts a new open mode , unless it has reached the last machine in the chain in which case it terminates . 2.5 Base Case A process with m = 1 is always in open mode . If this process is not the main process , then it is in charge of cleaning just one bit j on one machine Ma -LRB- since the width of a Q-rectangle is bounded by its height -RRB- . The correction -LRB- if necessary -RRB- and process message are issued by Ma ? 1 , and the process terminates at Ma . If the main process has m = 1 , i.e. the database consists of just one bit , then it simply traverses the chain , always issuing corrections as it goes . 3 Analysis It will be simplest to make both the message analysis and the time analysis in the \ word model '' , where each message consists of O -LRB- log m -RRB- bits , and requires up to a unit of time . The following theorem summarizes the analysis : Theorem 3.1 In the word model our algorithm has message complexity O -LRB- n + ? -RRB- and time complexity O -LRB- -LRB- n + minfm ; ? g -RRB- log2 m -RRB- . Our main result is obtained by straightforward conversion to the bit model : Theorem 3.2 In the bit model our algorithm has message complexity O -LRB- -LRB- n + ? -RRB- log m -RRB- and time complexity O -LRB- -LRB- n + minfm ; ? g -RRB- log3 m -RRB- . 3.1 Message Analysis First consider the correction messages : there is exactly one of these per error , for a total of ? . Second , consider the process messages . The main process travels across the entire chain , thus costing n messages . Each other process , of height m0 < = m , uses at most m0 process messages , and is created as either the upper or lower child of a process -LRB- say Q -RRB- of height 2m0 , after Q encountered 2m0 errors in open mode . We use these errors to pay for both of Q 's children . The cost per error is thus 1 . The total number of messages is therefore at most 2 ? + n. 3.2 Time Analysis The standard model of time complexity in an asynchronous setting involves the following assumptions : -LRB- a -RRB- We ignore the computation time at each node . -LRB- b -RRB- We assume that each message traverses its link in at most one unit of real time . Definition 3.3 The quiescence time T ? of a protocol is defined as the maximum over all combinations of link traversal times , of the real time required for the protocol to terminate . We will bound the quiescence time of our protocol with another quantity , T , that we shortly define . First however note that if ? < m , the main process will never enter split mode , and the total number of messages transmitted will be n + ? . Hence -LRB- in the word model -RRB- the time complexity will be O -LRB- n + ? -RRB- . We will therefore focus on obtaining an O -LRB- -LRB- n + m -RRB- log2 m -RRB- bound for arbitrary ? . The Dependency Graph First we describe the dependency graph D among messages in a protocol . This is a directed acyclic graph , whose vertices are the messages sent from the various machines to their successors ; and in which arc -LRB- v , w -RRB- indicates that the transmission of message w is contingent upon that of message v . There are two types of dependencies in D -LRB- here by a link we mean the directed communication link between two processors -RRB- : 1 . Message w on link Mb ! Mc can only be sent after message v has been sent on Mb ! Mc . -LRB- \ Link-internal '' dependence . -RRB- 2 . Message w on link Mb ! Mc can only be sent after message v on link Ma ! Mb has been received . -LRB- \ Between-links '' dependence . -RRB- These categories can arise in any distributed protocol ; in ours they occur in the following ways -LRB- see figure 7 for some examples -RRB- : 1 . Link-internal . Here v and w are both messages from Mx to Mx +1 , and v has higher priority than w . The ways in which this can arise are : -LRB- a -RRB- w is a process message for a process Q in open mode : v is either the highest-altitude correction message issued by Q at Mx , or , if there are no such corrections , then the highest-altitude process message at Mx , whose altitude is less than that of w -LRB- if such exists -RRB- . -LRB- b -RRB- w is a process message for a process in split mode : v is the process message for its upper child . -LRB- c -RRB- w is a correction message for a process Q in open mode : v is either the next-highestaltitude correction message issued by Q -LRB- or its parent -RRB- at Mx , or , if there are no such corrections , then the process message for the process that is of highest altitude among those beneath Q at Mx -LRB- if such exists -RRB- . 2 . Between-links . Here v is a message from Mx ? 1 to Mx , and w is from Mx to Mx +1 . The ways in which this can arise are : -LRB- a -RRB- w is the very first process message for a new process Q : v is the process message carrying Q 's parent to Mx . -LRB- b -RRB- w is the process message for a continuing process Q : v is the process message which brought Q to Mx . -LRB- c -RRB- w is a correction message issued by process Q : v is the process message which carried Q to Mx . Link-internal arcs always point straight up in our diagrams , while between-links arcs point either across , or down and across . Note that every message has at most one dependence of each of these categories , i.e. every vertex of D has at most one incoming edge of each kind . Properties of the Dependency Graph Let T be the length of a longest path in D. Lemma 3.4 The quiescence time T ? of our algorithm is less than or equal to T . QUCorr.LQULQCorr .1 -LRB- b -RRB- 1 -LRB- a -RRB- 1 -LRB- c -RRB- 2 -LRB- b -RRB- 2 -LRB- a -RRB- 2 -LRB- c -RRB- 2 -LRB- a -RRB- Figure 7 : \ Link-internal '' and \ Between-links '' dependencies among messages . -LRB- Correction messages , and process messages for Q and its children U and L -RRB- . Proof : Note that a processor can indeed transmit a message as soon as that message 's predecessors in D have been transmitted . 2 This allows us to discuss our time bound entirely in terms of path lengths in D . We also point out that in fact there is no loss suffered in the bounding of T ? by T -LRB- although strictly speaking this does not matter for our result -RRB- : Lemma 3.5 In our algorithm , T ? is greater than or equal to T . Proof : Our algorithm has the property that the sequence of messages received and transmitted at each machine does not depend on the link traversal times . 2 If Q is a process , denote the process message which takes it from machine Mx to Mx +1 by Q -LRB- x -RRB- . Definition 3.6 The time-line of Q is the horizontal line at altitude y -LRB- Q -RRB- -LRB- the altitude of process Q -RRB- ; the time-line starts at the processor where Q is created and ends at the processor where Q terminates . If a path in D passes through process messages of Q , and also through messages of of both higher and lower altitude than Q , on machines between where Q starts and terminates , then we say that the path crosses the time-line of Q . If some messages of altitude less than altitude -LRB- Q -RRB- precede some messages of altitude more than altitude -LRB- Q -RRB- on the path , then we say that the path crosses the time-line going up ; and if some messages of altitude more than altitude -LRB- Q -RRB- precede some messages of altitude less than altitude -LRB- Q -RRB- , then we say that the path crosses the time-line going down . Lemma 3.7 A path through D can cross the time-line of any process at most once , and that is while the path is going up . Proof : Observe the following fact regarding D. Let Q be a process which splits at some machine Mx , creating a lower child L and an upper child U . Then the between-links arc from Q -LRB- i.e. from the process message Q -LRB- x ? 1 -RRB- -RRB- extends to the first message generated by L , which will be either the process message L -LRB- x -RRB- , or the first correction in the L-rectangle . Thus the between-links arc into L -LRB- x -RRB- comes from Q -LRB- x ? 1 -RRB- . Next observe that for x0 > x , the betweenlinks arc into L -LRB- x0 -RRB- comes from L -LRB- x0 ? 1 -RRB- . These two facts , along with our earlier note that there is only one between-links arc into any message , lead to the following -LRB- informal -RRB- conclusion : No event in the U - rectangle has any bearing on events in the L-rectangle . Let us say that a message is in the Q-rectangle if it is generated by Q or by a process that is a descendent of Q. -LRB- Thus , its altitude will be between yQ ? mQ + 1 and yQ , where yQ and mQ are the altitude and height of Q -RRB- . Then , more formally : No message in the L-rectangle is a descendent of a message in the U - rectangle . Pictorially this can be viewed as saying that no path crosses the time-line of a process , going down . The lemma follows . 2 On the other hand note that events in the upper child U are dependent on those in the lower child L . In particular , no message in the U-rectangle can depart a machine Mx0 until after L -LRB- x0 -RRB- does so . Figure 8 : Time-lines of three generations of processes , and a path through D . The path can descend only between time-lines . We can now observe a property of the dependency graph which allows us to make our time analysis separately for the stages during which the main process is in open and in split mode . Observation 3.8 -LRB- See figure 9 -RRB- : 1 . Suppose Q is a process which starts a split mode at machine Mi . Suppose v is a message in the open subrectangle preceding Mi , and w is a message in the following split subrectangle . Then all paths in D from v to w , pass through the message Q -LRB- i ? 1 -RRB- . 2 . Suppose Q is a process which finishes a split mode at machine Mj . Suppose w is a message in the split subrectangle preceding Mj , and z is a message in the following open subrectangle . Then all paths in D from w to z , pass through the message Q -LRB- j ? 1 -RRB- . An examination of the algorithm shows that the main process undergoes d2n = me iterations of the open mode -LCB- split mode sequence , each iteration ending at an m = 2-column . -LRB- Recall that even if no split is created , the error counter is reinitialized at the m = 2-column anyway -RRB- . Thus in view of observation 3.8 , we can multiply a bound on the algorithm in the special case n = m = 2 by the factor d2n = me to obtain a bound for the general case . Therefore theorem 3.1 is implied by the following analysis of the case n = m = 2 . jLUMiMwvzQ -LRB- i-1 -RRB- Q -LRB- j-1 -RRB- Figure 9 : The constraints on a path -LRB- dashed -RRB- between consecutive open and split modes . The path from v to w must pass through Q -LRB- i ? 1 -RRB- , and that from w to z must pass through Q -LRB- j ? 1 -RRB- . Time analysis for the case n = m = 2 Theorem 3.9 Let n = m = 2 . Then T , the length of a longest path in the dependency graph , is O -LRB- m -LRB- log m -RRB- 2 -RRB- . Proof : We are concerned with the dependency graph D among the messages in an -LRB- m = 2 -RRB- ? m rectangle -LRB- i.e. involving machines M0 ; : : : ; Mm = 2 -RRB- . For each m0 a power of 2 , 1 < = m0 < = m , we introduce a partition of the -LRB- m = 2 -RRB- ? m rectangle into -LRB- m0 = 2 -RRB- ? m0 rectangles which we call m0 - rectangles . Each m0-rectangle is of height m0 , and includes the bit indices km0 +1 ; : : : ; -LRB- k + 1 -RRB- m0 -LRB- for some k -RRB- ; it is of width m0 = 2 and extends between adjacent m0 = 2-columns . Observe that the rectangle to be cleaned by any process of height m0 intersects at most two m0-rectangles -LRB- side by side between m0-columns -RRB- . The idea of the analysis will now be to consider any path through D , and to assign the messages in to the m0-rectangles of various sizes , that encounters . Then we will bound the number of messages that might be assigned to any m0-rectangle . In this way the total length of will be related to the number of m0-rectangles of each size that it encounters . We will bound these numbers with a recurrence relation derived from properties that must satisfy . The combination of these elements will give us a bound on the length of . In order to accomplish this we first describe how to assign messages -LRB- vertices of D -RRB- to m0-rectangles . Let w be a message issued by a process Q of height m0 . We assign w to the m0-rectangle that contains it -LRB- which is one of those that the Q-rectangle intersects -RRB- . -LRB- This assignment actually has two cases : w is either a correction message issued by Q in an open mode ; or a process message for Q , in either mode . -RRB- Now , there is at most one process of height m0 intersecting any given m0-rectangle . -LRB- This process is of height m0 ; its altitude is the index of the highest bit in the m0-rectangle ; and it terminates at the nearest m0-column , which is either at the right-hand border of this rectangle , or another m0 = 2 processors to the right . Whether or not the process is run depends of course on the pattern of errors . -RRB- This process will use at most m0 = 2 process messages in the m0-rectangle , and issue at most m0 corrections . Hence an m0-rectangle is assigned at most 3m0 = 2 messages . Now we consider just the vertices of . We will say that encounters an m0-rectangle , if it contains a message w that is assigned to that rectangle by the above rule . Let a ? denote the number of m = 2 ? - rectangles that encounters . Since D is acyclic , no vertex appears more than once in . Therefore , with m0 = m = 2 ? , we find that the number of messages in is at most : lg m X ? = 0 a ? 3 2 m 2 ? -LRB- 1 -RRB- Our definition of \ encountering '' vacuously demonstrates that no edge of is charged to an m0-rectangle that does not encounter . The definition further implies that can not encounter an m0-rectangle without also encountering the enveloping 2m0-rectangle . For , let w be a vertex of in the m0-rectangle . Then above w lies the time-line of the process P of height 2m0 , which is the parent of the process that issued w , and which is currently in split mode . If exits the P - rectangle on a link-internal arc during this split mode , then it passes through a process message for P ; while observation 3.8.2 implies that this is also the case even if exits the split mode on a between-links arc . We now study the numbers fa?g . Proposition 3.10 a ? < = 2 ? -LRB- ? 2 + 1 -RRB- . Proof : The main step in the proof is to establish the following recurrence relation among the a ? . Lemma 3.11 For every ? > = 1 , a ? < = 2a ?? 1 + 2 ?? 1 . Proof : Throughout the proof of this lemma let m0 = m = 2 ?? 1 . Recall that can encounter an m0 = 2-rectangle only if it encounters the enveloping m0 - rectangle . Now consider any m0-rectangle that encounters . We claim that can encounter only three of the four m0 = 2-rectangles it contains : for , to encounter the upper-left one means that the process of height m0 has split before the m0 = 4-column running down the middle of the m0-rectangle . Observe that in this case it continues in split mode until the m0 = 2-column at the righthand end of the m0-rectangle . Since never proceeds to the left , the lower-right m0 = 2 - rectangle can only be encountered after the upper-left one . However as observed earlier , no arcs descend from an upper child 's rectangle into the time-line of a lower one . Hence can not encounter both the upper-left and lower-right subrectangles . Next , examine any of the 2 ?? 1 vertical bands of m0-rectangles , into which the array is partitioned by the m0 = 2-columns . enters this band from the left , and leaves it to the right . It encounters some subset of the m0-rectangles of this band , and we claim that it visits them in order of increasing altitude . That is , if v and w are messages issued by two different processes of height m0 in this band , and v is issued by that of lower altitude , then can not visit w before v. For , let Q be the process which is the first common ancestor of the two processes . Then , since Q itself issues neither v nor w , it must be in split mode at least as early -LRB- i.e. as far to the left -RRB- as the earliest of v and w . But then for to go from w to v , it would have to descend past the time-line of Q 's lower child L : this it can not do . See figure 10 . Finally , we claim that in all but one of the m0-rectangles of the band , can visit only two -LRB- rather than three -RRB- of the subrectangles . For , since never travels leftwards , it has only one edge which crosses the m0 = 4-column running down the middle of this band . If this edge occurs within some m0-rectangle , then can encounter three of its subrectangles ; but in lower m0-rectangles can encounter only the lefthand subrectangles , and in higher m0-rectangles it can encounter only the righthand subrectangles . See figure 11 . Therefore if encountered s m0-rectangles in this band , it can encounter within it at most 2s + 1 m0 = 2-rectangles . The lemma follows by summing over all 2 ?? 1 bands . 2 Now we rewrite the recurrence we obtained in the lemma , in matrix form : ? a ? 2 ? ? < = = ? 2 1 2 ?? a ?? 1 2 ?? 1 ? timeline of Ltimeline of Qm ' / 2-columnvwtimeline of U Figure 10 : The m0-rectangles in a band are visited in ascending order . Both eigenvalues of the transition matrix are 2 , therefore the upper bound for a ? obtained by saturating the inequality in this recurrence , has a closed form expression as 2 ? times a polynomial in ? . The solution , taking into account the boundary condition a0 = 1 , is 2 ? -LRB- ? 2 +1 -RRB- . 2 We now derive theorem 3.9 from the proposition : T < = lg m X ? = 0 a ? 3 2 m 2 ? < = 3 2m lg m X -LRB- ? 2 + 1 -RRB- = 3m -LRB- 1 + lg m -RRB- -LRB- 4 + lg m -RRB- = 8 : 2 As mentioned in the introduction , there is a geometric sense to the above proof . A causal path can be specified by indicating first what m-rectangles it encounters ; then what m = 2 - rectangles it encounters ; then what m = 4-rectangles it encounters ; and so forth . Imagining that this process could go on indefinitely -LRB- which has no interpretation in terms of our protocol as it requires m to be infinite -RRB- , the quantity dim -LRB- -RRB- = lim ?! 1 lg a ? ? , measuring the rate of growth of the a ? , is a fractal measure of the dimension of the path , known as its Pontrjagin-Schnirelman dimension -LSB- KT59 , Man82 , PS32 -RSB- . In our finite scenario , let us set d = sup ? lg a ? ? by analogy . The bound -LRB- 1 -RRB- on the length of is now at most 32 lg m sup ? a ? m2 ? < = 32 lg m sup ? 2 ? d m2 ? which , for d > = 1 -LRB- the only case of interest -RRB- is dominated by the term ? = lg m ; and so the length of is bounded by 32 md lg m . This is why showing that our protocol terminates in time m1 + o -LRB- 1 -RRB- , essentially amounts to showing that the dimension of a causal path is 1 . Moreover the demonstration that the runtime is O -LRB- m lg2 m -RRB- -LRB- in the word model , which is more natural for the present discussion -RRB- , amounts to a bound on the rate of convergence of lg a ? ? to the dimension . Acknowledgments We thank Rainer Gawlick and Stephen Ponzio for helpful comments , and the anonymous referees for careful reviews which substantially improved the paper . m ' / 4columnm ' / 2columnm ' / 2column -LRB- k +2 -RRB- m ' -LRB- k +1 -RRB- m ` km ' Figure 11 : The subrectangles encountered in a band of m0-rectangles . References -LSB- ACG +90 -RSB- Baruch Awerbuch , Israel Cidon , Inder Gopal , Marc Kaplan , and Shay Kutten . Distributed control for paris . In Proc . 9th ACM Symp . on Principles of Distributed Computing , pages 145 -LCB- 160 , 1990 . -LSB- ACK90 -RSB- Baruch Awerbuch , Israel Cidon , and Shay Kutten . Optimal maintenance of replicated information . In Proc . 31st IEEE Symp . on Foundations of Computer Science , 1990 . -LSB- ACK +91 -RSB- Baruch Awerbuch , Israel Cidon , Shay Kutten , Yishay Mansour , and David Peleg . Broadcast with partial knowledge . In Proc . 10th ACM Symp . on Principles of Distributed Computing , 1991 . -LSB- BGJ +85 -RSB- A. E. Baratz , J. P. Gray , P. E. Green Jr. , J. M. Jaffe , and D.P. Pozefski . SNA networks of small systems . IEEE Journal on Selected Areas in Communications , SAC-3 -LRB- 3 -RRB- :416 -LCB- 426 , May 1985 . -LSB- CG88 -RSB- I. Cidon and I. S. Gopal . PARIS : An approach to integrated high-speed private networks . International Journal of Digital & Analog Cabled Systems , 1 -LRB- 2 -RRB- :77 -LCB- 86 , April-June 1988 . -LSB- KT59 -RSB- A. N. Kolmogorov and V. M. Tihomirov . Epsilon-entropy and epsilon-capacity of sets in functional spaces . Uspekhi Matematicheskikh Nauk -LRB- N. S. -RRB- , 14:3 -LCB- 86 , 1959 . Translated in American Mathematical Society Translations -LRB- Series 2 -RRB- 17:277 -364 -LRB- 1961 -RRB- . -LSB- Man82 -RSB- B. Mandelbrot . The Fractal Geometry of Nature . W. H. Freeman and Company , San Francisco , 1982 . -LSB- MRR80 -RSB- John McQuillan , Ira Richer , and Eric Rosen . The new routing algorithm for the arpanet . IEEE Trans . on Commun. , 28 -LRB- 5 -RRB- :711 -LCB- 719 , May 1980 . -LSB- PS32 -RSB- L. Pontrjagin and L. Schnirelman . Sur une propri?et?e m?etrique de la dimension . Annals of Mathematics , 33:156 -LCB- 162 , 1932 . -LSB- Wec80 -RSB- S. Wecker . DNA : the digital network architecture . IEEE Transactions on Communication , COM-28 :510 -LCB- 526 , April 1980 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

distributed computing
routing
topology update

distributed databases
network management