Abstract:
A method and circuit for precisely synchronizing clocks in separate nodes on a communication network is provided by adjusting timestamps and related data in network messages. The circuit will allow a daisy-chain connection of the nodes and will forward time synchronization frames while accounting for delays in a manner that does not use boundary clocks, but does not depart from the IEEE 1588 standard protocol. The delays will be added on the fly to synchronization packets and the IP checksum and frame CRC will be adjusted. Deterministic data delivery and redundant data paths are also provided in a full duplex Ethernet network.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]     Not Applicable  
       STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT  
       [0002]     Not Applicable 
     
    
     TECHNICAL FIELD  
       [0003]     The present invention relates generally to industrial control devices for the control of machines and processes and in particular, industrial control devices which can be connected to a distributed high speed network.  
       BACKGROUND ART  
       [0004]     In industrial control, there is a class of distributed motion control applications that require both precision time synchronization and deterministic data delivery. Precision time synchronization at the nodes can be achieved with a network communication protocol according to IEEE 1588, Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, 2002, and by using frequency-compensated clocks as disclosed in our prior U.S. patent application Ser. No. 10/347,658 filed Jul. 22, 2003. Motion control applications also require deterministic data delivery, which means that input data will be received and output data will be transmitted at specific time points based on predetermined periodic intervals. This requires coordination of network bandwidth with resources at the intermediate and end nodes. One way to coordinate network bandwidth uses precise and detailed scheduling of both data production and network transmissions for data delivery. Another way uses a combination of coarse scheduling of data production and the use of frame priorities to prioritize network transmissions for data delivery according to IEEE 802.3, Part 3, Standard for Carrier Sense Multiple Access with Collision Detection Access Method and Physical Layer Specification, 2002.  
         [0005]     In distributed control applications, it is desirable to have a daisy-chain network bus topology due to simplified wiring requirements. It is also desirable to provide a redundant data delivery path in case of a network failure. This bus topology can be accomplished through half duplex Ethernet, but this type of network has several drawbacks such as collisions, a 100-meter copper cable length limit and technology obsolescence. To avoid collisions in this type of network, fine scheduling and control of transmissions are necessary. Further, data throughput is limited to 100 Mbps by the half duplex nature of network. These limitations make it undesirable to use half duplex Ethernet for distributed motion control applications.  
         [0006]     Full duplex Ethernet uses switching technology to avoid collision domains and doubles peak data throughput to 200 Mbps through concurrent transmission and reception. The use of switches in network topology results in a typical star configuration. The switches avoid collision by queuing Ethernet frames on a per port basis. In order to avoid propagating errors on received frames, most switches use store and forward architecture, in which the frames are queued even when there is no resource contention on a port. This results in a delay corresponding to frame size plus intrinsic queuing and switching delay.  
         [0007]     It is also possible to connect switches in a daisy-chain topology with full duplex Ethernet. The maximum copper cable length limit is raised to (N+1)*100 meters for N switches. However, significant problems result for time synchronization and deterministic data delivery in a network with this topology. There are random time delays introduced by the switches that affect time synchronization resulting in loss of synchronization precision and stability. Under current technology with IEEE Standard 1588, a boundary clock can be used on every switch node to manage time synchronization between an upstream master clock and downstream slave clocks. Even with use of boundary clocks on switches, it is difficult to achieve sub-microsecond level precision synchronization required for distributed motion control, when more than four switches are cascaded.  
         [0008]     As mentioned above, in order to avoid propagating errors on received frames, most switches use store and forward architecture, in which the frames are queued even when there is no resource contention on a port. With store and forward architecture, significant random cumulative delays are introduced in the data delivery path resulting in non-deterministic data delivery and other performance issues.  
         [0009]     One object of the invention is to provide time synchronization of the daisy-chain connected network nodes. Another object of the invention is to provide deterministic data delivery. Another object of the invention is to provide a redundant data path in the event of a network failure.  
       SUMMARY OF THE INVENTION  
       [0010]     The present invention provides a method and circuit for time synchronization of daisy-chained node clocks. The circuit includes a network switch, which can be included in each node in the network. The switch will allow a cascaded connection of the nodes in any binary tree topology and will forward time synchronization frames while accounting for delays in a manner that does not use boundary clocks, but does not depart from the IEEE 1588 standard protocol.  
         [0011]     To achieve precision time synchronization, the node switch will accurately account for delays through the switch. The delays will be added on the fly to synchronization packets and the UDP checksum and frame CRC will be adjusted. This approach will result in significant improvement over systems using boundary clocks.  
         [0012]     Deterministic bidirectional data delivery for distributed motion control is facilitated by the cut through forwarding nature of embedded switch, enforcement of frame priorities encoded by origin nodes on network transmissions by embedded switch and by coarse scheduling of motion control loops. Since motion control loops are synchronized to a coarse schedule and all nodes are precisely time synchronized, all nodes will transmit almost at the same relative point every time resulting in minimal contention on switches. With these changes, the daisy chain with distributed embedded switches will look like a single switch for an end device. It should be noted that none of these changes is a departure from the IEEE 802.3 standard or the IEEE 1588 standard.  
         [0013]     In a further aspect of the invention, redundancy is provided by extending the daisy chain to a ring topology. In this case, a designated supervisory device will have one master clock with two specialized ports and a specialized signaling protocol for providing redundancy. The end nodes will measure and save delay times of two paths of ring topology through two ports of the master node. During normal operation, the supervisory device will break endless circulation of packets from the second port to the first port and vice versa, and will simultaneously monitor traffic by sending special packets on the first port and tracking them on the second port. Simultaneously, the supervisory device and end nodes will monitor link status of their ports periodically and the end nodes will notify the supervisory device in case of failure of a port through other port. When the supervisory device detects or is notified of a network failure, it will broadcast this status to all nodes through two different messages on its two ports. Furthermore, it will forward all packets from one port to other, effectively converting the network to bus topology. On receiving the broadcast, those end nodes that received the message from second port on supervisory device will switch to measured and saved delay of second path through second port of master clock. Those end nodes that received broadcast from the first port on supervisory device will take note of situation and will continue using measured delay through first path. By switching the time delay, time synchronization will continue to function correctly. By switching to bus topology, data delivery will continue to function correctly. Since the end nodes can tolerate short-term loss of synchronization messages and control data from network failure to topology transition, the system will function continuously. Through additional messages the supervisory device can pinpoint failure and signal an operator for network maintenance. After the operator notifies about completion of maintenance, the system will go through a reverse process to return to normal mode of operation. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0014]      FIG. 1  is a system diagram illustrating a network with nodes according to the present invention;  
         [0015]      FIG. 2  is a detailed block diagram of a switch used in the nodes of  FIG. 1 ;  
         [0016]      FIG. 3  is a detailed block diagram of the frequency compensated clock, time stamp registers and target timers portion of  FIG. 2 ;  
         [0017]      FIGS. 4   a - 4   c  are diagrams of possible node connections using the switch of  FIG. 2 ;  
         [0018]      FIG. 5  is a flow chart for the transmit channel hardware logic of any port in the switch of  FIG. 2 ;  
         [0019]      FIG. 6  is a flow chart for the receive channel hardware logic of any port in the switch of  FIG. 2 ;  
         [0020]      FIG. 7  is a system diagram illustrating a normal mode of operation for a ring topology network;  
         [0021]      FIG. 8  is a system diagram illustrating failure mode of operation for network of  FIG. 7 ;  
         [0022]      FIG. 9  is an abridged message frame map of a time-synchronization message that is transmitted on the network of  FIGS. 1, 7  and  8 ; and  
         [0023]      FIG. 10  is an abridged message frame map of a beacon message which is periodically transmitted on the network of  FIG. 7 .  
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0024]     Referring now to  FIG. 1 , nodes  10 ,  11 ,  12  and  13  are connected through a communication network  15 . In this embodiment the network  15  is a full duplex Ethernet network operating at data rates up to 200 Mbps. Nodes  10 ,  11 ,  12  and  13  may be, for example, industrial controllers, network bridges, remote I/O modules, standard motor drives, servo motor drives, or human-machine interfaces. If any of these nodes are controllers, then these controllers may be connected to local I/O modules and devices, including motion control devices for controlling motors, robots and other motion devices. To provide redundant network connections, the network  15  can take the form of a ring with two data paths  15   a  and  15   b  communicating with two respective ports  10   c  and  10   d  on a designated supervisory node device  10 . The designated supervisory node device  10  may be a special controller, a special network bridge or any other special device designed for this role. This device has one master clock from which timestamp data is communicated through two specialized ports: a first port  10   c  and a second port  10   d.    
         [0025]     To facilitate a full duplex daisy chain a special purpose switch  10   a,    11   a,    12   a,    13   a,  as exemplified by switch  12   a,  in the form of an FPGA (field programmable gate array) or other ASIC (application specific integrated circuit) is provided for each node. Referring to  FIG. 2 , the switch has three daisy chain ports  19 ,  20 ,  21  and a local port  22 . All three daisy chain ports are identical in capabilities and are interchangeable. Two daisy chain ports can be used for uplinking and downlinking on the daisy chain. The third daisy chain port  21  can be used to start a new daisy chain (like  15   c  to node  14  on  FIG. 1 ). Referring to  FIG. 1 , each of the nodes  10 ,  11 ,  12 ,  13  and  14  includes a CPU  10   b,    11   b,    12   b,    13   b  and  14   b  respectively. Referring to  FIG. 2 , in the switch  12   a,  the local port  22  and host CPU bus interface  23  are used to communicate with local host CPU.  
         [0026]     With three daisy chain ports on the embedded switch, complex daisy chains of any binary tree topology can be constructed. As seen in  FIG. 4   a,  with a simple bus topology daisy chain the usual connection of nodes N 1 -N 2 -N 3 -N 4 -N 5  occurs along a single data path.  FIG. 4   b  illustrates ring topology redundant data paths by closing the loop from node N 5  to node N 1 .  FIG. 4   c  illustrates a complex daisy chain network made possible by the third daisy chain port  21  in the switch  12   a.  The main data path is through nodes N 1  to N 5 . An additional duplex data path is formed from node N 2  to N 6  and another duplex data path from node N 3  to nodes N 7  and N 10 , and still another duplex data path is formed from node N 4  to nodes N 8 , N 9  and N 11 .  
         [0027]     Referring again to  FIG. 1 , each of the nodes  10 ,  11 ,  12 ,  13  and  14  sends and receives message frames defined by IEEE standard 1588, which is hereby incorporated by reference. Following this protocol, the node  10  as a time master node may send a “synchronize” time synchronization message frame  85  to network time slave nodes  11 ,  12 ,  13  and  14 . Similarly, nodes  11 ,  12 ,  13  and  14 , as time slaves, may send a “delay request” time synchronization message frame  85  to the master node  10 . As seen in  FIG. 9 , the time synchronization message frame  85  includes fields for a preamble  86 , a start of frame delimiter  87 , a destination address  88 , a source address  89 , a frame priority number  90 , a UDP checksum  91 , a time synchronization message type identifier  92 , an origin timestamp  93  and a CRC  94  (cyclic redundancy checking code).  
         [0028]     The CPUs  10   b,    11   b,    12   b,    13   b  and  14   b  on network nodes  10 ,  11 ,  12 ,  13  and  14  encode a highest priority to time synchronization message frames, a lower priority to motion control data message frames, a still lower priority to message frames with discrete or process I/O data and a lowest priority to message frames with non-critical configuration and other data. The switch uses encoded priority information to prioritize network transmissions and accords lowest priority to message frames without any priority information. The term “frame” means a unit of transmitted data under the applicable IEEE standards.  
         [0029]     In the present embodiment, the motion control data is managed by coarse schedulers in the motion controller and in the servo drives. The coarse schedulers may require an update every 250 microseconds and the 250-microsecond loop starts on all nodes (both controller and drives) within one microsecond of each other. Alternatively, the coarse schedulers may stagger the 250-microsecond loops and the loops will have to start within one microsecond from required starting points. In either case, the latter is a phase relationship that requires accurate time synchronization.  
         [0030]     Time synchronization is a fundamental requirement for distributed motion control and certain classes of distributed control. This is different from traditional process/discrete control systems.  
         [0031]     For instance, a process/continuous controller may require I/O updates once every 20 milliseconds at most, but there is no explicit need to synchronize phase relationships. Similarly a discrete control controller may require I/O updates once every 1 millisecond at most without a need to maintain phase relationship.  
         [0032]     Without compensation for time differences, individual nodes will drift apart and report different times. For most systems, including networked computers, an accuracy on the order of one to ten milliseconds is sufficient and this can be obtained in software. For distributed motion control systems, a more stringent requirement of sub-microsecond accuracy is needed.  
         [0033]     The CPUs  10   b,    11   b,    12   b,    13   b  and  14   b  on network nodes  10 ,  11 ,  12 ,  13  and  14  each communicate with the network switches  10   a,    11   a,    12   a,    13   a  and  14   a,  respectively and in particular with their registers as seen in  FIG. 2  for switch  12   a.    
         [0034]      FIG. 3  shows the circuitry  24  included in switch  12   a  in  FIG. 2 . A 64-bit delay time counter  31  in every switch is set initially to zero. In one embodiment, an oscillator circuit (not shown) provides a clock signal to an accumulator  40  causing the accumulator  40  to add a value received from an addend register  41  to the current contents of the accumulator  40 . Repeated addition operations triggered by the oscillator cause a regular overflow of the accumulator  40  at an overflow output of the accumulator  40 . The signal from the overflow output strobes a count input of the delay time counter  31  for the node. The value in the addend register  41  may be written and read by the CPU  12   b,  so that CPU  12   b  may effectively control the frequency of delay time counter  31 . In an alternative embodiment, an oscillator circuit may directly strobe the count input of delay time counter  31 .  
         [0035]     There are four ports on the network switch  12   a,  each with transmit and receive channels for a total of eight channels that are operating in parallel. One timestamp register  32 - 39  based on delay time counter  31  is provided for each channel. A 64-bit system time clock  30  is provided for tracking synchronized time in every node. The count input of system time clock  30  is strobed by the overflow output of accumulator  40 . Two timestamp registers  42  and  43  based on the system time clock are provided for timestamping “synchronize” and “delay request” time synchronization messages. Two message detector circuits (not shown) in transmit  22   a  and receive  22   b  channels of local port  22  trigger timestamp on registers  42  and  43 . The host CPU  12   b  uses these registers to compute values for the addend register  41 . Further details on system time clock  30 , addend register  41 , accumulator  40 , two timestamp registers  42 ,  43 , message detector circuits and the procedure to compute values for addend register  41  are described in U.S. patent application Ser. No. 10/347,658, cited above, which description is incorporated herein by reference. Additional timestamp registers  48  and  49  based on the system time clock are provided for timestamping “delay request” messages through second port, a feature useful in redundancy and capturing external events such as synchronization with a global positioning system or external clocks. The target time registers  44 ,  45  are provided to set future time notification. When one of the comparators  46 ,  47  sees that the system time clock equals target time in its associated register  44 ,  45 , it will send an interrupt signal to the host CPU  12   b.  Multiple target timers are provided so that host CPU  12   b  can use each for a dedicated purpose, for example, one for normal scheduling and the other for redundancy.  
         [0036]     Next, it will be explained how the hardware logic in transmit and receive channels of any port in switch  12   a  updates the origin timestamp  93  in time synchronization messages “on the fly,” as shown in  FIGS. 5 and 6 . It should be noted that the blocks in charts of  FIG. 5  and  FIG. 6  describe hardware logic that executes in parallel. As seen in  FIG. 5 , the hardware logic associated with the receive channel of any port in switch  12   a  starts receiving preamble  86  of a frame, as represented by start block  50 . As represented by process block  51 , when the timestamp point according to IEEE 1588 standard has been reached during frame reception, a timestamp trigger signal is sent to associated timestamp register  32 - 39  to capture receive timestamp (Rxts) from delay time counter  31 . The captured receive timestamp (Rxts) is then copied from timestamp register  32 - 39  and saved in a frame descriptor block in memory (not shown) for the received frame. Next, as represented by process block  52 , the destination  88  and source  89  network addresses in the message are extracted and saved into the frame descriptor block as and when they are received. Next, as represented by process block  53 , a check is made for the presence of frame priority number  90  in the frame, when an appropriate point is reached during frame reception. If there is no frame priority number  90 , as represented by the “no” branch from decision block  53 , then the frame is assigned the lowest priority and the priority is saved to the frame descriptor block, as represented by process block  54 . If there is a frame priority number  90 , as represented by the “yes” branch from decision block  53 , then the frame priority  90  is extracted from the message and saved to frame descriptor block as represented by process block  55 . After executing process block  54  or process block  55 , the hardware logic proceeds to decision block  56 .  
         [0037]     As represented by decision block  56 , a check is made on frame priority, and if it is the highest priority, as signified by the “yes” result, this signifies that it may be a time synchronization message has been received. Then, as represented by process block  57 , a UDP checksum  91  ( FIG. 9 ) is extracted from the message and saved to the frame descriptor block as and when it is received. Subsequently, multiple locations in the frame, as and when they are received, are probed to confirm predefined values until a time synchronization message type identifier  92  is received to verify that this is a time synchronization message, as represented by decision block  58 . If the answer is “yes,” as represented by the “yes” branch from decision block  58 , then a time sync flag is set in the frame descriptor block as represented by process block  59 . Then, the origin timestamp field  93  (Orts) from the message is extracted and saved to a buffer descriptor block in memory when it is received, as represented by process block  60 . The hardware logic then proceeds to decision block  61 .  
         [0038]     Returning to decision block  58 , if the result of this decision is “no,” then the message is not a time synchronization message and the hardware logic proceeds to decision block  61 . Returning to decision block  56 , if the result of this decision is “no”, then the message is not a time synchronization message and the hardware logic proceeds to decision block  61 . At decision block  61 , a check is made to see if the frame source address is same as local port address. If the answer to this decision is “yes”, then another check is made to see if the currently executing receive channel is part of local port as represented by decision block  62 . If the answer to this decision is “no”, then the frame is discarded as represented by end block  63 . This discarding of frame prevents frames from going in endless loops in misconfigured ring topology networks and during network failure recovery transition from bus topology to ring topology. If the answer to decision block  62  is “yes” or if the answer to decision block  61  is “no”, the hardware logic proceeds to decision block  64 . At decision block  64 , a check is made to see if the frame destination address is same as local port address. If the answer to this decision is “yes”, then the frame is forwarded only to the transmit channel of the local port as represented by end block  66 . If the answer to decision block  64  is “no”, then the frame is forwarded to transmit channels of other daisy chain ports and of the local port as represented by end block  65 . It should be noted that at end block  65  and end block  66 , the receive channel hardware logic of a port will not forward frames to the transmit channel of its own port.  
         [0039]     Referring next to  FIG. 6 , the hardware logic in transmit channel of any port begins receiving a forwarded frame as represented by start block  70  from block  65  or block  66  in  FIG. 5 . Then, a decision block  71  is executed to determine if the transmit channel is free to transmit the forwarded frame. If not, as represented by the “no” branch from decision block  71 , then the message is queued according to priority in transmit channel queue as represented by connector block  72 . If the answer is “yes,” as represented by the “yes” branch from decision block  71 , then transmission of a frame preamble is initiated as represented by process block  73 . If a time synchronization flag has been set in  FIG. 5 , this is detected by execution of decision block  71  in  FIG. 6 . When a timestamp point according to IEEE 1588 standard is reached during transmission, a timestamp trigger is sent to associated timestamp register  32 - 39  to capture transmit timestamp (Txts) from delay time counter  31 , as represented by process block  79 . Next, the switching delay experienced by the frame inside switch is calculated by subtracting a saved receive timestamp (Rxts) from a transmit timestamp (Txts), as represented by process block  80 . Next, as represented by process block  81 , the UDP checksum for the time synchronization message is recomputed from the saved UDP checksum, for the added switching delay to origin timestamp at block  82  and inserted at appropriate location in frame. Next, as represented by process block  82 , the switching delay is added to the saved origin timestamp and is inserted at the appropriate location in frame. Then, the CRC error checking code for the entire frame is computed and inserted at the end of frame, as represented by process block  83 . The frame transmission is completed, followed by inter-frame gap according IEEE 802.3 standard and the transmit channel is ready for transmission as represented by process block  75 .  
         [0040]     If the message was not a time synchronization message, as represented by the “no” branch from decision block  74 , then blocks  79 - 83  are skipped, the transmission of forwarded frame simply continues until completion, followed by inter-frame gap according IEEE 802.3 standard and the transmit channel is ready for transmission as represented by process block  75 . In either event, the transmit channel queue is checked as represented by decision block  76 . If the queue is empty, as represented by the “yes” result from executing decision block  76 , then the hardware logic will wait for the next forwarded frame, as represented by end block  78 . If the queue has one or more frames, as represented by the “no” result from executing decision block  76 , then the hardware logic will dequeue the highest priority message, as represented by process block  77  and begin transmitting it, as represented by process block  73 .  
         [0041]     By adding delay in the switch to the received origin timestamp  93 , the switch  12   a  becomes transparent to any downstream clocks. The adjustment accounts for random delays through the switch  12   a,  and then only fixed delays on the network media remain, which can be easily measured and compensated for. It should be noted that the switching delays are fully accounted for time synchronization messages in both master-to-slave and slave-to-mater paths.  
         [0042]     Next, the redundancy aspects of the invention will be described in more detail.  FIG. 7  shows a ring topology network in normal mode of network operation with redundant data paths  15   a,    15   b.    FIG. 8  shows a mode of operation when there has been an interruption in communication at some point  17  in the ring. In both modes of operation, the special features required to support redundancy are enabled by CPU  10   b  on node  10 , by setting appropriate control bits in switch control registers  25  on switch  10   a.    
         [0043]     In  FIG. 7 , the designated supervisory device  10  transmits all message frames through only one port  10   c,  but receives message frames through both ports  10   c  and  10   d.  All message frames which are transmitted from daisy-chained nodes  11 ,  12 ,  13  and  16  will appear at both ports  10   c  and  10   d,  but the second port  10   d  will discard most of the frames except for “delay request” messages to master clock on node  10  intended to measure delay for data path  15   b  through second port  10   d  and network failure notification frame from nodes  11 ,  12 ,  13  and  16  to node  10 . On the other hand, port  10   c  will accept all frames except for “delay request” messages to master clock on node  10  intended to measure delay for data path  15   b  through second port  10   d.  Using port  10   c  and timestamp register  43  on switch  10   a  the slaves can measure delay to master clock on node  10  for data path  15   a.  Using port  10   d  and timestamp register  48  on switch  10   a  the slaves can measure delay to master clock on node  10  for data path  15   b.  In normal operation, and as part of a startup procedure, the nodes  11 ,  12 ,  13  and  16  will measure and save delay information relative to the master clock in the supervisory device  10  and further in relation to both data paths  15   a  and  15   b.    
         [0044]     The supervisory device  10  transmits a beacon message frame  95  illustrated in  FIG. 8  from the first port  10   c  to the second port  10   d,  once every scheduled period, for example, every 250 microseconds, and monitors arrival of at least one beacon message on the second port  10   d  before a timeout (say 500 microseconds). As seen in  FIG. 8 , the beacon message frame  95  includes a preamble  96 , a destination address  97  and a beacon message identifier  98  and a CRC  99  error checking code.  
         [0045]     In addition, all nodes  10 ,  11 ,  12 ,  13  and  16  monitor the link status of their two ports from IEEE 802.3 physical layer (PHY) devices once every specified period, such as 250 microseconds. If there is a failure of communication due to a fault  17  as represented in  FIG. 8 , then nodes  12  and  13  will discover it through link status failure on ports  12   c  and  13   d,  and they will send link failure message to supervisory device  10  through their other working ports  12   d  and  13   c.    
         [0046]     In general, the supervisory device  10  may detect a link status failure on its ports  10   c  and  10   d,  or receive a link failure message from one of the nodes  11 - 13 ,  16 , and enter failure mode. Alternatively, the supervisory device  10  will fail to receive at least one beacon message before timeout (500 microseconds), and will enter failure mode. Upon entering failure mode, the supervisory device  10  will then broadcast two different failure messages through the two ports  10   c,    10   d  to all nodes  11 ,  12 ,  13  and  16  about the failure. The supervisory device  10  will then, by setting appropriate control bits in switch control registers  25  on switch  10   a,  start forwarding all message frames from port  10   c  to  10   d  and vice versa, effectively converting ring topology to bus topology. Daisy-chained nodes  11 - 13 ,  16  that receive a failure message from port  10   d  will change their delay relative to the master clock to the measured and saved delay information for data path  15   b.  While those nodes that received the failure message from port  10   c  will take note of the situation and will continue using measured delay information for data path  15   a.  This behavior ensures that time synchronization continues to work correctly. Meanwhile the nodes with failed link status ports will disable failed ports by setting appropriate control bits in control registers  25  on their switches. Since nodes are set up to tolerate data loss for a period more than timeout, the system will continue functioning normally. The supervisory device  10  then identifies link failure location and an alarm is set off for an operator through a human-machine interface. After the operator has restored the failed network link, the operator will reset the alarm and request normal operation. Upon receiving this request, the supervisory device  10  will broadcast a message with suitable time in future when all nodes  10 - 13  and  16  will return to normal mode of operation. The supervisory device  10  and all nodes  11 - 13 ,  16  will then return to normal mode precisely at appointed time. This involves re-enabling of disabled ports in the daisy-chained nodes  11 - 13 ,  16  by resetting appropriate control bits in control register  25  on their switches, with the daisy-chain connected nodes switching back to the measured delay information through data path  15   a  and the supervisory device  10  returning to its normal mode of operation by resetting appropriate control bits in control register  25  on switch  10   a.  The latter action converts the network back from bus topology to ring topology. As mentioned earlier in  FIG. 5 , the switches have a safety feature whereby frames are prevented from going into endless loops during this transition.  
         [0047]     This has been a description of the preferred embodiment. It will be apparent to those of ordinary skill in the art, that certain details of the preferred embodiment may be modified to arrive at other embodiments without departing from the spirit and scope of the invention as defined by the following claims.