Impending control plane disruption indication using forwarding plane liveliness detection protocols

Techniques are described for providing an indication of an impending control plane disruption of a router using forwarding plane liveliness detection protocols. A forwarding plane of the router outputs liveliness detection messages that, when received by a peer router, provide an indication that the forwarding plane is operational and able to forward packets. When constructing the liveliness detection messages, the forwarding plane may embed additional information indicative of any impending disruption in the control plane of the network device. In this way, the forwarding plane of the transmitting router provides an indication to the peer router that, although currently operational, the control plane of the router may in the near term become non-operational. The peer router, in response to receiving an enhanced liveliness detection message indicating an impending control plane disruption, suppresses a recovery action otherwise triggered by a loss of communication with a control plane of the network device.

TECHNICAL FIELD

This disclosure relates generally to computer networks, and more specifically, to mitigating control plane disruptions within computer networks.

BACKGROUND

A computer network is a collection of interconnected computing devices that can exchange data and share resources. In a packet-based network, such as the Internet, the computing devices communicate data by dividing the data into small blocks called packets, which are individually routed across the network from a source device to a destination device. The destination device extracts the data from the packets and assembles the data into its original form. Dividing the data into packets enables the source device to resend only those individual packets that may be lost during transmission.

Certain devices, referred to as routers, maintain routing information that describes available routes through the network. Each route defines a path between two locations on the network. In order to maintain an accurate representation of a network, each router typically executes one or more routing protocols in a “control plane” of the device. In general, the “control plane” of a router refers to dedicated hardware or other resources that provide the intelligence to communicate with other peer routers to learn the topology of a network including available routes through the network. By execution of the routing protocols, the control planes of routers maintain control-plane communication sessions through which the protocols exchange routing information that reflects the current topology of the network.

Based on the topology, the control plane of a router makes path selection and programs or otherwise configures a “data plane” or “forwarding plane” of the router, which refers to the dedicated hardware or other resources that is responsible for processing the packets traversing the network. Upon receiving an incoming packet, for example, the forwarding plane of the router examines information within the packet and forwards the packet in accordance with forwarding information installed within the forwarding plane by the control plane of the router.

SUMMARY

In general, techniques are described for providing an indication of an impending control plane disruption using an enhanced forwarding plane liveliness detection protocol. For example, techniques are described in which a protocol used for monitoring operational status of a forwarding plane of a router or other network device is enhanced to carry additional information that provides an indicator of an impending disruption in the control plane of the network device.

In one example, liveliness detection messages sent by a router in accordance with the forwarding plane liveliness detection protocol may provide an indication that the forwarding plane of a router is operation and able to forward packets. The liveliness detection messages may be enhanced to include additional information to provide an indication of impending disruption in the control plane of the network device. In some cases, the liveliness detection messages may be sent with periodically by the forwarding plane detection protocol with short periodicity (e.g., on the order of a few tens of milliseconds). In other examples, the liveliness detection messages may be sent on demand by the forwarding plane, i.e., when needed or in response to an event or received communication. In either case, by leveraging liveliness detection messages between forwarding planes, peer routers or other devices, such as network controllers, may thus be quickly informed of an impending loss of operation of the control plane of the router. In response, peer routers may adjust their own control plane operation so as to prevent various control plane protocols triggering traffic re-route or other recovery processes upon detecting unreachability of the router. That is, the peer router may suppress, at least temporarily a control-plane recovery or re-convergence process that would otherwise be triggered upon loss of control-plane communication with the router. In some examples, the additional information embedded within the messages of the forwarding plane liveliness detection protocol may further include an optional field specifying an expected duration for the impending control plane disruption. Moreover, in some examples, a subsequent forwarding plane message may be used to affirmatively indicate an end to the control plane disruption period.

Although described herein with reference to the liveliness detection protocols, the techniques may be applied with other protocols (e.g., layer two or layer three protocols) that execute in the forwarding plane or are otherwise delegated to the forwarding plane and that utilize period/on-demand messages with peer devices to indicate forwarding plane connectivity. Examples of such protocols include Address Resolution Protocol (ARP), the Neighbor Discover (ND) Protocol.

In one example, a method comprises executing, within a control plane of a network device, one or more routing protocols to exchange network topology information with one or more other network devices and to program a packet forwarding component of the network device to forward packets in accordance with the network topology information. The method further comprises detecting, with the packet forwarding component, an impending disruption to operation of the control plane of the network device and constructing, with the packet forwarding component, a liveliness detection message in accordance with a forwarding-plane liveliness detection protocol. The message comprises an indication of the impending disruption to operation of the control plane. The method includes outputting, by the packet forwarding component, the liveliness detection message to a peer network device in accordance with the forwarding plane liveliness detection protocol.

In another example, a network device comprises a routing component having a hardware-based processor to execute one or more control plane protocols and exchange network topology information with a peer network device. The network device further comprises a packet forwarding component of the network device programmed by the routing component to forward packets in accordance with the network topology information. The packet forwarding component exchanges liveliness detection messages with packet forwarding components of the peer network device in accordance with a forwarding-plane liveliness detection protocol. Responsive to an impending disruption to operation of the routing component of the network device, the packet forwarding component constructs one of the liveliness detection messages to include an indication of the impending disruption to operation of the routing component and outputs the liveliness detection message to the peer network device in accordance with the forwarding plane liveliness detection protocol.

In another example, a network device comprises a routing component having a hardware-based processor that provides a control plane for execution of one or more protocols. A packet forwarding component of the network device is programmed by the routing component to forward packets in accordance with the network topology information. The packet forwarding component executes a forwarding-plane liveliness detection protocol to receive liveliness detection messages from a peer network device, a first one of the messages comprising an indication of an impending disruption to operation of the routing component. The routing component, in response to receipt of the first one of the liveliness detection message by the packet forwarding component, suppresses a recovery action otherwise triggered by a loss of communication with a control plane of the peer network device.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating an example network system10in which the techniques described herein may be implemented. In this example, network system10includes service provider network19, which is communicatively coupled to customer networks16A and16B (“customer networks16”). In turn, service provider network19includes routers12A-12F (“routers12”), which operate and interact with one another in accordance with the techniques described herein. Routers12are communicatively coupled to one another, either directly, or indirectly, via physical links14A-14G (“links14”). Links14represent any physical medium, such as a copper wire, a coaxial cable, any of a host of different fiber optic lines, a wireless connection, and various combinations thereof. In the specific example ofFIG. 1, routers12C and12F represent edge routers. More specifically, routers12C and12F provide an interface between service provider network19and respective customer networks16.

Additionally, customer networks16include respective customer edge routers17A and17B (“CE routers17”). As shown, each of CE routers17is linked to a respective edge router of routers12. Edge routers12C and12F communicate with CE routers17to provide customer networks16with access to service provider network19. As shown, each of customer networks16may be a network for a site of an enterprise. Each of customer networks16may include one or more computing devices (not shown), such as personal computers, laptop computers, handheld computers, workstations, servers, switches, printers, or other devices. Service provider network19may be coupled to one or more networks administered by other service providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet. Consequently, customer networks16may be viewed as edge networks of the Internet.

The service provider may provide computing devices within customer networks16with access to the Internet via service provider network19, which allows computing devices within one of customer networks16to communicate with computing devices within the Internet or the other one of customer networks16.FIG. 1may be a simplified view of network system10. For instance, service provider network19can include a variety of other network devices (not shown), such as routers, switches, servers, and the like and may be connected to other networks. In contrast to edge routers12C and12F, the remainder of routers12represents intermediate routers. More specifically, intermediate routers12A,12B,12D, and12E provide various communication channels between edge routers12C and12F.

Routers12execute control-plane routing protocols to maintain accurate representation of the topology of service provider network19. For example, routing components within routers12maintain peering sessions with each other and exchange routing information for routes or links within service provider network19in accordance with one or more routing protocols. In the example ofFIG. 1, routers12A,12B are shown as maintaining a control-plane communication session18A for exchanging routing information, such as topology information and available routes through service provider network19. Similarly, routers12C,12D are shown as maintaining a control-plane communication session18B for exchanging routing information. Although only shown by way of example with respect to routers12A,12B and12C,12D, other combinations of routers12,17typically maintain control-plane communication sessions for exchanging routing information using various routing protocols. Example protocols include the Border Gateway Protocol (BGP) distance vector routing protocol and the Open Shortest Path First (OSPF) and Intermediate System-to-Intermediate System (IS-IS) link state routing protocols.

In addition to routing protocols, the control planes of routers12typically execute a variety of other so called “control plane” protocols, such as the Address Resolution Protocol (ARP), the Neighbor Discover (ND) Protocol, various Operations, Administration, and Maintenance (OAM) protocols and the like.

In the example ofFIG. 1, service provider network19may also include a centralized controller35that provides complete control-plane functionality for the service provider network. As described herein, controller35may provide seamless end-to-end path selection and routing control over routers12through service provider network19. That is, controller35may include a sophisticated control plane that interacts with control planes of routers12for ultimately programming and controlling data planes of the routers.

In association with the control plane communication sessions for conveying network topology information, routers12and controller35may periodically send status inquiries (e.g., send “periodic packets” or “periodic data”) to one another in order to provide and continue to prove/verify the control plane communication between the peer routers. That is, by sending periodic packets and detecting receipt of similar periodic packets, routers12and controller35detect any failures in control plane communications between the peer routers, either as a result of failure of an internal routing component or a given control-plane protocol of one or more of routers12. In this example ofFIG. 1, control plane components of routers12A,12B are shown as exchanging periodic packets20for continuing to verify control plane communications between the routers. Although not shown, other routers12and controller35typically exchange periodic packets between respective control plane components (e.g., routing components) in accordance with control plane protocols for verifying control plane connectivity between the routers.

Each router typically maintains a relatively long timeout value in association with the monitoring of control plane connectivity, such as three to twenty seconds or more as examples. Upon detecting loss of control plane communication, the detecting router12may elect to update one or more internal representations of the topology of service provider network19, and outputs session messages to the other routers12to inform the other routers12of the topology changes. For example, upon detecting loss of control plane communication with router12B by way of periodic packets20, router12A may elect to update its routing information so as to direct packets destined for customer network16B to router12E, thereby routing around router12B.

In some cases, router12B may support various features such as “non-stop forwarding” and “graceful restart.” For example, packet forwarding components (e.g., hardware-based packet forwarding engines) of router12B may support “non-stop forwarding” and, therefore, is capable of continuing to forward packets even though the control plane of router12B may no longer be operational. This allows the packet forwarding components of router12B to forward packets in accordance with its last know state as to the topology of service provider network12. As such, even though the control plane of router12B may be out of communication, the forwarding plane of router12B continues operation in uninterrupted fashion. Moreover, the control plane of router12B may support “graceful restart,” which refers to the capability of preserving the forwarding information while restarting a routing communication session, e.g., a BGP session, in the control plane. In other words, even though router12B may have lost control plane communication via session20, router12B may be able to continue to forward packets and may, upon restoring control plane functionality, may be able to restart certain control plane routing protocols without substantially disrupting routing services. As such, by way of periodic packets20, router12A may detect loss of control plane communication with router12B but typically only after a significant period of time (e.g., after reaching a threshold timeout value of 3-20 seconds or more). In response, router12A may direct traffic around12B or may elect to defer redirection when router12B supports features such as non-stop forwarding and graceful restart; however, in either case the forwarding plane of router12B may have been utilizing state forwarding information for some time prior to detection of the loss of control plane communication between the routers.

In addition to utilizing protocols and corresponding periodic packets for monitoring an ability to communicate directly between control plane components, routers12may separately execute data plane protocols to monitor operation of the underlying packet forwarding components (also referred to as packet forwarding engines or a “forwarding plane”) to confirm that the forwarding components of the routers are currently able to forward packets. In the example ofFIG. 1, routers12A and12B exchange data-plane related liveliness detection messages22to detect the liveliness of the forwarding plane of each other, and the routers may exchange liveliness detection messages22for detecting forwarding plane liveliness on-demand or at a periodic rate that is significantly faster (e.g., every 10-50 ms) than the rate at which the routers exchange packets20(e.g., 3-20 seconds) for monitoring control plane communication capability between the routers. Protocols for monitoring the operation of packet forwarding components may generally be referred to as forwarding plane liveliness detection protocols. Although described herein with reference to the liveliness detection protocols, the techniques may be applied with other protocols (e.g., layer two or layer three protocols) that execute in the forwarding plane or are otherwise delegated to the forwarding plane and that utilize period/on-demand messages with peer devices to indicate forwarding plane connectivity. Examples of such protocols include Address Resolution Protocol (ARP), the Neighbor Discover (ND) Protocol.

When a forwarding plane message22associated with a forwarding plane liveliness detection protocol is not received in the allotted time frame, a packet forwarding component of the router expecting receipt of the periodic packet determines that a network event has occurred causing the forwarding plane of the peer router to be unable to process packets. In such case, the router may elect to immediately reroute traffic around the peer router since the peer router is no longer forwarding packets. For example, router12A may, upon detecting failure lack of responsiveness from the forwarding plane of router12B via liveliness detection messages22, immediately reroute traffic around router12B since the forwarding plane of router12B appears to no longer be forwarding packets. One exemplary forwarding plane liveliness detection protocol is the bidirectional forwarding detection (BFD) protocol. The BFD protocol provides a very short interval of time between which the forwarding plane of routers12must transmit periodic messages, and thus may facilitate a fast detection of failures by packet forwarding engines for any of routers12that are in active BFD sessions. Further details of the BFD protocol may be found in the proposed standard for BFD, by D. Katz and D. Ward (Juniper Networks, June 2010, ISSN: 2070-1721), the entire content of which is incorporated herein by reference.

In general, techniques are described for preemptively providing an indication of an impending control plane disruption using forwarding plane liveliness detection protocols. For example, techniques are described in which a data plane protocol used for monitoring operational status of a forwarding plane of a router or other network device is enhanced to carry additional information that provides an indicator of an impending disruption in the control plane of the network device. With respect to the example ofFIG. 1, liveliness detection messages22output by router12B not only provide an indication to router12A of the operational status of the forwarding plane of router12B but have been enhanced to carry additional information to indicate any impending control plane disruption of the routing components of router12B. For example, receipt of liveliness detection messages22by router12A in accordance with the forwarding plane liveliness detection protocol provides an indication that the forwarding plane of router12B is operation and able to forward packets. In addition, any of liveliness detection messages22may include additional information to indicate an impending disruption of operation of the control plane of router12B, thereby providing an indication that router12A should expect an upcoming loss of communication with the control plane of router12B.

In this way, peer router12A or other devices, such as network controller35, may thus be quickly informed of an impending loss of communication with the control plane of router12B. As a result, peer router12A may adjust its own control plane operation so as to prevent various control plane protocols from triggering traffic re-route or other recovery processes upon detecting unreachability of the control plane of router12B. Further, the additional information embedded within liveliness detection messages22by the forwarding plane liveliness detection protocol of router12B may include an optional field indicating an expected duration for the impending control plane disruption. In some examples, a subsequent packet22may be output by router12B to affirmatively indicate an end to the control plane disruption period.

Leveraging a forwarding path liveliness detection protocol to convey in internal state of communications between the forwarding plane (e.g., a packet forwarding component) and a control plane (e.g., a routing component) of a router may provide certain advantages. For example, by providing such an indication, peer routers may, in response to liveliness detection messages22conveying an indication of an impending loss of control plane functionality and upon detecting an actual loss of communication with the control plane of the peer network device via periodic packets20, suppresses reroute of one or more packet flows around peer router12B. As another example, in response to receiving an indication of an impending loss of control plane functionality, peer network router12A may suppress a recovery process of one or more control plane protocols upon detecting loss of communication with the control plane of the peer network device. For example, router12A may execute ARP, OAM, ND or other control plane protocols, and router12A may configure any of the protocols to at least temporary suspend triggering a recovery process in response to loss of control plane communication with router12B.

Although described herein with reference to the BFD protocol as an example, the techniques may be applied with other protocols (e.g., layer two or layer three protocols) that execute in the forwarding plane or are otherwise delegated to the forwarding plane and that utilize period messages to indicate forwarding plane connectivity. Other example forwarding plane liveliness detection protocols that may be used could, for example, be OAM (Operations, Administration and Management) protocols that may be used to verify forwarding plane connectivity. Moreover, the techniques may be applied to instances where routers12employ the BFD protocol, for example, in conjunction with other control plane protocols, such as routing protocols like BGP, OSPF or IS-IS. In situations where multiple protocols are sharing one or more common BFD sessions, peer routers, and protocols executed by those protocols, can respond differently to an indication of an impending disruption to all or portions of the control plane. Furthermore, in the event no existing BDF sessions between certain pairs of routers12, BFD sessions may be established between the routers using topology information learned by other protocols executing in the control plane, such as IGP, BGP or EGP routing protocols. Once established, the BFD sessions may execute in forwarding components of the routers to exchange enhanced periodic messages as described herein.

In one example, routers12negotiate timeout values during an initial discover process. For example, upon initial discovery using ND or an Interior Gateway Protocol (IGP), routers12A and12B exchange time duration values, such as maximum suppression durations, for one or more types of potential control plane disruptions. For each potential control plane disruptions, routers12A and12B may agree upon or otherwise be pre-configured with timeout value. Moreover, such timeouts may be specified with respect to the entire control plane or individual control plane protocols. For example, routers12A and12B may negotiate indicators (codes) for indicating an impending disruption of specific control plane features as well as respective durations for suppressing any recovery or re-convergence process by the control plane protocol. One example of such negotiated indicators and time duration values is as follow:

In this way, peer routers may selectively suppress keep-alive requirements for specific control plane components. For example, in the event router12B issues a packet22having an indication of an impending disruption to the ARP protocol in the control plane (indicator code4), router12A may reconfigure the ARP protocol in its control plane to suppress triggering a recovery process in the event the ARP protocol loses communication with the ARP protocol on router12B. Moreover, other control plane components of router12A remain unaffected. As a result, a routing protocol executing in the control plane of router12A may still trigger a reroute of traffic in the event communication is lost with a routing protocol executing on router12B.

FIG. 2is a block diagram illustrating exemplary router30in accordance with the disclosure herein. Router30provides an example implementation of a router that may implement the techniques described herein. For example, router30may be of any of routers12illustrated inFIG. 1. While router30illustrates one possible router implementation to perform the techniques described herein, it will be appreciated that various other implementations are possible in accordance with this disclosure.

In this example, router30includes a control unit31that comprises a routing engine32that provide control plane functionality and a packet forwarding engine34that provides forwarding plane functionality. In addition, router30includes a set of interface cards (IFCs)50A-50N (collectively, “IFCs50”) for communicating packets via inbound links52A-52N (collectively, “inbound links52”) and outbound links54A-54N (collectively, “outbound links54”).

Routing engine32primarily provides an operating environment for execution of control plane protocols, such as those included in protocols40. For example, one or more routing protocols (“RP”)47maintains routing information36to reflect the current topology of a network and other network entities to which it is connected. In particular, RP47may communicate with protocols executing in the control plane of other routers to exchange topology information or others state information for a computer network and, based on the exchanged communication, update routing information36to accurately reflect the topology of the network and other entities. Example routing protocols include Multi-Protocol Border Gateway Protocol (mpBGP), the Intermediate System to Intermediate System (ISIS) routing protocol, the Open Shortest Path First (OSPF) routing protocol and the like.

As shown inFIG. 2, routing engine32and forwarding engine34internally communicate within router30via communication link33. As one example, internal communication link33may be a high-speed Ethernet-type network connection. In this example, each of forwarding engine34and routing engine32may incorporate network interface hardware and corresponding software providing a network stack so as to allow the components to communicate with each via one or more internal network communication sessions, such as Transmission Control Protocol (TCP) sessions.

Internal communication link33provides bi-directional communication between forwarding engine34and routing engine32. For example, inbound control-plane communications conveying topology information from other peer routers are received on interface cards50and forwarded to the routing engine via internal communication link33. In response, routing engine32processes the control plane packets to update routing information36, generates forwarding information in the control plane and, by way of the internal communication link33, programs the forwarding engine34with forwarding information38that associates network destinations with specific next hops and corresponding interface ports of IFCs50in accordance with routing information36. Routing engine32may generate forwarding information38in the form of a radix tree having leaf nodes that represent destinations within the network. In this way, internal communication link33allows routing engine32of the control plane to update forwarding information38within the forwarding plane in response to control-plane messages received from peer routers, thereby preventing the forwarding information from becoming stale.

Based on forwarding information38, forwarding engine34forwards packets received from inbound links52A-52N to outbound links54A-54N that correspond to next hops associated with destinations of the packets. U.S. Pat. No. 7,184,437 provides details on an exemplary embodiment of a router that utilizes a radix tree for route resolution, the contents of which is incorporated herein by reference in its entirety.

In one example, forwarding engine34is a rich and dynamic shared forwarding plane, optionally distributed over a multi-chassis router. Moreover, forwarding plane34may be provided by dedicated forwarding integrated circuits normally associated with high-end routing components of a network router. Further details of one example embodiment of router30can be found in U.S. Provisional Patent Application 61/054,692, filed May 20, 2008, entitled “STREAMLINED PACKET FORWARDING USING DYNAMIC FILTERS FOR ROUTING AND SECURITY IN A SHARED FORWARDING PLANE,” which is incorporated herein by reference.

As shown inFIG. 2, protocols40executing within routing engine32include one or more MPLS protocols for engineering traffic flows to flow along particular paths. For example, RSVP-TE45may generate and maintain a traffic engineering database49, including bandwidth reservations for paths associated with a network, such as service provider network20ofFIG. 1. Constrained Shortest Path First (CSPF) process61computes a shortest path or paths for a communication session based on specified constraints and bandwidth availability information associated with the links within the network. RP47may, in turn, advertise the calculated bandwidth availability information in TED49to other peer routers. In response to topology changes, RSVP-TE45and/or RP47may elect to select different paths in accordance with updated routing information36, generate updated forwarding information38and program the forwarding information into forwarding engine34by way of internal communication link33.

Moreover, as shown inFIG. 2, forwarding engine34includes BFD module39that is enhanced in accordance with the techniques described herein. In some cases, BFD module39of forwarding engine34may be configured by BFD module39′ or similar logic (not shown) executing in the control plane. Moreover, BFD module39may execute on any of IFCs50that utilize BFD protocol-based techniques to exchange BFD packets with peer devices to monitor the liveliness of the forwarding plane. Although described herein with reference to BFD, the techniques may be applied with other protocols (e.g., layer two or layer three protocols) that execute in the forwarding plane or are otherwise delegated to the forwarding plane and that utilize period/on-demand messages with peer devices to indicate forwarding plane connectivity. Examples of such protocols include Address Resolution Protocol (ARP), the Neighbor Discover (ND) Protocol. In these examples, enhanced BFD module39executing in the forwarding plane may be replaced with an enhanced ARP or ND forwarding-plane module.

In general, BFD module39implements BFD protocol-based functionalities, such as transmitting and monitoring for periodic BFD packets in the data-plane, thereby conserving resources that would otherwise be expended by routing engine32. In case of a detected connectivity failure, BFD module39may be configured to output a failure notification, or other similar indication. In some examples, forwarding engine34responds to the failure notification by triggering pre-programmed actions, such as rerouting traffic or initiating one or more timers. In other example, BFD module39outputs the failure indication to BFD module39′ or another component of routing engine32. In response to receiving the failure notification from BFD module39of forwarding engine34, BFD module39′ or another component of routing engine32causes RP47to update the network topology currently stored to routing information36, to reflect the failed link(s) represented by the BFD failure.

In accordance with the techniques described herein, BFD module39has been enhanced to generate outbound BFD packets so as to include additional information that provides an indication an impending disruption of operation of the control plane (e.g., routing engine32) of router32. For example, as a forwarding-plane liveliness detection protocol, outbound packets generated by BFD module39may provide other routers an indication that the forwarding plane of router32is operational and able to forward packets

In one example, BFD module39constructs the BFD packets to embed additional information based on a status signal51received from a component of routing engine32via internal communication interface53. For example, a component of routing engine32may output a command informing forwarding engine34of an impending loss of internal communication, such as a planned or recently initiated software upgrade to routing engine32. In any of these cases, BFD module39constructs and outputs BFD packets to include additional information that, although forwarding engine34may be able to continue to forward packets in accordance with forwarding information38, operation of routing engine32may subsequently become disrupted.

The architecture of router30illustrated inFIG. 2is shown for exemplary purposes only. In other embodiments, router30may be configured in a variety of ways. In one embodiment, for example, control unit31and its corresponding functionality may be distributed within IFCs50. Control unit31may be implemented solely in software, or hardware, or may be implemented as a combination of software, hardware, or firmware. For example, control unit31may include one or more processors which execute software instructions. In that case, the various software modules of control unit31, such as protocols40, may comprise executable instructions stored on a computer-readable medium, such as computer memory or hard disk.

FIG. 3is a block diagram illustrating further details of an example routing engine60and an example packet-forwarding engine82A of router having a plurality of forwarding engines82A-83N. The components ofFIG. 3may, for example, represent a more detailed example of any of routers12ofFIG. 1or router30ofFIG. 2.

In this example, routing engine60includes high-level, control plane software processes62. In this example, software processes include command-line interface daemon64(“CLI64”), routing protocol daemon66(“RPD66”), and Simple Network Management Protocol daemon68(“SNMP68”). In this respect, routing engine60may provide routing plane, service plane, and management plane functionality for the router. Various instances of routing engine60may include additional software processes62not shown inFIG. 3that perform other control, management, or service plane functionality and/or drive and otherwise manage data plane functionality for router30.

RPD66interacts with kernel72(e.g., by way of API calls) to update routing information base (RIB)74based on routing protocol messages received by router30. RPD66may, for example, execute various routing protocols, such as LDP and RSVP to establish LSPs within a network. RIB74may include information defining a topology of a network, including one or more routing tables and/or link-state databases. Kernel43executes on routing engine microprocessor78and may comprise, for example, a UNIX operating system derivative such as Linux or Berkeley Software Distribution (BSD). Kernel72processes kernel calls from RPD66and generates forwarding information in the form of FIBs76based on the network topology represented in RIB74, i.e., performs route resolution. Typically, RPD66generates FIB76in the form of radix or other lookup tree to map packet information (e.g., header information having destination information and/or a label stack) to next hops and ultimately to interface ports of interface cards associated with respective PFEs82. Routing engine microprocessor78of kernel72then communicates with communication interface105of PFE82A by way of communication interface101and internal communication link103. Routing engine microprocessor78may program PFEs82and install copies of the FIBs as software FIB86. Microprocessor78may comprise one or more general- or special-purpose processors such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other equivalent logic device. Accordingly, the terms “processor” or “controller,” as used herein, may refer to any one or more of the foregoing structures or any other structure operable to perform techniques described herein.

In this example, ASICs90are microcode-controlled chipsets programmably configured by a forwarding engine microprocessor84executing on each of PFEs82A-82N (e.g., PFE82A). Specifically, one or more of ASICs90is controllable by microcode92programmed by forwarding engine microprocessor84. The forwarding engine microprocessor84programs a hardware FIB96into internal memory of ASIC90within the data plane88based on software FIB86. When forwarding packets, control logic94traverses HW FIB96and, upon reaching a FIB entry for the packet (e.g., a leaf node), microcode-implemented control logic94automatically selects a forwarding next hop and processes the packets in accordance with the operations defined within the next hop. Additionally, microcode92includes BFD module98, which implements various functionalities in accordance the BFD protocol. As examples, BFD module98outputs enhanced BFD packets to embed status information as to communications between PFE82A and routing engine60.

Command line interface daemon64(“CLI64”) provides an interface by which an administrator or other management entity may modify the configuration of router30using text-based commands. Simple Network Management Protocol daemon68(“SNMP68”) comprises an SNMP agent that receives SNMP commands from a management entity to set and retrieve configuration and management information for router30. Using CLI64, SNMP68, and BFD70, management entities may enable/disable and configure services, install routes, enable/disable and configure rate limiters, and configure interfaces, for example. RPD66, CLI64, SNMP68, and BFD module70configure router30to implement configured services, add/modify/delete routes, and otherwise modify packet forwarding paths by installing forwarding structures to PFEs82.

In accordance with the techniques described herein, forwarding engine microprocessor84implements BFD control80. BFD control80, in turn, implements one or more of the techniques described herein. More specifically, microcode-implemented BFD module98is configured by BFD control80, responsive to signal107, to generate outbound BFD packets to include an indication of an impending control plane disruption using forwarding plane liveliness detection protocols routing engine60. Moreover, BFD module98may, when processing inbound BFD packets, extract information as to an impending control plane disruption of routing components of peer routers and provide such information to BFD control80for relaying to other control plane of routing engine60. For example, upon detecting via an inbound BFD packet that a forwarding component of a peer router remains operational but has been informed of or otherwise detected an upcoming disruption to its control plane, any of control logic94of forwarding ASICs90, forwarding engine microprocessor84, and/or components of routing engine60may alter operation. For example, RPD66may, upon subsequently detecting loss of communication with the control plane of the peer router, suppress reroute of traffic around the router, at least until expiration of one or more timers. As other examples, other protocols62may suppress or otherwise delay triggering a recovery process.

FIG. 4is a diagram illustrating an exemplary BFD control message138that may be sent by a packet forwarding component of a router, e.g., router12B ofFIG. 1, with respect to liveliness detection of the forwarding plane within a network. In this example, control message138complies with the BFD protocol, such as BFD28D ofFIG. 2A, so that only certain changes need be implemented to the protocols executing on the sending or receiving forwarding components of the routers. Control message138therefore may be referred to below as “BFD control message138.” BFD control message138may further be formulated as a packet and receive the moniker of “BFD control packet138.” Again, although described in reference to the BFD protocol and BFD control message138in particular, the techniques may apply to any protocol that uses periodic messages to detect liveliness of the forwarding plane and that may be modified, as described herein, to also provide a status of internal communications between the forwarding plane and the control plane of the sending router.

In the example shown inFIG. 4, BFD control message138comprises 6 rows of 32 bits (0-31 across the top of BFD control message138inFIG. 3). BFD control message138further comprises fields140A-40K (“fields140”), where each of fields140may specify information relevant to specifying a response interval. Version field140A, for example, specifies the version number of the protocol, which may enable the receiving router to determine whether it supports BFD control message138. Diagnostic field140B specifies a coded reason for the last session state change to states of “Down” or “AdminDown.” State field140C specifies the current BFD session state as seen by the router transmitting BFD control message138. Typical, BFD session states comprises “AdminDown,” “Down,” “Init,” and “Up.” Modes field140D comprises a number of bits, where each bit specifies a particular BFD session mode. For example, one bit of modes field140D may specify whether the session requires authentication, e.g., whether the session is in “authentication mode.”

Detection timer multiplier field140E specifies a value that when multiplied by the value specified within desired minimum transfer interval140I provides the detection time for the router transmitting BFD control message138in “asynchronous mode.” Length field140F specifies the length of BFD control message138in bytes. My discriminator field140G specifies a unique, nonzero discriminator value generated by the router transmitting BFD control message138. My discriminator field140G may be used to demultiplex multiple BFD sessions between the same set of receiving and transmitting routers. Your discriminator field140H specifies the discriminator received from the corresponding router of the particular BFD session. Your discriminator field140H may specify a zero value, if the discriminator of received from the corresponding router is unknown by the transmitting router.

Desired minimum transfer interval field140I specifies the minimum interval, in microseconds, that the local system would like to use when transmitting BFD control message138and subsequent BFD control messages. Required minimum receive interval field140J specifies the minimum interval, in microseconds, between received BFD control messages that the router transmitting BFD control message138is capable of supporting. If the transmitting router sets required minimum receive interval field140J to zero, the transmitting router does not want the remote or receiving router to send any periodic BFD control messages. Required minimum echo receive interval field140K specifies the minimum interval, in microseconds, between received BFD echo messages that the transmitting router is capable of supporting. Specifying a zero to this field indicates that the transmitting router does not support the receipt of BFD echo packets.

A router, such as any of the routers ofFIGS. 1-3, may therefore generate and output BFD control message138. In general, the techniques described herein may use any mechanism for embedding information within BFD control message138to convey a status of internal communications between a forwarding plane and a control plane of the sending router, thereby providing an indication as to whether the forwarding information used by the forwarding plane is current or stale. In one example, the transmitting router utilizes one or more bits or sets one or more codes within diagnostic field140B to convey this additional information. For example, BFD diagnostic codes may be defined as follow:

TABLE 2CODEWORDMEANING0No Diagnostic1Control Detection Time Expired2Echo Function Failed3Neighbor Signaled Session Down4Forwarding Plane Reset5Path Down6Concatenated Path Down7Administratively Down8Reverse Concatenated Path Down9Planned Graceful Restart of theControl Plane10Unplanned Graceful Restart of theControl Plane11Warm restart of the Control Plane12Planned ARP disruption within theControl Plane13Planned ND disruption within theControl Plane14Planned Routing Protocol (RPD)disruption within the Control Plane.
In the above example, an additional code word ‘9’ has been defined for diagnostic field140B to indicate an impending disruption in the control plane.

As another example, BFD message139may include an additional type-length-value (TLV) field so as to include the additional information. Moreover, BFD message139may further include an optional field indicating an expected duration of the impending event. For further information concerning the BFD protocol generally, BFD control message, and BFD control message fields, see the Internet Draft published by the Network Working Group of the Internet Engineering Task Force (IETF), titled “Bidirectional Forwarding Detection,” written by D. Katz and D. Ward, and dated March, 2007, the entire contents of which are hereby incorporated by reference as if fully set forth herein.

FIG. 5is a flowchart illustrating example operation of two routers in accordance with the techniques described here. For purposes of example, a first one of the routers is referred to as the “sending router” and a second one of the routers is referred to as a “receiving router.”

In operation, control plane components of the routers receive control plane routing protocol messages from other routers and, in response, update topology information and perform route selection (202,212).

In addition, the packet forwarding components of the routers establish a communication session in accordance with the forwarding-plane liveliness detection protocol, such as BFD (204,214). The forwarding plane liveliness detection protocol may, for example, comprise a bidirectional forwarding detection (BFD) protocol. At this time, the two packet forwarding components of the routers begin exchanging liveliness detection messages that, when received by the other router, provide an indication that the packet forwarding engines are operational and able to forward packets.

In the example ofFIG. 5, the packet forwarding component of the sending router constructs a liveliness detection message in accordance with the forwarding-plane liveliness detection protocol (206). At this time, the packet forwarding component of the sending router may be informed of, or otherwise detect, an impending disruption of operation of its control plane (208). As explained herein, an impending disruption of operation indicates that operation of the control plane may in the near future be compromised such that at that point control-plane communication with peer routers or controllers may be lost. For example, the routing engine(s) of the router may be undergoing a planned in-service software upgrade (ISSU), where loss of communication between the routing components and the peer routers may be compromised.

In the event the packet forwarding component determines that loss of control plane operation is impending (YES OF209), the packet forwarding component includes within the liveliness detection message an indicator of such impending disruption (210). If no fault condition is detected, the packet forwarding engine does not include the indicator within the liveliness detection message.

After construction, the packet forwarding component of the sending router outputs the liveliness detection message to a peer network device in accordance with the forwarding plane liveliness detection protocol (211). The packet forwarding component of the sending router repeats this transmission at a high rate, such as every 1-10 ms to provide an indication of the operational status of the forwarding plane of the sending router.

Receipt of the liveliness detection message by the packet forwarding component of the receiving router confirms to the receiving router that the packet forwarding component of the sending router is currently able to forward packets (216). In addition, the packet forwarding component of the receiving router further processes the received liveliness detection message to detect whether the packet forwarding component of the packet forwarding engine has determined that the control plane of the sending router has been marked for impending operational disruption (217).

If no impending loss of control plane functionality has been indicated in the inbound liveliness detection message, the receiving router continues to forward traffic according to normal operation (218).

However, if the liveliness detection message indicates an impending loss of control plane operation for the sending router, the receiving router may reconfigure its control place and/or forwarding plane to at least temporarily suppress certain actions, such as rerouting of packet flows or control-plane recovery processes in the event control-plane communications are subsequently lost with the sending router (220).