Patent Description:
In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing (e.g., compute nodes) and/or storage capacity to run various applications. For example, a data center comprises a facility that hosts applications and services for subscribers, i.e., customers of data center. The data center for example, hosts all of the infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via high-speed switch fabric provided by one or more tiers of physical network switches and routers. More sophisticated data centers provide infrastructure spread throughout the world with subscriber support equipment located in various physical hosting facilities.

Periodically, the physical network switches and routers of the high-speed switch fabric may need to be updated. For example, software installed and running on network devices within the data center may need to be either upgraded or replaced. Conventionally, installing the software update typically requires the network device to transition offline and stop execution or management of network traffic so that the software update may be performed. During this period, customer traffic flow and customer applications executing within the cloud data center may be interrupted or entirely unavailable.

<CIT> relates to 'In-service software upgrade of software-defined networking controller'. <CIT> relates to 'Intelligent non-stop software upgrade'.

In general, the disclosure describes techniques for deploying software upgrades to a mixed network of In-Service Software Upgrade (ISSU)-capable and ISSU-incapable network devices without interrupting network traffic serviced by the mixed network. In one example, a centralized controller of a network, such as a Software-Defined Networking (SDN) controller or an administrator computing device, determines which network devices of the network are ISSU-capable and which network devices are ISSU-incapable. The centralized controller instructs the network devices that are ISSU-capable to perform an ISSU operation, e.g. to apply a software update. Further, the centralized controller prompts the network devices that are ISSU-incapable to transmit messages to neighbor network devices, such as the upgraded network devices. The messages indicate that the network devices are ISSU-incapable. The receiving network devices may process such messages to determine that the ISSU-incapable network devices may be unavailable to process or forward network traffic while performing the software upgrade. In response to determining that the ISSU-incapable devices may (or will) be unavailable, the peer network devices redirect traffic to avoid forwarding the traffic to the ISSU-incapable network devices so as to avoid interruption of network traffic across the network.

The techniques of the disclosure provide specific technical improvements to the field of software maintenance of computer networks, particularly cloud networks and data centers that include large numbers of network devices in complex arrangements. For example, the techniques of the disclosure enable deployment of software upgrades to a mixed network of ISSU-capable and ISSU-incapable network devices without interrupting network traffic serviced by the mixed network. While an administrator may still need to upgrade ISSU-incapable network devices manually, the administrator is able to take advantage of the ISSU capabilities of ISSU-capable network devices even in a mixed network of ISSU-capable and ISSU-incapable network devices, thereby substantially reducing the burden of deploying software upgrades to a mixed network. Furthermore, the techniques of the disclosure provide a mechanism for ISSU-incapable network devices to notify peer network devices that they are ISSU-incapable, thereby allowing the peer network devices to temporarily route traffic around the ISSU-incapable network devices while the ISSU-incapable network devices are unable to process or forward network traffic. Thus, the techniques of the disclosure allow ISSU-incapable network devices to avoid causing interruptions to customer traffic and customer applications serviced by the network while performing software upgrades.

In one example, this disclosure describes a method comprising: determining, by a centralized controller of a network, that first network devices of a plurality of network devices for the network are In-Service Software Upgrade (ISSU)-capable and second network devices of the plurality of network devices are not ISSU-capable; transmitting, by the centralized controller, one or more messages instructing the first network devices to perform an ISSU operation; and transmitting, by the centralized controller, one or more messages instructing each network device of the second network devices to transmit messages to neighbor network devices, wherein the messages: indicate that each network device of the second network devices is not ISSU-capable; and directs the neighbor network devices to redirect traffic to avoid forwarding the traffic to each network device of the second network devices.

In another example, this disclosure describes a centralized controller of a network configured to: determine that first network devices of a plurality of network devices for the network are In-Service Software Upgrade (ISSU)-capable and second network devices of the plurality of network devices are not ISSU-capable; transmit messages instructing the first network devices to perform an ISSU operation; and transmit messages instructing each network device of the second network devices to transmit a message to neighbor network devices of the network device, wherein the message: indicates that each network device is not ISSU-capable; and directs the neighbor network devices to redirect traffic to avoid forwarding the traffic to each network device of the second network devices.

In another example, this disclosure describes a first network device of a plurality of network devices of a network, the first network device configured to: receive, from a second network device, a message indicating that the second network device is not In-Service Software Upgrade (ISSU)-capable, wherein the second network device is a neighbor of the first network device; and in response to receiving the message indicating that the second network device is not ISSU-capable, redirecting traffic to avoid forwarding the traffic to the second network device.

The details of one or more examples of the techniques of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques will be apparent from the description and drawings, and from the claims.

<FIG> is a block diagram illustrating an example network system in which examples of the techniques described herein may be implemented. Network system <NUM> in the example of <FIG> includes data center <NUM> interconnected with customer networks associated with customers <NUM> via a service provider network <NUM>. In general, data center <NUM> provides an operating environment for applications and services for customers <NUM> coupled to the data center by service provider network <NUM>. Data center <NUM> may, for example, host infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. Service provider network <NUM> may be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet.

In some examples, data center <NUM> may represent one of many geographically distributed network data centers. As illustrated in the example of <FIG>, data center <NUM> may be a facility that provides network services for customers <NUM>. Customers <NUM> may be collective categories such as enterprises and governments or individuals. For example, a network data center may host web services for several enterprises and end users. Other example services may include data storage, virtual private networks, traffic engineering, file service, data mining, scientific- or super-computing, and so on. In some embodiments, data center <NUM> may be individual network servers, network peers, or otherwise.

In this example, data center <NUM> includes a set of storage systems and application servers 12A-12X (herein, "servers <NUM>") interconnected via high-speed switch fabric <NUM> provided by one or more tiers of physical network switches and routers. Switch fabric <NUM> is provided by a set of interconnected top-of-rack (TOR) switches 16A-16BN (collectively, "TOR switches <NUM>") coupled to a distribution layer of chassis switches 18A-<NUM> (collectively, "chassis switches <NUM>"). Although not shown, data center <NUM> may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.

In this example, TOR switches <NUM> and chassis switches <NUM> provide servers <NUM> with redundant (multi-homed) connectivity to IP fabric <NUM> and service provider network <NUM>. Chassis switches <NUM> aggregate traffic flows and provides high-speed connectivity between TOR switches <NUM>. TOR switches <NUM> may be network devices that provide layer two (e.g., MAC) and/or layer <NUM> (e.g., IP) routing and/or switching functionality. TOR switches <NUM> and chassis switches <NUM> may each include one or more processors and a memory, and that are capable of executing one or more software processes. Chassis switches <NUM> are coupled to IP fabric <NUM>, which performs layer <NUM> routing to route network traffic between data center <NUM> and customers <NUM> by service provider network <NUM>.

SDN controller <NUM> provides a logically and in some cases physically centralized controller for facilitating operation of one or more virtual networks within data center <NUM>, such as data center 10A, in accordance with one or more embodiments of this disclosure. As described herein, the terms SDN controller and Virtual Networking Controller (VNC) may be used interchangeably. In some examples, SDN controller <NUM> may operate in response to configuration input received from network administrator <NUM>.

In some examples, SDN controller <NUM> operates in response to configuration input received from orchestration engine <NUM>, which in turn operates in response to configuration input received from administrator <NUM>. Additional information regarding SDN controller <NUM> operating in conjunction with other devices of data center 10A or other software-defined network is found in <CIT>, and entitled PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS.

In some examples, orchestration engine <NUM> manages functions of data center <NUM> such as compute, storage, networking, and application resources. For example, orchestration engine <NUM> may create a virtual network for a tenant within data center <NUM> or across data centers. Orchestration engine <NUM> may attach virtual machines (VMs) to a tenant's virtual network. Orchestration engine <NUM> may connect a tenant's virtual network to some external network, e.g. the Internet or a VPN. Orchestration engine <NUM> may implement a security policy across a group of VMs or to the boundary of a tenant's network. Orchestration engine <NUM> may deploy a network service (e.g. a load balancer) in a tenant's virtual network.

In some examples, SDN controller <NUM> manages the network and networking services such load balancing, security, and allocate resources from servers <NUM> to various applications. For example, SDN controller <NUM> implements high-level requests from orchestration engine <NUM> by configuring physical switches, e.g. TOR switches <NUM>, chassis switches <NUM>, and switch fabric <NUM>; physical routers; physical service nodes such as firewalls and load balancers; and virtual services such as virtual firewalls in a VM. SDN controller <NUM> maintains routing, networking, and configuration information within a state database.

Typically, the traffic between any two network devices, such as between network devices within IP fabric <NUM> (not shown) or between servers <NUM> and customers <NUM> or between servers <NUM>, for example, can traverse the physical network using many different paths. A packet flow (or "flow") can be defined by the five values used in a header of a packet, or "five-tuple," i.e., the protocol, Source IP address, Destination IP address, Source port and Destination port that are used to route packets through the physical network. For example, the protocol specifies the communications protocol, such as TCP or UDP, and Source port and Destination port refer to source and destination ports of the connection. A set of one or more packet data units (PDUs) that match a particular flow entry represent a flow. Flows may be broadly classified using any parameter of a PDU, such as source and destination data link (e.g., MAC) and network (e.g., IP) addresses, a Virtual Local Area Network (VLAN) tag, transport layer information, a Multiprotocol Label Switching (MPLS) or Generalized MPLS (GMPLS) label, and an ingress port of a network device receiving the flow. For example, a flow may be all PDUs transmitted in a Transmission Control Protocol (TCP) connection, all PDUs sourced by a particular MAC address or IP address, all PDUs having the same VLAN tag, or all PDUs received at the same switch port.

From time to time, an administrator may desire to perform a software upgrade on one or more network devices <NUM>, <NUM> that make up switch fabric <NUM>. Conventionally, the administrator may have to manually install the software upgrade to each individual network device <NUM>, <NUM>, which is time consuming and a large administrative burden on data center <NUM>. Further, during this period, the device is transitioned offline and is unable to process customer traffic, causing interruptions to customer applications executing within the cloud data center. Some network devices are ISSU-capable in that they are able to perform such software upgrades without transitioning offline or interrupting customer traffic flow.

Some network devices are In-Service Software Upgrade (ISSU)-capable in that they are able to perform such software upgrades without transitioning offline or interrupting customer traffic flow. Some data centers may include a mixture of ISSU-capable and ISSU-incapable devices. For example, as data center providers expand and upgrade their infrastructure, they commonly implement a "mixed network" that includes newer, ISSU-capable devices as well as legacy, ISSU-incapable devices. However, the use of such a mixed network may prevent an administrator of the data center from deploying an ISSU software upgrade to all of the network devices in the mixed network because such an operation may cause the ISSU-incapable devices, which may make up a substantial portion of switch fabric <NUM>, to transition offline together, causing interruptions to customer traffic flow and applications. Thus, for a mixed network, an administrator may be unable to take advantage of the ISSU capabilities offered by individual ISSU-capable devices and still be forced to manually upgrade each of the network devices <NUM>, <NUM> in switch fabric <NUM> to prevent service disruption. Further, manually upgrading each network device <NUM>, <NUM> is cumbersome and a large administrative burden for data centers with large numbers of devices, as well as prone to error.

In accordance with the techniques of the disclosure, methods, devices, and systems are disclosed for deploying software upgrades to a mixed network of ISSU-capable and ISSU-incapable network devices <NUM>, <NUM> without interrupting network traffic serviced by the mixed network. In one example, a centralized controller, such as SDN controller <NUM> or an administrator computing device <NUM> of cloud data center <NUM>, determines which network devices <NUM>, <NUM> of cloud data center <NUM> are ISSU-capable and which network devices <NUM>, <NUM> are ISSU-incapable. For example, SDN controller <NUM> sends a message instructing network devices <NUM>, <NUM> that are ISSU-capable to perform an ISSU operation, e.g. to apply a software update. Further, SDN controller <NUM> sends a message prompting network devices <NUM>, <NUM> that are ISSU-incapable to transmit messages to peer network devices <NUM>, <NUM>. The messages indicate that the ISSU-incapable network devices are ISSU-incapable. In some examples, the use of "peer" network devices <NUM>, <NUM> refers to network devices <NUM>, <NUM> that neighbor a network device <NUM>, <NUM>, such as those network devices <NUM>, <NUM> that are BGP peers of the network device <NUM>, <NUM>. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-capable and have completed the ISSU operation. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-incapable but are not currently undergoing a software update. The peer network devices <NUM>, <NUM> that receive such messages may process the messages to determine that the ISSU-incapable network devices <NUM>, <NUM> may be unavailable to process or forward network traffic while performing the software upgrade. In response to determining that the ISSU-incapable devices <NUM>, <NUM> may (or will) be unavailable, the peer network devices <NUM>, <NUM> redirect traffic to avoid forwarding the traffic to the ISSU-incapable network devices <NUM>, <NUM> so as to avoid interruption of network traffic across, e.g., switch fabric <NUM>.

The techniques of the disclosure provide for deploying software upgrades to a mixed network of ISSU-capable and ISSU-incapable network devices <NUM>, <NUM> without interrupting network traffic serviced by the mixed network. While an administrator may still need to upgrade ISSU-incapable network devices <NUM>, <NUM> manually, the administrator is able to take advantage of the ISSU capabilities of ISSU-capable network devices <NUM>, <NUM>, even in a mixed network of ISSU-capable and ISSU-incapable network devices <NUM>, <NUM>, thereby substantially reducing the burden of deploying software upgrades to a mixed network. Furthermore, the techniques of the disclosure provide a mechanism for ISSU-incapable network devices <NUM>, <NUM> to notify peer network devices <NUM>, <NUM> in switch fabric <NUM> that the ISSU-incapable network devices <NUM>, <NUM> will go offline for an upgrade, thereby allowing the peer network devices <NUM>, <NUM> to temporarily route traffic around the ISSU-incapable network devices <NUM>, <NUM> while the ISSU-incapable network devices are unable to process or forward network traffic (e.g., such as during a software upgrade). Thus, the techniques of the disclosure allow ISSU-incapable network devices <NUM>, <NUM> to avoid causing interruptions to customer traffic and customer applications serviced by data center <NUM> while performing software upgrades. In this way, the techniques allow for scalable deployment of software upgrades and reduce the complexity and simplify management of such software upgrades to mixed networks of ISSU-capable and ISSU-incapable network devices <NUM>, <NUM>.

In some examples, an extension of Border Gateway Protocol (BGP) is described for communicating whether a device is ISSU-capable or ISSU-incapable. For example, a network device <NUM>, <NUM> may indicate its ISSU capability via BGP. Such a BGP message may specify the ISSU capability of network device <NUM>, <NUM>, such as whether network device <NUM>, <NUM> is "ISSU-CAPABLE" or "ISSU-NOT-CAPABLE," as an extended community for BGP. Additional information with respect to the implementation of BGP extended communities is described in "<NPL>. A network device <NUM>, <NUM> may use the BGP message to indicate its ISSU capability to, e.g., peer network devices <NUM>, <NUM> or to SDN controller <NUM>. Further information with respect to BGP may be found in "<NPL> and "<NPL>.

<FIG> is a block diagram illustrating an example implementation of data center 10A of <FIG> in further detail. In the example of <FIG>, data center 10A includes an overlay network that extends switch fabric <NUM> from physical switches <NUM>, <NUM> to software or "virtual" switches 30A-30X (collectively, "virtual routers <NUM>"). Virtual routers <NUM> dynamically create and manage one or more virtual networks <NUM> usable for communication between application instances. In one example, virtual routers <NUM> execute the virtual network as an overlay network, which provides the capability to decouple an application's virtual address from a physical address (e.g., IP address) of the one of servers 12A-12X ("servers <NUM>") on which the application is executing. Each virtual network may use its own addressing and security scheme and may be viewed as orthogonal from the physical network and its addressing scheme. Various techniques may be used to transport packets within and across virtual networks <NUM> over the physical network. In some examples, the techniques described in this disclosure provide multicast service within virtual networks <NUM> without requiring multicast support in the underlying physical network.

Each virtual router <NUM> may execute within a hypervisor, a host operating system or other component of each of servers <NUM>. Each of servers <NUM> may represent an x86 or other general-purpose or special-purpose server capable of executing virtual machines <NUM>. In the example of <FIG>, virtual router 30A executes within hypervisor <NUM>, also often referred to as a virtual machine manager (VMM), which provides a virtualization platform that allows multiple operating systems to concurrently run on one of servers <NUM>. In the example of <FIG>, virtual router 30A manages virtual networks <NUM>, each of which provides a network environment for execution of one or more virtual machines (VMs) <NUM> on top of the virtualization platform provided by hypervisor <NUM>. Each VM <NUM> is associated with one of the virtual networks VN0-VN1 and may represent tenant VMs running customer applications such as Web servers, database servers, enterprise applications, or hosting virtualized services used to create service chains. In some cases, any one or more of servers <NUM> or another computing device may host customer applications directly, i.e., not as virtual machines. In some cases, some of VMs <NUM> may represent containers, another form of virtualized execution environment. That is, both virtual machines and container are examples of virtualized execution environments for executing workloads.

In general, each VM <NUM> may be any type of software application and may be assigned a virtual address for use within a corresponding virtual network <NUM>, where each of the virtual networks may be a different virtual subnet provided by virtual router 30A. A VM <NUM> may be assigned its own virtual layer three (L3) IP address, for example, for sending and receiving communications but may be unaware of an IP address of the physical server 12A on which the virtual machine is executing. In this way, a "virtual address" is an address for an application that differs from the logical address for the underlying, physical computer system, e.g., server 12A in the example of <FIG> or <FIG>.

In one implementation, each of servers <NUM> includes a corresponding one of virtual network (VN) agents 35A-35X (collectively, "VN agents <NUM>") that controls the overlay of virtual networks <NUM> and that coordinates the routing of data packets within server <NUM>. In general, each VN agent <NUM> communicates with SDN controller <NUM>, which generates commands to control routing of packets through data center 10A. VN agents <NUM> may operate as a proxy for control plane messages between virtual machines <NUM> and SDN controller <NUM>. For example, a VM <NUM> may request to send a message using its virtual address via the VN agent 35A, and VN agent 35A may in turn send the message and request that a response to the message be received for the virtual address of the VM <NUM> that originated the first message. In some cases, a VM <NUM> may invoke a procedure or function call presented by an application programming interface of VN agent 35A, and the VN agent 35A may handle encapsulation of the message as well, including addressing.

In some example implementations, each server <NUM> further includes an orchestration agent (not shown in <FIG>) that communicates directly with orchestration engine <NUM>. For example, responsive to instructions from orchestration engine <NUM>, the orchestration agent communicates attributes of the particular VMs <NUM> executing on the respective server <NUM>, and may create or terminate individual VMs.

In one example, network packets, e.g., layer three (L3) IP packets or layer two (L2) Ethernet packets generated or consumed by the instances of applications executed by virtual machines <NUM> within the virtual network domain may be encapsulated in another packet (e.g., another IP or Ethernet packet) that is transported by the physical network. The packet transported in a virtual network may be referred to herein as an "inner packet" while the physical network packet may be referred to herein as an "outer packet" or a "tunnel packet. " Encapsulation and/or de-capsulation of virtual network packets within physical network packets may be performed within virtual routers <NUM>, e.g., within the hypervisor or the host operating system running on each of servers <NUM>. As another example, encapsulation and de-capsulation functions may be performed at the edge of switch fabric <NUM> at a first-hop TOR switch <NUM> that is one hop removed from the application instance that originated the packet. This functionality is referred to herein as tunneling and may be used within data center 10A to create one or more overlay networks. Besides IPinIP, other example tunneling protocols that may be used include IP over GRE, VxLAN, MPLS over GRE, MPLS over UDP, etc..

As noted above, SDN controller <NUM> provides a logically centralized controller for facilitating operation of one or more virtual networks within data center 10A. SDN controller <NUM> may, for example, maintain a routing information base, e.g., one or more routing tables that store routing information for the physical network as well as one or more overlay networks of data center 10A. Similarly, switches <NUM>, <NUM> and virtual routers <NUM> maintain routing information, such as one or more routing and/or forwarding tables. In one example implementation, virtual router 30A of hypervisor <NUM> implements a network forwarding table (NFT) <NUM> for each virtual network <NUM>. In general, each NFT <NUM> stores forwarding information for the corresponding virtual network <NUM> and identifies where data packets are to be forwarded and whether the packets are to be encapsulated in a tunneling protocol, such as with a tunnel header that may include one or more headers for different layers of the virtual network protocol stack.

In accordance with the techniques of the disclosure, a software upgrade is deployed to a mixed network of ISSU-capable and ISSU-incapable network devices <NUM>, <NUM> without interrupting network traffic serviced by the mixed network. In one example, SDN controller <NUM> of cloud data center <NUM> determines which network devices <NUM>, <NUM> of cloud data center <NUM> are ISSU-capable and which network devices <NUM>, <NUM> are ISSU-incapable. SDN controller <NUM> may, for example, request from each network device <NUM>, <NUM>, configuration information for the network device <NUM>, <NUM>. Such configuration information may include, e.g., a device model or a current software version of the network device <NUM>, <NUM>. SDN controller <NUM> may receive the configuration information and compare the configuration information to a reference table that describes ISSU capabilities for a plurality of different types of network devices <NUM>, <NUM>. In the example of <FIG>, SDN controller <NUM> determines that chassis switch 18A and TOR switch 16A are ISSU-capable, while chassis switch 18B and TOR switch 16B are ISSU-incapable. In some examples, SDN controller <NUM> stores, for each network device <NUM>, <NUM>, an indication of whether the network device <NUM>, <NUM> is ISSU-capable or ISSU-incapable.

In some examples, SDN controller <NUM> may include a policy controller (not depicted) which is configured to tag network devices <NUM>, <NUM> with ISSU-capability "tags. " As described herein, a "tag" refers to a data structure that provides information to categorize an object according to a particular value or set of values. In some examples, the tag comprises a key / value pair, wherein the key describes a category of the object, and the value provides a descriptor for the category. As an example, where tag comprises a first key / value pair, the key of the tag is an "ISSU capability," and the value associated with the key is a "ISSU-capable" or "ISSU-NOT-CAPABLE" capability. Furthermore, as used herein, "tagging" an object refers to categorizing the object in a category specified by a tag included in a policy rule. For example, SDN controller <NUM> may use the policy controller to tag chassis switch 18A and TOR switch 16A with "ISSU-CAPABLE" tags <NUM> and chassis switch 18B and TOR switch 16B with "ISSU-NOT-CAPABLE" tags <NUM>. The use of such tags may allow SDN controller <NUM> to distribute, via the policy controller and based on the applied tags, network policies for application to network devices <NUM>, <NUM>. Such policies may include, for example, deployment of ISSU operations to apply software updates. Further explanation with regard to the implementation of a policy controller and the application of tags to infrastructure equipment is provided in <CIT>.

SDN controller <NUM> transmits messages to ISSU-capable network devices <NUM>, <NUM> (e.g., chassis switch 18A and TOR switch 16A) instructing chassis switch 18A and TOR switch 16A to perform an ISSU operation. In some examples, SDN controller <NUM> may perform a "single trigger" operation in that SDN controller <NUM> may receive, from a user or administrator, a command to perform an ISSU operation on ISSU-capable network devices <NUM>, <NUM>. In response to the command, SDN controller <NUM> transmits one or more messages instructing each of the ISSU-capable network devices <NUM>, <NUM> in switch fabric <NUM> to initiate the ISSU operation substantially simultaneously with a period of time that each other of the ISSU-capable network devices <NUM>, <NUM> initiates the ISSU operation.

In some examples, the message instructing the ISSU-capable devices to perform the ISU may include one or more software packages for the ISSU-capable network devices <NUM>, <NUM> to install. In other examples, the message may specify a location from which the ISSU-capable network devices <NUM>, <NUM> may retrieve the one or more software packages for the ISSU-capable network devices <NUM>, <NUM> to install. In some examples, upon successfully completing an ISSU operation, the ISSU-capable network devices <NUM>, <NUM> notify SDN controller <NUM> of the completion of the ISSU operation.

If the ISSU operation is not successful, one or more of the ISSU-capable network devices <NUM>, <NUM> notify SDN controller <NUM> of the failure of the ISSU operation. In response, SDN controller <NUM> may, for example, instruct the failed network devices <NUM>, <NUM> to attempt to retry the ISSU, to abort the ISSU, provide logging information describing the failed ISSU, notify an administrator of the failure, or take some other action. In some examples, SDN controller <NUM> may re-classify the failed network devices <NUM>, <NUM> as ISSU-incapable such that a manual upgrade is performed on the failed network devices <NUM>, <NUM>.

Further, SDN controller <NUM> transmits messages to ISSU-incapable network devices <NUM>, <NUM> (e.g., chassis switch 18B and TOR switch 16B) prompting ISSU-incapable network devices <NUM>, <NUM> to transmit messages to peer network devices <NUM>, <NUM>, such as chassis switch 18A and TOR switch 16A. The messages indicate that the ISSU-incapable network devices are ISSU-incapable. In some examples, each ISSU-incapable network device <NUM>, <NUM> sends a message to each BGP peer of the ISSU-incapable network device. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-capable and have completed the ISSU operation. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-incapable but are not undergoing a software update at the present time (e.g., have already completed the software update or are scheduled to do so at a later time).

In some examples, the ISSU-incapable network device <NUM>, <NUM> may transmit such messages via BGP. In some examples, the BGP message specifies the ISSU capability of network device <NUM>, <NUM>, such as whether network device <NUM>, <NUM> is "ISSU-CAPABLE" or "ISSU-NOT-CAPABLE," as an extended community for BGP. For example, each ISSU-incapable network device <NUM>, <NUM> may send a BGP message to peer network devices <NUM>, <NUM> that specifies that the sending network device <NUM>, <NUM> is ISSU-NOT-CAPABLE as an extended community for BGP.

The peer network devices <NUM>, <NUM> that receive such messages may process the messages to determine that the ISSU-incapable network devices <NUM>, <NUM> may be unavailable to process or forward network traffic while performing the software upgrade. In response to determining that the ISSU-incapable devices <NUM>, <NUM> may (or will) be unavailable, the peer network devices <NUM>, <NUM> redirect traffic to avoid forwarding the traffic to the ISSU-incapable network devices <NUM>, <NUM> so as to avoid interruption of network traffic across, e.g., switch fabric <NUM>. In some examples, the peer network device <NUM>, <NUM> may delete an entry describing the ISSU-incapable network device <NUM>, <NUM> from a Routing Information Base (RIB) of the peer network device <NUM>, <NUM> so as to avoid forwarding traffic to the ISSU-incapable network device <NUM>, <NUM> while the ISSU-incapable network device <NUM>, <NUM> is unavailable to process or forward traffic because the ISSU-incapable network device <NUM>, <NUM> is performing the upgrade.

In some examples, after the ISSU-incapable network device <NUM>, <NUM> has completed the software upgrade, the ISSU-incapable network device <NUM>, <NUM> may inform peer network devices <NUM>, <NUM> that the ISSU-incapable network device <NUM>, <NUM> is online and capable of processing network traffic. This may be done through the use of conventional BGP messages as described in "BGP MPLS-Based Ethernet VPN," RFC <NUM>, referenced above.

<FIG> is a block diagram illustrating an example implementation of the Software-defined Networking (SDN) controller of <FIG> in further detail. In the example of <FIG>, SDN controller <NUM> includes one or more analytic nodes 50A-50X (collectively, "analytic nodes <NUM>"), one or more configuration nodes 52A-52X (collectively, "configuration nodes <NUM>") and control nodes 54A-54X (collectively, "control nodes <NUM>"). In general, each of the nodes <NUM>, <NUM>, and <NUM> may be implemented as a separate software process, and the nodes may be distributed across multiple hardware computing platforms that provide an environment for execution of the software. Moreover, each of the nodes maintains state data <NUM>, which may be stored within a centralized or distributed database. In some examples, state database <NUM> is a NoSQL database. In some examples, state database <NUM> is a database cluster.

In general, analytic nodes <NUM> are tasked with collecting, storing, correlating, and analyzing information from virtual and physical network elements within data center <NUM>. This information may include statistics, logs, events, and errors for use in managing the routing and network configuration of data center <NUM>. Analytic nodes <NUM> store this information in state database <NUM>.

Configuration nodes <NUM> translate the high-level data model of orchestration engine <NUM> into lower level models suitable for interacting with network elements, such as physical switches <NUM>, <NUM> and VR agents <NUM>. Configuration nodes <NUM> keep a persistent copy of the configuration state of SDN controller <NUM> within state database <NUM>.

Control nodes <NUM> implement a logically centralized control plane responsible for maintaining ephemeral network state. Control nodes <NUM> interact with each other and with network elements, such as virtual routers <NUM> of servers <NUM>, to ensure that the network state is eventually consistent with desired state as specified by orchestration engine <NUM>. In general, control nodes <NUM> receive configuration state of SDN controller <NUM> from configuration nodes <NUM>, and exchange routes with each other via IBGP to ensure that all control nodes <NUM> have the same network state. Further, control nodes <NUM> exchange routes with VR agents <NUM> on servers <NUM> via XMPP. Control nodes <NUM> communicate the configuration state information, such as routing instances and forwarding policy, to VR agents <NUM>, e.g., via XMPP, for installation within respective virtual routers <NUM>. In some examples, control nodes <NUM> proxy traffic on behalf of servers <NUM>. These proxy requests may be received over XMPP. Further, control nodes <NUM> exchange routes with SDN gateway <NUM> via BGP, and exchange the configuration state of SDN controller <NUM> with service nodes <NUM> via Netconf.

Configuration nodes <NUM> provide a discovery service that customers <NUM> may use to locate various services available within the network. For example, if VR agent 35A attempts a connection with control node 54A, it uses a discovery service provided by configuration nodes <NUM> to discover the IP address of control node 54A. Clients executing on VMs <NUM> may use local configuration, DHCP or DNS to locate the service discovery server within configuration nodes <NUM>.

In some examples, configuration nodes <NUM> present northbound API that interfaces with orchestration engine <NUM>. Orchestration engine <NUM> uses this interface to install configuration state using the high-level data model. Configuration nodes <NUM> further include a message bus to facilitate communications amongst internal components. Configuration nodes <NUM> further include a transformer that discovers changes in the high-level model of orchestration engine <NUM> and transforms these changes into corresponding changes in the low level data model managed by SDN controller <NUM>. Configuration nodes <NUM> further include an IF-MAP server that provides a southbound API to push computed low-level configuration down to control nodes <NUM>. Furthermore, configuration nodes <NUM> include a distributed applications manager used to allocate unique object identifiers and to implement transactions across data center <NUM>.

In accordance with the techniques of the disclosure, one or more control nodes <NUM> deploy a software upgrade to network devices <NUM>, <NUM> of <FIG> without interrupting network traffic serviced by network devices <NUM>, <NUM>. In one example, control nodes <NUM> determine which network devices <NUM>, <NUM> are ISSU-capable and which network devices <NUM>, <NUM> are ISSU-incapable. Control nodes <NUM> may, for example, request from each network device <NUM>, <NUM>, configuration information for the network device <NUM>, <NUM>. Such configuration information may include, e.g., a device model or a current software version of the network device <NUM>, <NUM>. Control nodes <NUM> may receive the configuration information and compare the configuration information to a reference table, stored within state data <NUM>, that describes ISSU capabilities for a plurality of different types of network devices <NUM>, <NUM>. In some examples, control nodes <NUM> stores, for each network device <NUM>, <NUM>, an indication of whether the network device <NUM>, <NUM> is ISSU-capable or ISSU-incapable in state data <NUM>.

In some examples, control nodes <NUM> implement a policy controller (not depicted) which is configured to tag network devices <NUM>, <NUM> with ISSU-capability "tags. " As an example, where tag comprises a first key / value pair, the key of the tag is an "ISSU capability," and the value associated with the key is a "ISSU-capable" or "ISSU-NOT-CAPABLE" capability. The use of such tags may allow control nodes <NUM> to distribute, via the policy controller and based on the applied tags, network policies for application to network devices <NUM>, <NUM>. Such policies may include, for example, deployment of ISSU operations to apply software updates.

Control nodes <NUM> transmit messages to ISSU-capable network devices <NUM>, <NUM> (e.g., chassis switch 18A and TOR switch 16A of <FIG>) instructing chassis switch 18A and TOR switch 16A to perform an ISSU operation. In some examples, control nodes <NUM> may perform a "single trigger" operation in that controller nodes <NUM> may receive, from a user or administrator, a command to perform an ISSU operation on ISSU-capable network devices <NUM>, <NUM>. In response to the command, control nodes <NUM> transmit one or more messages instructing each of the ISSU-capable network devices <NUM>, <NUM> in switch fabric <NUM> of <FIG> to initiate the ISSU operation substantially simultaneously with a period of time that each other of the ISSU-capable network devices <NUM>, <NUM> initiates the ISSU operation.

In some examples, the message instructing the ISSU-capable devices to perform the ISU may include one or more software packages for the ISSU-capable network devices <NUM>, <NUM> to install. In other examples, the message may specify a location from which the ISSU-capable network devices <NUM>, <NUM> may retrieve the one or more software packages for the ISSU-capable network devices <NUM>, <NUM> to install. In some examples, upon successfully completing an ISSU operation, the ISSU-capable network devices <NUM>, <NUM> notify one or more of control nodes <NUM> of the completion of the ISSU operation.

Further, control nodes <NUM> transmit messages to ISSU-incapable network devices <NUM>, <NUM> (e.g., chassis switch 18B and TOR switch 16B of <FIG>) prompting ISSU-incapable network devices <NUM>, <NUM> to transmit messages to peer network devices <NUM>, <NUM>, such as chassis switch 18A and TOR switch 16A. The messages indicate that the ISSU-incapable network devices <NUM>, <NUM> are ISSU-incapable. In some examples, each ISSU-incapable network device <NUM>, <NUM> sends a message to each BGP peer of the ISSU-incapable network device. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-capable and have completed the ISSU operation. In some examples, the peer network devices <NUM>, <NUM>, are neighboring network devices <NUM>, <NUM> that are ISSU-incapable but are not undergoing a software update at the present time (e.g., have already completed the software update or are scheduled to do so at a later time). The peer network devices <NUM>, <NUM> that receive such messages may process the messages to determine that the ISSU-incapable network devices <NUM>, <NUM> may be unavailable to process or forward network traffic while performing the software upgrade. In response to determining that the ISSU-incapable devices <NUM>, <NUM> may (or will) be unavailable, the peer network devices <NUM>, <NUM> redirect traffic to avoid forwarding the traffic to the ISSU-incapable network devices <NUM>, <NUM> so as to avoid interruption of network traffic across, e.g., switch fabric <NUM>. Thus, by prompting the ISSU-incapable network devices <NUM>, <NUM> to send such messages indicating that the ISSU-incapable network devices <NUM>, <NUM> are ISSU-incapable, controller nodes <NUM> may orchestrate a software upgrade across switch fabric <NUM> without interrupting network traffic serviced by network devices <NUM>, <NUM>.

<FIG> is a block diagram illustrating an example of a router in which examples of the techniques described herein may be implemented. Network device <NUM> may operate as any physical device of switch fabric <NUM>, such as one of chassis switches <NUM> or TOR switches <NUM> of <FIG>. Further, network device <NUM> maybe an example of another type of network device, such as a gateway, switch, or router, that is not expressly depicted in <FIG>.

In the illustrated example of <FIG>, network device <NUM> includes a control unit <NUM> with a routing engine <NUM> that provides control plane functionality for the network device and a forwarding engine <NUM> that provides forwarding or data plane functionality for the network device to send and receive traffic by a set of interface cards 484A-484N ("IFCs <NUM>") that typically have one or more physical network interface ports. Control unit <NUM> may include one or more daemons (not shown) that comprise user-level processes that run network management software, execute routing protocols to communicate with peer routers or switches, maintain and update one or more routing tables in routing engine <NUM>, and create one or more forwarding tables for installation in forwarding engine <NUM>, among other functions.

Forwarding engine <NUM> performs packet switching and forwarding of incoming data packets for transmission over a network. As shown in <FIG>, forwarding engine <NUM> includes a forwarding information base (FIB) <NUM> that stores forwarding data structures associating network destinations with next hops and outgoing interfaces. Although not shown in <FIG>, forwarding engine <NUM> may comprise a central processing unit (CPU), memory and one or more programmable packet-forwarding application-specific integrated circuits (ASICs).

Routing engine <NUM> includes various protocols <NUM> that perform routing functions for network device <NUM>. In the illustrated example of <FIG>, routing engine <NUM> includes BGP <NUM> and IGP <NUM> as routing protocols used to exchange routing information with other routing devices in a network in order to discover the network topology and update a routing information base (RIB) <NUM>. In the examples described in this disclosure, IGP <NUM> may be a link-state routing protocol such as open shortest path first (OSPF) or intermediate system-intermedia system (IS-IS).

In examples where network device <NUM> is ISSU-capable, routing engine <NUM> may include ISSU <NUM> to provide network device <NUM> with ISSU capabilities for performing software upgrades without interrupting the traffic processing and forwarding capabilities of network device <NUM>. In examples where network device <NUM> is ISSU-incapable, routing engine <NUM> may not include ISSU <NUM>. In some examples, one or more functions of network device <NUM> may be virtualized, such as where network device <NUM> is a virtual router. Additional information on how a virtual router may perform an ISSU is described in <CIT> to Venkata et al. , entitled "IN-SERVICE SOFTWARE UPGRADE OF VIRTUAL ROUTER WITH REDUCED PACKET LOSS" and filed on June <NUM>, <NUM>.

RIB <NUM> may describe the topology of the network in which network device <NUM> resides, and may also describe various routes within the network and the appropriate next hops for each route, i.e., the neighboring routing devices along each of the routes. Routing engine <NUM> analyzes the information stored in RIB <NUM> to generate forwarding information. Routing engine <NUM> then installs forwarding data structures into FIB <NUM> within forwarding engine <NUM>. FIB <NUM> associates network destinations with specific next hops and corresponding interface ports within the forwarding plane. LIB <NUM> maintains mappings of next hop labels to the next hops for each route within the network from RIB <NUM>.

According to the techniques of the disclosure, in examples where network device <NUM> is ISSU-incapable, network device <NUM> may be unable to process and forward network traffic while performing a software upgrade. For example, while performing a software upgrade, network device <NUM> may need to temporarily transition offline or temporarily suspend, deactivate, restart, reboot, or otherwise disable certain functionality, such as software modules, or hardware features. Therefore, in response to determining that network device <NUM> will perform a software upgrade, network device <NUM> may transmit messages to peer network devices <NUM>, <NUM> indicating that network device <NUM> is ISSU-incapable. In some examples, network device <NUM> may transmit such messages via BGP <NUM>. In some examples, the BGP message specifies the ISSU capability of network device <NUM>, such as whether network device <NUM> is "ISSU-CAPABLE" or "ISSU-NOT-CAPABLE," as extended community <NUM> for BGP <NUM>. For example, network device <NUM> may transmit a BGP message to a neighboring network device <NUM>, <NUM> that specifies that network device <NUM> is ISSU-NOT-CAPABLE as extended community <NUM> for BGP <NUM>. In some examples, network device <NUM> may transmit such a message in response to a message from a centralized controller, such as SDN controller <NUM> or an administrator locally accessing network device <NUM>. In some examples, after network device <NUM> has completed the software upgrade, network device <NUM> may inform peer network devices <NUM>, <NUM> that network device <NUM> is online and capable of processing network traffic. This may be done through the use of conventional BGP messages as described in "BGP MPLS-Based Ethernet VPN," RFC <NUM>, referenced above.

Furthermore, network device <NUM> may receive a message as described above that indicates that another network device <NUM>, <NUM> is ISSU-incapable. In response to receiving the message indicating that the other network device <NUM>, <NUM> is ISSU-incapable, network device <NUM> is configured to redirect traffic to avoid forwarding the traffic to the other network device <NUM>, <NUM> so as to avoid interruption of network traffic serviced by data center <NUM>. In some examples, network device <NUM> may delete an entry describing the other ISSU-incapable network device <NUM>, <NUM> from RIB <NUM> so as to avoid forwarding traffic to the other ISSU-incapable network device <NUM>, <NUM> while the other ISSU-incapable network device <NUM>, <NUM> is performing the upgrade, and thus unable to process or forward network traffic. In some examples, network device <NUM> calculates a new route to reach a destination of the network traffic. The new route may include, e.g., another one of network devices <NUM>, <NUM> as a next hop for the network traffic as an alternate route so as to bypass the other ISSU-incapable network device <NUM>, <NUM>. Network device <NUM> stores the new route in RIB <NUM> and forwarding structure in FIB <NUM>. Network device <NUM> may be configured to process such a message indicating that the other network device <NUM>, <NUM> is ISSU-incapable regardless of whether network device <NUM> itself is ISSU-capable or ISSU-incapable.

<FIG> is a block diagram illustrating an example message that includes ISSU capability information in accordance with techniques described herein. Extended BGP UPDATE message <NUM> conforms to Multiprotocol Extensions for BGP-<NUM> (MP-BGP). Additional information with respect to MP-BGP is provided by<NPL>.

BGP UPDATE messages, such as example message <NUM>, may be used to transfer routing information between BGP peers such as between network devices <NUM>, <NUM>. The information in UPDATE message <NUM> may be used by one of network devices <NUM>, <NUM> to construct a graph that describes the relationships of various peer devices to the one of network devices <NUM>, <NUM>. By applying rules set forth in such a BGP UPDATE message, a network device <NUM>, <NUM> may detect routing information loops and other anomalies remove such problems from inter-AS routing.

One of network devices <NUM>, <NUM> may use BGP UPDATE messages, such as example message <NUM>, to advertise feasible routes that share common path attributes to a peer network device <NUM>, <NUM>, or to withdraw multiple unfeasible routes from service. A BGP UPDATE message may simultaneously advertise a feasible route and withdraw multiple unfeasible routes from service. A BGP UPDATE message includes a fixed-size BGP header, and may includes other fields (e.g., such as the extended community described herein). Some fields may not be present in every BGP UPDATE message.

Example BGP UPDATE message <NUM> includes withdrawn routes <NUM>. For purposes of illustration, extended BGP UPDATE message <NUM> is illustrated using glyphs, rather than with packet fields. Withdrawn routes <NUM> may include a preceding integer that indicates a total length of the withdrawn routes. A value of <NUM> indicates that no routes are being withdrawn from service, and that no withdrawn routes are present in the UPDATE message. Withdrawn routes <NUM> may further include a list of IP address prefixes for the routes that are being withdrawn from service.

Example BGP UPDATE message <NUM> further includes Path Attributes <NUM>. Path Attributes <NUM> is a variable-length sequence that specifies path attributes for UPDATE message <NUM>. Path Attributes <NUM> is typically present in BGP UPDATE messages, except where a BGP UPDATE message specifies only withdrawn routes. Path Attributes <NUM> further specifies one or more attribute flags. The attribute flags specify, e.g., whether the attribute is optional or well known, whether an optional attribute is transitive or non-transitive, whether the information contained in the optional transitive attribute is partial or complete, an attribute length, and an attribute type.

The attribute type specified by Path Attributes <NUM> may include, e.g., an Origin, an AS-Path, and a Next-Hop. The Origin attribute type is a well-known mandatory attribute that defines the origin of the path information. For example, the Origin may specify one of Interior Gateway Protocol (IGP) (e.g., Network Layer Reachability Information (NLRI) is interior to the originating AS, Exterior Gateway Protocol (EGP) (e.g., NLRI is learned via the EGP protocol), or Incomplete (e.g., NLRI is learned by some other means).

The AS-Path attribute type is a well-known mandatory attribute that is composed of a sequence of AS path segments. The AS-Path may specify an unordered set of ASes a route in UPDATE message <NUM> has traversed, or an ordered set of ASes a route in the UPDATE message <NUM> has traversed.

The Next Hop attribute type is a well-known mandatory attribute that defines an IP address (e.g., in unicast) of the network device that should be used as a next hop to a destinations listed in MP-REACH-NLRI field <NUM> of UPDATE message <NUM>.

Each of NLRI fields <NUM> and <NUM> contain a list of IP address prefixes. For example, MP-REACH-NLRI <NUM> carries a set of reachable destinations together with the next hop information to be used for forwarding to these destinations. MP-UNREACH-NLRI <NUM> carries a set of unreachable destinations. Both of these attributes are optional and non-transitive. This way, a BGP speaker, such as one of network devices <NUM>, <NUM>, that does not support multiprotocol BGP capabilities may ignore the information carried in these attributes and will not pass it to other BGP speakers.

MP-REACH-NLRI <NUM> specifies, e.g., an Address Family Identifier (AFI), a Subsequent Address Family Identifier (SAFI), a next hop, and NLRI. The AFI field, in combination with the SAFI field, identifies a set of Network Layer protocols to which the address carried in the Next Hop field belongs, the way in which the address of the next hop is encoded, and the semantics of the NLRI. If the Next Hop is allowed to be from more than one Network Layer protocol, the encoding of the Next Hop provides a way to determine the Network Layer protocol used. The NLRI field lists NLRI for the feasible routes advertised in MP-REACH-NLRI <NUM>.

MP-UNREACH-NLRI <NUM> specifies, e.g., an AFI, an SAFI, and NLRI for one or more withdrawn routes. The AFI and SAFI of MP-UNREACH-NLRI <NUM> are similar to the AFI and SAFI of MP-REACH-NLRI <NUM>. Withdrawn Routes NLRI lists NLRI for one or more routes that are being withdrawn from service. In some examples, message <NUM> contains MP_UNREACH_NLRI <NUM> and is not required to carry any other path attributes.

UPDATE message <NUM> advertises one set of path attributes <NUM> but may advertise multiple destinations, provided that the destinations share path attributes <NUM>. Path Attributes <NUM> contained in a given UPDATE message <NUM> applies to all destinations specified by NLRI fields <NUM> and <NUM> of UPDATE message <NUM>.

UPDATE message <NUM> may list multiple routes that are to be withdrawn from service. Each such route is identified by its destination (expressed as an IP prefix), which unambiguously identifies the route in the context of the BGP speaker - BGP speaker connection to which it has been previously advertised.

UPDATE message <NUM> may advertise only routes that are to be withdrawn from service, in which case message 500does not include path attributes <NUM> or NLRI. Conversely, UPDATE message <NUM> may advertise only a feasible route, in which case Withdrawn Routes field <NUM> need not be present.

UPDATE message <NUM> typically does not include the same address prefix in Withdrawn Routes field <NUM> as in NLRI fields <NUM> or <NUM>. However, a BGP speaker may support processing UPDATE message <NUM> in this form. A BGP speaker typically treats an UPDATE message <NUM> of this form as though Withdrawn Routes field <NUM> does not contain the address prefix.

Extended communities attribute <NUM> includes ISSU Capability field <NUM> that defines an ISSU capability of a network device. For example, network device <NUM>, <NUM> may send message <NUM> that includes ISSU Capability field <NUM> to indicate the device's ability to perform an ISSU. ISSU field <NUM> may specify, e.g., "ISSU-CAPABLE" to indicate that a network device <NUM>, <NUM> is ISSU-capable, or "ISSU-NOT-CAPABLE" to indicate that a network device <NUM>, <NUM> is ISSU-incapable. In this way, a network devices <NUM>, <NUM> that receives a BGP UPDATE message <NUM> indicating that a peer network device is ISSU-incapable may determine that the peer network device may be unable to process or forward network traffic while performing a software upgrade. The network devices <NUM>, <NUM> may further route network traffic around the peer network device so as to allow switch fabric <NUM> to continue to operate to transport customer traffic while the peer network device is performing the software update.

<FIG> is a flowchart illustrating an example operation in accordance with the techniques of the disclosure. For convenience, <FIG> is described with respect to <FIG>, <FIG>, and <FIG>. In the example of <FIG>, first network device 400A and second network device 400B are examples of network device <NUM> of <FIG>. Further, first network device 400A and second network device 400B are BGP peers of one another.

In the example of <FIG>, a software upgrade is deployed to a mixed network of network devices <NUM>, <NUM> that include ISSU-capable network device 400A and ISSU-incapable network device 400B without interrupting network traffic serviced by the mixed network. In one example, SDN controller <NUM> of cloud data center <NUM> determines which network devices <NUM>, <NUM> of cloud data center <NUM> are ISSU-capable and which network devices <NUM>, <NUM> are ISSU-incapable. In the example of <FIG>, SDN controller <NUM> determines that first network device 400A is ISSU-capable and second network device 400B is ISSU-incapable (<NUM>). SDN controller <NUM> may, for example, request from network devices 400A and 400B configuration information for the network device. Such configuration information may include, e.g., a device model or a current software version of the network device. SDN controller <NUM> may receive the configuration information and compare the configuration information to a reference table that describes ISSU capabilities for a plurality of different types of network devices. In some examples, SDN controller <NUM> stores, for each of network devices 400A and 400B, an indication of whether the network device is ISSU-capable or ISSU-incapable.

SDN controller <NUM> transmits a message instructing network device 400A to perform an ISSU operation (<NUM>). In some examples, the instructions may include one or more software packages for network device 400A to install. In other examples, the instructions may specify a location from which network device 400A may retrieve the one or more software packages for network device 400A to install. In response to receiving the instructions, network device 400A performs the ISSU operation (<NUM>). In some examples, upon successfully completing the ISSU operation, network device 400A transmits a message to SDN controller <NUM> to notify SDN controller <NUM> of the completion of the ISSU operation.

Further, SDN controller <NUM> transmits instructions to network device 400B instructing network device 400B to transmit a message indicating that network device 400B is ISSU-incapable (<NUM>). While in the example operation of <FIG>, SDN controller <NUM> transmits the instructions that network device 400B will go offline, in other examples, network device 400B may receive such instructions directly from an administrator accessing network device 400B locally. In some examples, SDN controller <NUM> does not transmit the instructions to network device 400B until after determining that network device 400A has completed the ISSU. SDN controller may wait until, e.g., receiving a message indicating successful completion of the ISSU from all of the ISSU-capable device <NUM>, <NUM> and including network device 400A, or receiving messages indicating successful completion of the ISSU from a substantial number of ISSU-capable device <NUM>, <NUM> (e.g., <NUM>%, <NUM>%, <NUM>%, <NUM>%, etc.).

In response to receiving the instructions from SDN controller <NUM>, network device 400B transmits a message to peer network devices, such as network device 400A, that network device 400B is ISSU-incapable (<NUM>). In some examples, network device 400B may transmit such a message as a BGP message including a BGP extended community specifying that network device 400B is "ISSU-NOT-CAPABLE" as described above.

In response to receiving such a message indicating that network device 400B is ISSU-incapable, network device 400A determines that network device 400B may subsequently be unable to process or forward traffic while performing a software upgrade (<NUM>). Further, network device 400A redirects traffic to avoid forwarding the traffic to network device 400B so as to avoid interruption of network traffic serviced by data center <NUM> while network device 400B is performing the software upgrade (<NUM>). In some examples, network device 400A deletes an entry describing network device 400B from a RIB of network device 400A so as to avoid forwarding traffic to network device 400B while network device 400B performing the upgrade. Network device 400A may further calculate a new route to reach a destination of the network traffic. The new route may include, e.g., another one of network devices <NUM>, <NUM> as a next hop for the network traffic as an alternate route so as to bypass network device 400B. Network device 400A may stores the new route and forwarding structure a FIB and RIB of network device 400A.

In some examples, after network device 400B has completed the software upgrade, network device 400B may inform neighboring network devices, such as network device 400A, that network device 400B is online and capable of processing network traffic. This may be done through the use of conventional BGP messages as described in "BGP MPLS-Based Ethernet VPN," RFC <NUM>, referenced above.

The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. Other examples of computer-readable media include transitory media such as carrier signals and transmission media.

Claim 1:
A method comprising:
determining, by a centralized controller of a network (<NUM>), that one or more first network devices (18A, 16A) of a plurality of network devices for the network are In-Service Software Upgrade, ISSU,-capable and that a second network device (18B, 16B) of the plurality of network devices is not ISSU-capable, wherein the one or more first network devices that are ISSU-capable are configured to at least one of process or forward network traffic while performing a software upgrade, and wherein the second network device that is not ISSU-capable is unable to process or forward network traffic while performing the software upgrade;
transmitting, by the centralized controller, one or more messages instructing the one or more first network devices to perform an ISSU operation; and
transmitting, by the centralized controller, one or more messages instructing the second network device to transmit a message to one or more neighbor network devices (16A, 18A, <NUM>, 16N) of the plurality of network devices that neighbor the second network device, wherein the message:
indicates that the second network device is not ISSU-capable; and
directs the one or more neighbor network devices of the plurality of network devices to redirect traffic to avoid forwarding the traffic to the second network device by:
removing, from a Routing Information Base of the one or more neighbor network devices, an entry describing the second network device; and/or
calculating a new route to reach a destination of the traffic, wherein the new route includes a third network device as a next hop, and storing the new route in a Forwarding Information Base of the one or more neighbor network devices.