Patent Description:
The present disclosure relates generally to failure protection for communication networks and, more particularly, to a N+<NUM> redundancy scheme for virtualized services with low latency fail-over.

There are two main failure protection schemes for maintaining continuity of service in the event that a network node in a communication network responsible for handling user traffic fails. The two main protection schemes are <NUM>+<NUM> protection and N+<NUM> protection. With <NUM>+<NUM> protection, N standby nodes are available for N network nodes to take over the function of a failed primary node or nodes. In <NUM>+<NUM> protection schemes, each network node has its own dedicated standby node, which can take over traffic currently being handled by its corresponding network node without loss of sessions. This is known as "hot standby. " One drawback of <NUM>+<NUM> protection is that it requires doubling of system resources. With N+<NUM> protection, <NUM> standby node is available for N network nodes to take over the function of a single failed primary node. However, the N+<NUM> redundancy scheme typically provides only "cold standby" protection so that traffic handled by the failed network node is lost at a switchover. " Existing N+<NUM> solutions, don't preserve state of the failed primary node resulting in tear-down of existing sessions. This is because the standby node is not dedicated to any specific one of the N primary nodes, so there was no solution on how to have the state of any one of the primary nodes available in the backup node after the failure. Ultimately, the only benefit is that capacity will not drop after a failure but it does not provide protection for ongoing sessions.

In case of Virtual Router Redundancy Protocol (VRRP)-based solutions, a standby node may take over the Internet Protocol (IP) address of a failed primary node as well as the functions of the failed primary node, but these solutions do not take over real-time state of the failed primary node that would be needed for preserving session continuity for sockets. Moreover, the operator of the network has to configure separate VRRP sessions with separate IP addresses for each VRRP relationship (i.e., the standby nodes need a separate VRRP context per each primary node it is deemed to protect). This way the configuration overhead in a bigger cluster makes the solution cumbersome.

Document <CIT> may be construed to disclose a radio network controller and methods for reestablishing sessions in a wireless network. At least a portion of session information associated with a first session is saved; and in response to detecting an unexpected degradation of the first session, reestablishment of the first session is triggered using the portion of the session information.

The present disclosure comprises methods and apparatus of providing N+<NUM> redundancy for a cluster of network nodes including a standby node and a plurality of primary nodes. When the standby node determines that a primary node in a cluster has failed, the standby node configures the standby node to use an IP address of the failed primary node. The standby node further retrieves session data for user sessions associated with the failed primary node from a low latency database for the cluster and restores the user sessions at the standby node. When the user sessions are restored, the standby node switches from a standby mode to an active mode.

According to the disclosure, there are provided a method, a network node and a non-transitory computer-readable storage media according to the independent claims. Further developments are set forth in the dependent claims.

A first aspect of the disclosure comprises methods of providing N+<NUM> redundancy for a cluster of network nodes. In one embodiment, the method comprises determining, by a standby node, that a primary node in a cluster has failed, configuring the standby node to use an IP address of the failed primary node, retrieving session data for user sessions associated with the failed primary node from a low latency database for the cluster, restoring the user sessions at the standby node, and switching from a standby mode to an active mode.

A second aspect of the disclosure comprise a network node configured as a standby node to provide N+<NUM> protection for a cluster of network nodes including the standby node and a plurality of primary nodes. The standby node comprises a network interface for communicating over a communication network and a processing circuit. The processing circuit is configured to determine that a primary node in a cluster has failed. Responsive to determining that a primary node has failed, the processing circuit configures the standby node to use an IP address of the failed primary node. The processing circuit is further configured to retrieve session data for user sessions associated with the failed primary node from a low latency database for the cluster and restore the user sessions at the standby node. After the user sessions are restored, the processing circuit switches the standby node from a standby mode to an active mode.

A third aspect of the disclosure comprises a computer program comprising executable instructions that, when executed by a processing circuit in a redundancy controller in a network node, causes the redundancy controller to perform the method according to the first aspect.

Referring now to the drawings, <FIG> illustrates a server cluster <NUM> with N+<NUM> redundancy protection that implements a virtual network function (VNF), such as a media gateway (MGW) function or Border Gateway Function (BGF). The server cluster <NUM> can be used, for example, in a communication network, such as an Internet Protocol Multimedia Subsystem (IMS) network or other telecom network. The server cluster <NUM> comprises a plurality of network nodes, which include a plurality of primary nodes <NUM> for handling user sessions and a standby node <NUM> providing N+<NUM> protection in the event that a primary node <NUM> fails. Each of the network nodes <NUM>, <NUM> in the cluster <NUM> can be implemented by dedicated hardware and processing resources. Alternatively, the network nodes <NUM>, <NUM> can be implemented as virtual machines (VMs) using shared hardware and processing resources.

User sessions (e.g., telephone calls, media streams, etc.) are distributed among the primary nodes <NUM> by a load balancing node <NUM>. An orchestrator <NUM> manages the server cluster <NUM>. A distributed, low-latency database <NUM> serves as a data store for the cluster <NUM> to store the states of the user sessions being handled by the primary nodes <NUM> as hereinafter described. An exemplary distributed database <NUM> is described in an article titled "<NPL>.

The network nodes <NUM>, <NUM> are part of the same subnet with a common IP prefix. Each user session is associated with a particular IP address, which identifies the primary node <NUM> that handles user traffic for the user session. State information for the user sessions is stored in a distributed, low latency, database <NUM> that serves the server cluster <NUM>. In the event that a primary node <NUM> fails, the standby node <NUM> can retrieve the state information of user sessions handled by the failed primary node <NUM> and restore the "lost" user sessions so that service continuity is maintained for the user sessions.

<FIG> shows the server cluster <NUM> of <FIG> in simplified form to graphically illustrates the basic steps of a fail-over procedure. It is assumed in <FIG> that Primary Node <NUM> has failed. At <NUM>, the failure is detected by the standby node <NUM> and the failed primary node <NUM> is identified. At <NUM>, the standby node retrieves the state information for Primary Node <NUM> from the database <NUM> and recreates the user sessions at the standby node <NUM>. At <NUM>, the standby node takes over the IP address of Primary Node <NUM> and configures its network interface to use the IP address. At <NUM>, the standby node advertises the location change of the IP address. Thereafter, the traffic for user sessions associated with IP Address <NUM> will be routed to the standby node <NUM> rather than to Primary Node <NUM>.

The failure protection scheme used in the present disclosure can be viewed as having three separate phases. In a first phase, referred to as the preparation phase, a redundant system is built so that the system is prepared for failure of a primary node <NUM>. The second phase comprises a fail-over process in which the standby node <NUM>, upon detecting a failure of then primary node <NUM>, takes over the active user sessions handled by the failed primary node <NUM>. After the fail-over process is complete, a post-failure process restores capacity and redundancy to the system that is lost by the failure of the primary node <NUM> so that backup protection is re-established to protect against future network node failures.

During the preparation phase, state information necessary to restore the user sessions is externalized and stored in the database <NUM> by each primary node <NUM>. Conventional log-based approaches or checkpointing can be used to externalize the state information. Another suitable method of externalizing state information is described in co-pending application <CIT>. Necessary data to be stored in the database <NUM> is dependent on the application and the communication protocol used in the application. For TCP sessions, such state information may comprise port numbers, counters, sequence numbers, various data on Transport Control Protocol (TCP) buffer windows, etc. Generally, all state information that is necessary to continue the user session should be stored externally.

In order to ensure that a backup is readily available to replace a primary node <NUM> that has failed, a "warm" standby node <NUM> is provisioned and made available to take over any user sessions for a failed primary node <NUM>. During the provisioning, system checks are performed to ensure that:.

It is not known in advance which of the primary nodes <NUM> will fail, however, the standby node <NUM> is ready to fetch necessary state information from the database <NUM> to take over for any one of the primary nodes <NUM>. This standby mode is referred to herein a "warm" standby.

The fail-over process is triggered by the failure of one of the primary nodes <NUM>. In some embodiments, failure is detected by the standby node <NUM> based on a "heartbeat" or "keepalive" signaling. In some embodiments, the primary nodes <NUM> may periodically transmit a "heartbeat" signal and a failure is detected when the heartbeat signal is not received by the standby node <NUM>. In other embodiments, the standby node <NUM> may periodically transmit a "keepalive" signal or "ping" message to each of the primary nodes <NUM>. In this case, a failure is detected when a primary node <NUM> fails to respond. This "keepalive" signaling process should be continuously run pairwise between the standby node <NUM> and each primary node <NUM>.

In other embodiments, the failure of a primary node <NUM> can be detected by another network entity and communicated to the standby node <NUM> in the form of a failure notification. For example, one primary node <NUM> may detect the failure of another primary node <NUM> and send a failure notification to the standby node <NUM>. In another embodiment, the database <NUM> may detect the failure of a primary node <NUM> and send a failure notification to the standby node <NUM>.

When a fail-over is triggered, the standby node <NUM> retrieves the IP address or addresses of the failed primary node <NUM>, as well as the session states (e.g., application and protocol dependent context data) necessary to re-initiate the user sessions at the standby node <NUM>. In some embodiments, the standby node <NUM> writes the network identify (e.g., IP address) of the failed primary node <NUM> into a global key called STDBY-IDENTITY, which is stored in the database <NUM> so that all nodes in the server cluster <NUM> are aware that the standby node <NUM> has assumed the role of the failed primary node <NUM>. Responsive to the failure detection or failure indication, the standby node <NUM> configures its network interface to use the IP address or addresses of the failed primary node <NUM> and loads the retrieved session states into its own tables. When the standby node <NUM> is ready to take over, the standby node <NUM> broadcasts a Gratuitous Address Resolution Protocol (GARP) message with its own Medium Access Control (MAC) address and its newly configured IP address(es), so that the routers in the subnet know to forward packets with the IP address(es) formerly used by the failed primary node <NUM> to the standby node's MAC address. The same general principles also apply to Internet Protocol version <NUM> (IPv6) interfaces (Unsolicited Neighbor Advertisement message).

During the post-failure phase, the original capacity of the server cluster <NUM> with N primary nodes <NUM> and <NUM> standby node <NUM> is restored. There are essentially two alternative approaches to restoring the system capacity.

In a first approach for the post-failure phase, the standby node <NUM> switches from a standby mode to an active mode and serves only temporarily as a primary node, reverting to a "warm" standby mode when done. The standby node <NUM> serves only the user sessions that were taken over from the failed primary node <NUM> and is not assigned to handle any new user sessions by the load balancing node <NUM>. When the orchestrator <NUM> learns about the failure of a primary node <NUM>, it re-establishes a new primary node <NUM> to replace the failed primary node <NUM> and restore system capacity according to a regular scale-out procedure. The orchestrator <NUM> should ensure that the IP addresses used by the failed primary node <NUM> on the user plane are reserved, because these addresses are taken over by the standby node <NUM>. In case of an OpenStack-based orchestrator <NUM>, reserving the IP addresses means that the "ports" should not be deleted when the failed primary node <NUM> disappears. This requires, however, garbage collection. The standby node <NUM>, when it terminates, sends a trigger to the orchestrator <NUM> indicating that the ports used by the affected IP addresses can be deleted. After this notification, the IP addresses can be assigned to new network nodes (e.g., VNFs).

During the post-failure phase, the operation of the load balancing node <NUM> needs to take into account the failed primary node <NUM>. Immediately after the failure, however, the load balancing node <NUM> does not assign new incoming sessions to either the failed primary node <NUM> or the standby node <NUM>. As noted above, the standby node <NUM> continues serving existing user sessions taken over from the failed primary node <NUM>, but does not receive new sessions. After the last session is finished at the standby node <NUM>, or upon expiration of a timer (MAX_STANDBY_LIFETIME), the standby node <NUM> erases or clears the STDBY_IDENTITY field in the database <NUM>, sends a notification to the orchestrator <NUM> indicating that the IP addresses of X can be released, and transitions back to a "warm" standby mode. The MAX_STANDBY_LIFETIME timer, if used, is started when the standby node <NUM> takes over for the failed primary node <NUM>.

In a second approach for the post-failure phase, the standby node <NUM> permanently assumes the role of the failed primary node <NUM> and the orchestrator re-establishes system capacity by initiating a new standby node <NUM>. In this case, The standby node <NUM> sends a notification message or other indication to the orchestrator <NUM> indicating that the IP address(es) of the failed primary node <NUM> were assumed or taken over by the standby node <NUM> so that the orchestrator <NUM> knows (i) to which primary node <NUM> the IP addresses belong, and (ii) that these IP addresses cannot be used for new instances of the primary nodes <NUM> in case of a scale-out. The standby node <NUM> (now fully a primary node <NUM>) triggers the orchestrator <NUM> to launch a new instance for the standby node <NUM> to restore the original redundancy protection.

There may be circumstance where primary node <NUM> fails only temporarily, typically because of a VM reboot. Following the restart, the primary node <NUM> may try to use its earlier IP address(es), which would cause a conflict with the standby node <NUM> that is serving the ongoing user sessions associated with those addresses. Before restarting, the primary node reads the STDBY_IDENTITY key in the database <NUM>. If the STDBY_IDENTITY key matches the identity of the primary node <NUM>, the primary node <NUM> pauses and waits until the key is erased, indicating that the IP address used by the standby node has been released, or asks for new configuration parameters from the orchestrator <NUM>.

<FIG> illustrates an exemplary fail-over procedure used in some embodiments of the present disclosure. When the standby node <NUM> detects a node failure or receives a failure notification (step <NUM>), it writes the network identity (e.g., IP address) of the failed primary node <NUM> into the global key STDBY-IDENTITY stored in database <NUM> (step <NUM>). The standby node <NUM> sends a GET message to the database <NUM> to request session information for the failed primary node <NUM> (step <NUM>). In response to the GET message, the database <NUM> sends the session data for the failed primary node <NUM> to the standby node <NUM> (step <NUM>). As previously described, the standby node <NUM> configures its network interface to use the IP address of the failed primary node <NUM> and broadcasts a GARP message to the network (step <NUM>). Upon broadcast of the GARP message, the routers in the network will route messages previously sent to the failed primary node <NUM> to the standby node <NUM> and the standby node <NUM> will handle the user sessions of the failed primary node <NUM>. When the load balancing node <NUM> is notified of the failure of a primary node <NUM>, the load balancing node <NUM> removes the primary node <NUM> from its list of primary nodes <NUM> so that no new sessions will be assigned to the failed primary node <NUM> (step <NUM>). Also, when the orchestrator <NUM> is notified of the failure of a primary node <NUM>, the orchestrator <NUM> instantiates a new instance of the primary node <NUM> to replace the failed primary node <NUM> (step <NUM>).

In the embodiment shown in <FIG>, it is assumed that the standby node <NUM> is only temporarily active and reverts to a standby mode when a standby timer expires or after the last session assumed by the standby node <NUM> ends. In this case, when the standby timer expires (step <NUM>), or when the last user session ends, the standby node <NUM> sends a release notification message to the orchestrater <NUM> to release the IP address assumed by the standby node <NUM>, so that the IP address is available for reassignment (step <NUM>). The standby node <NUM> also clears the standby identity key stored in the database <NUM> (step <NUM>).

<FIG> illustrates another exemplary fail-over procedure used in some embodiments of the present disclosure where the standby <NUM> permanently replaces the failed primary node <NUM>. Steps <NUM>-<NUM> are the same as the fail-over procedure shown in <FIG>. After becoming active, the standby node <NUM> sends a notification message to the orchestrator <NUM> and/or load balancing node <NUM> to notify the orchestrator <NUM> and/or load balancing node <NUM> that it taken over the IP address of the failed primary node <NUM> (step <NUM>). The orchestrator <NUM> then instantiates a new instance of a standby node <NUM> to replace the previous standby node <NUM> (step <NUM>). In some embodiments, the orchestrator <NUM> may notify the load balancing node <NUM> that the standby node <NUM> is now designated as a primary node <NUM>. The load balancing node <NUM> adds the standby node <NUM> to its list of available primary nodes <NUM> in response to the notification from the standby node <NUM> or orchestrator <NUM> (step <NUM>).

<FIG> illustrates an exemplary method <NUM> implemented by a standby node <NUM> in a server cluster <NUM> including a plurality of primary nodes <NUM>. When the standby node <NUM> determines that a primary node <NUM> in a cluster <NUM> has failed (block <NUM>), the standby node <NUM> configures its network interface to use an IP address of the failed primary node <NUM> (block <NUM>). The standby node <NUM> further retrieves from a low latency database for the cluster, session data for user sessions associated with the failed primary node <NUM> (block <NUM>) and restores the user sessions at the standby node <NUM> (block <NUM>). When the user sessions are restored, the standby node <NUM> switches from a standby mode to an active mode (block <NUM>).

In some embodiments of the method <NUM>, determining that a primary node <NUM> in a cluster <NUM> has failed comprises sending a periodic keepalive message to one or more primary nodes <NUM> in the cluster <NUM>, and determining a node failure when the failed primary node <NUM> fails to response to a keepalive message.

In some embodiments of the method <NUM>, determining that a primary node <NUM> in a cluster has failed comprises receiving a failure notification. As an example, the failure notification can be received from the database <NUM>.

In some embodiments of the method <NUM>, configuring the standby node <NUM> to use an IP address of the failed primary node <NUM> comprises configuring a network interface to use the IP address of the failed primary node <NUM>.

In some embodiments of the method <NUM>, configuring the standby node <NUM> to use an IP address of the failed primary node <NUM> further comprises announcing a binding between the IP address and a MAC address of the standby node <NUM>.

Some embodiments of the method <NUM> further comprise setting a standby identity key in the database to an identity of the failed primary node <NUM>.

Some embodiments of the method <NUM> further comprise, after a last one of the user sessions ends, releasing the IP address of the failed primary node <NUM> and switching from the active mode to the standby mode.

Some embodiments of the method <NUM> further comprise, after a last one of the user sessions ends, clearing the standby identity key in the database <NUM>.

Some embodiments of the method <NUM> further comprise notifying an orchestrator <NUM> that the standby node <NUM> has replaced the failed primary node <NUM> and receiving new user sessions from a load-balancing node <NUM>.

<FIG> illustrates an exemplary method <NUM> of failure recovery implemented by a primary node <NUM> in a cluster <NUM> of network nodes following a temporary failure of the primary node <NUM>. Following a restart by the primary node <NUM>, the primary node <NUM> determines whether an IP address of the primary node <NUM> is being used by a standby node <NUM> in the cluster <NUM> of network nodes (block <NUM>). Upon determining that the IP address is being used by a standby node <NUM>, the primary node <NUM> obtains a new IP address or waits for the IP address to be released by the standby node <NUM> (block <NUM>). In the former case, the primary node <NUM> reconfigures its network interface with the new IP address and returns to an active mode (block <NUM>, <NUM>). In the latter case, the primary node <NUM> detects release of the IP address by the standby node <NUM> (block <NUM>) and, responsive to such detection, returns to an active mode (block <NUM>).

The primary node <NUM> determines whether an IP address of the primary node <NUM> is being used by a standby node <NUM> in the cluster <NUM> of network nodes by getting a standby identity from a database <NUM> serving the cluster <NUM> of network nodes, and comparing the standby identity to an identity of the primary node <NUM>.

The primary node <NUM> determines when the IP address is released by monitoring the standby identity stored in the database <NUM> and determining that the IP address is released when the standby identity is cleared or erased.

<FIG> illustrates an exemplary network node <NUM> according to an embodiment. The network node <NUM> can be configured as a primary node <NUM> or as a standby node <NUM>. The network node <NUM> includes a network interface <NUM> for sending and receiving messages over a communication network, a processing circuit <NUM>, and memory <NUM> The processing circuit <NUM> may comprise one or more microcontrollers, microprocessors, hardware circuits, firmware, or a combination thereof. Memory <NUM> comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit <NUM> for operation. Memory <NUM> may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory <NUM> stores a computer program <NUM> comprising executable instructions that configure the processing circuit <NUM> to implement the procedures and methods as herein described, including one or more of the methods <NUM>, <NUM> shown in <FIG> and <FIG>. A computer program <NUM> in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program <NUM> for configuring the processing circuit <NUM> as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program <NUM> may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium. In some embodiments, memory <NUM> stores virtualization code executed by the processing circuit <NUM> for implementing the network node <NUM> as a virtual machine.

A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

Claim 1:
A method (<NUM>) of providing N+<NUM> redundancy for a cluster (<NUM>) of network nodes including a standby node (<NUM>, <NUM>) and a plurality of primary nodes (<NUM>, <NUM>), the method (<NUM>) being performed by the standby node (<NUM>, <NUM>) and comprising:
determining (<NUM>), by the standby node (<NUM>, <NUM>), that one of the primary nodes (<NUM>, <NUM>) in the cluster (<NUM>) has failed;
configuring (<NUM>) the standby node (<NUM>, <NUM>) to use an Internet Protocol, IP, address of the failed primary node (<NUM>, <NUM>), comprising:
- configuring a network interface (<NUM>) of the standby node (<NUM>, <NUM>) to use the IP address of the failed primary node (<NUM>, <NUM>); and
- announcing a binding between the IP address and a Medium Access Control, MAC, address of the standby node (<NUM>, <NUM>);
retrieving (<NUM>), from a low latency database (<NUM>) for the cluster (<NUM>), session data for user sessions associated with the failed primary node (<NUM>, <NUM>);
restoring (<NUM>) the user sessions at the standby node (<NUM>, <NUM>); and
switching (<NUM>) from a standby mode to an active mode.