Patent Description:
In some modern data centers, the function of a device or appliance may not be tied to a specific, fixed hardware configuration. Rather, processing, memory, storage, and accelerator functions may in some cases be aggregated from different locations to form a virtual "composite node. " A contemporary network may include a data center hosting a large number of generic hardware server devices, contained in a server rack for example, and controlled by a hypervisor. Each hardware device may run one or more instances of a virtual device, such as a workload server or virtual desktop.

<CIT> relates to network functions virtualization ("NFV") leveraging unified traffic management and real-world event planning. A virtual network traffic management system ("VNTMS") can receive, via a traffic congestion monitor, from a virtual network function ("VNF"), a traffic congestion indicator that includes a traffic congestion state indicative of network traffic congestion experienced by the VNF. The VNTMS can analyze the traffic congestion indicator to determine a traffic management action to be taken to alleviate at least a portion of the network traffic congestion experienced by the VNF, and instruct a traffic congestion action responder to provide the traffic management action to the VNF.

<CIT> describes a method comprising: determining that a network function virtualization infrastructure (NFVI) failure occurs, and acquiring self-healing policy information, where the self-healing policy information indicates whether to perform self-healing operation for the NFVI failure. Self-healing operation is performed for the NFVI failure when the self-healing policy information indicates that the self-healing operation is to be performed for the NFVI failure, where the self-healing strategy information comprises updated policy information.

"Network Functions Virtualisation (NFV); Resiliency Requirements" (ETSI DRAFT; ETSI GS NFV-REL <NUM>, URL:docbox. org\ISG\NFV\Open\Publications pdf\Specs-Reports\NFVREL001v1. <NUM>- GS - Resiliency Requirements. pdf) relates to network and service resiliency in a virtualized network environment.

The invention is defined in the claims. In the following description, any embodiment referred to and not falling within the scope of the claims is merely an example useful to the understanding of the invention.

The present disclosure is best understood from the following detailed description when read with the accompanying figures. It is emphasized that, in accordance with the standard practice in the industry, various features are not necessarily drawn to scale, and are used for illustration purposes only. Where a scale is shown, explicitly or implicitly, it provides only one illustrative example. In other embodiments, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

The following disclosure provides many different embodiments, or examples, for implementing different features of the present disclosure. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. Further, the present disclosure may repeat reference numerals and/or letters in the various examples. Different embodiments may have different advantages, and no particular advantage is necessarily required of any embodiment.

A contemporary data center or telecommunication backend may employ techniques such as software defined networking (SDN) and network function virtualization (NFV) to provide flexibility, scalability, and resiliency in a data center. Using SDN and NFV, available hardware resources can be dynamically allocated and scaled out to provide an optimized networking platform. As network demands change, and as workloads shift, nodes such as load balancers, firewalls, antivirus services, deep packet inspection (DPI) services, routers, and other network functions can be dynamically allocated according to the present demands. If demand increases, more resources can be allocated, while if demand falls, resources can be spun down to save power and operating costs.

According to the distributed nature of a contemporary computing architecture, a management and orchestration (MANO) system may not be locally co-located with the nodes it controls. For example, a data center in Dublin, Ireland may be providing network services according to network function virtualization infrastructure (NFVI) definitions provided by a management console in San Francisco, California. As long as connectivity remains robust, there is no issue with a node in one part of the world being controlled by a management system in another part of the world.

However, in severe cases a local data center may be cut off from its management console. In that case, the local platform may not have a method of providing resiliency to local virtual network functions (VNFs) that may encounter errors while the disruption persists. For example, in the previous case where a Dublin data center is managed by a San Francisco management console, a natural disaster in either of those places could cause the data center to be cut off from the management console. This may be even more problematic if the natural disaster is in the region of the data center, as that data center may be expected to provide certain emergency services during the natural disaster. For example, some jurisdictions lawfully require telecommunication providers to maintain a designated uptime, and to provide persistent emergency communication services even in the case of natural disasters. If the data center cannot reach the management console, then an error in a VNF may cause a disruption of service that is unacceptable. Even worse, in the face of such a natural disaster, there is an increased likelihood that the local data center will encounter errors, as some equipment may be disrupted or damaged. Thus, it is all the more important for this data center to be able to maintain resiliency. Furthermore, in some extreme cases, a failed node may begin "chattering. " Chattering occurs when a failed node continues to attempt to operate even after it is no longer functioning correctly. This failed node may continue to receive packets on its ingress interfaces, process those packets incorrectly, and then send out garbage on its egress interfaces. This garbage can then be propagated out to the network, and in severe cases, a single failed, chattering node can bring down an entire network.

It is therefore advantageous to provide a local resiliency controller on a local hardware platform that can handle resiliency functions according to a resiliency policy or other directives received from the management system. Having a local resiliency controller not only provides for resiliency in the case of catastrophic disasters, but even in the case of ordinary node failures, the localized resiliency controller can respond to a failure more quickly than a remote management console.

The local resiliency controller of the present specification maximizes the availability of an NFV-based service on a local platform using a multi-domain aware resiliency controller to automatically restore service on the platform, even when the management system is unavailable. The resiliency service described herein uses
a preloaded resiliency policy received from the management system.

This enables a local platform to automatically apply immediate corrective local actions based on knowledge of the local physical domain resources, virtual resources, and best recovery methods for the NFV service. Advantageously, the local resiliency controller may have a more detailed or more immediate view of available local resources, and thus, in some cases, may be able to customize the resiliency policy to work best on the local platform. The local resiliency service can minimize outage times and revenue lost as a result of a fault of an NFV-based platform. A localized resiliency service may also help to ensure compliance with jurisdictional requirements for service uptime, including in the event of natural disasters.

By way of illustration, a typical mobile data service may experience approximately five outages or service deteriorations in one year, meaning approximately one every other month. In the aggregate, these outages can accrue into the billions of dollars of revenue losses, despite the several operators' efforts to minimize and plan for such outages.

An average telecommunication service provider may spend between <NUM> and <NUM>% of its annual revenue repairing network issues. The primary drivers of such outages include, by way of nonlimiting example, congestion, physical link failures, overloading, and network equipment failures. As discussed above, service providers may typically employ a MANO, which uses a management stack in conjunction with analytics and an energy management system (EMS) to detect and correct hardware issues. Service providers may also use custom proprietary agents, which may be used on Big Iron appliances or original design manufacturer (ODM) platforms to help detect and correct hardware issues. Furthermore, cloud service providers (CSPs) may use dedicated hardware and software solutions to provide resiliency.

However, a MANO may fall short in detecting and associating physical and virtual faults. MANO, and other existing local platform controllers, may not successfully correlate physical and virtual domains to the appropriate faults and resource domains to trigger corrective actions.

Furthermore, a remote MANO may not be responsive enough to meet latency demands. For example, a telecommunications appliance may have a resiliency recovery scheme that requires restoration of a service from in between <NUM> milliseconds, up to one second. Beating these recovery times may require novel approaches as new fault domains arise in platforms, including hypervisor, virtual switch, and virtual resources.

Some existing solutions also do not provide service-specific resiliency. Local platform controllers may not be configured with resiliency policies specific to the service provided by the VNF being deployed on a platform.

It is also advantageous to provide survivability during management outages. If the management system (such as MANO) fails, it may be necessary to continue providing the best possible service until central management is restored. Failure to beat minimum local standards of service, or to continue providing resiliency during a loss of connectivity with the MANO, may result in penalties to a service provider, or action by watchdog groups for failure to meet advertised availability and speed benchmarks.

Scalability is also a concern. Some existing NFV systems may lack scalable resiliency to provide the best local action to handle storms of widespread faults, which can occur during a natural disaster or denial of service attack. The scalability problem may apply to multiple single platforms, as in conventional systems and disaggregated systems such as Intel@ Rack Scale Architecture (RSA) and Rack Scale Design (RSD).

The custom proprietary agents discussed above may also lack a view of the virtual domain running on the platform, and how these hardware failures impact VNFs running on the platform.

Furthermore, while cloud services have provided solutions to attempt to address these concerns, they have at least in some cases failed to meet carrier grade requirements for core telecommunications infrastructure, such as "five nines" (commonly defined as <NUM>%) reliability.

In an example of the present specification, a local resiliency service is provided by a local resiliency controller. The local resiliency controller operates a management interface to communicatively couple to a management system, such as an operational support system, business support system (BSS), MANO, or similar, to receive a resiliency policy or other similar directives. The local resiliency controller preloads the resiliency policy on the platform, and is thus prepared to handle physical faults, virtual faults, and connectivity faults, which may be uniquely identified for each VNF. In the event of a fault, the resiliency controller automatically applies immediate corrective local actions based on the local physical domain resources, virtual resources, and best available recovery methods for the NFV service. The resiliency controller can then operate the management interface to notify the management system of the resiliency action taken. Note that notification may be via a one-directional message on the management interface, wherein the one-directional message is a message that may receive but does not require an acknowledgement or response from the management controller. Advantageously, this one-directional message does not itself become a bottleneck if connectivity to the management controller is lost.

The resiliency control of the present specification also differentiates fault domains and provides policies to handle management connectivity, virtual switch connectivity, physical resource failures, and virtual resource failures. The resiliency controller can also identify virtual resource behavior, denial of service behavior, and process behavior, and can mitigate these without management intervention.

Furthermore, the resiliency controller of the present specification can inspect the results of a resiliency action, and determine whether the resiliency action was successful. If the resiliency action is unsuccessful, the local resiliency controller can notify the management controller with a one-directional message on the management interface, and can then place the affected service into a failsafe mode. Placing the affected service or node into a failsafe mode may include, by way of example, disabling one or more egress ports (either hardware egress ports or virtual egress ports) of the affected node, to ensure that the node cannot begin chattering and negatively affect other nodes in the network.

A contemporary computing platform, such as a hardware platform provided by Intel@ or similar, may include a capability for monitoring device performance and making decisions about resource provisioning. For example, in a large data center such as may be provided by a CSP, the hardware platform may include rackmounted servers with compute resources such as processors, memory, storage pools, accelerators, and other similar resources.

By way of nonlimiting example, as used in the present specification, a processor includes any programmable logic device with an instruction set. Processors may be real or virtualized, local or remote, or in any other configuration. A processor may include, by way of nonlimiting example, an Intel@ processor (e.g., Xeon®, Core™, Pentium®, Atom®, Celeron®, x86, or others). A processor may also include competing processors, such as AMD (e.g., Kx-series x86 workalikes, or Athlon, Opteron, or Epyc-series Xeon workalikes), ARM processors, or IBM PowerPC and Power instruction set architecture (ISA) processors, to name just a few.

As used herein, "cloud computing" includes network-connected computing resources and technology that enables ubiquitous (often worldwide) access to data, resources, and/or technology. Cloud resources are generally characterized by great flexibility to dynamically assign resources according to current workloads and needs. This can be accomplished, for example, via virtualization, wherein resources such as hardware, storage, and networks are provided to a virtual machine (VM) via a software abstraction layer.

As used in the present disclosure, a VM is an isolated partition within a computing device that allows usage of an operating system and other applications, independent of other programs on the device in which it is contained. Containerization is a method of providing system resources wherein instances of network functions are provided in "containers" that are separated from one another, but that share underlying operating system, memory, and driver resources. VMs, containers, and similar may be generically referred to as "guest" systems.

A system and method for providing localized service resiliency will now be described with more particular reference to the attached FIGURES. It should be noted that throughout the FIGURES, certain reference numerals may be repeated to indicate that a particular device or block is wholly or substantially consistent across the FIGURES. This is not, however, intended to imply any particular relationship between the various embodiments disclosed. In certain examples, a genus of elements may be referred to by a particular reference numeral ("widget <NUM>"), while individual species or examples of the genus may be referred to by a hyphenated numeral ("first specific widget <NUM>-<NUM>" and "second specific widget <NUM>-<NUM>").

<FIG> is a block diagram of network function virtualization (NFV) management and orchestration without localized service resiliency, according to one or more examples of the present specification.

The block diagram of <FIG> illustrates management and orchestration that may occur according to existing or known methods. Note that the local resiliency service of the present specification does not necessarily replace existing management infrastructure, but rather may supplement that infrastructure by providing localized resiliency services that can operate with less latency than remote resiliency services, and that can continue to operate, even in the event of failure of the management interface.

In the example of <FIG>, an NFV <NUM> is running on a local node including hardware resources <NUM>. Hardware resources <NUM> may include, by way of illustrative and nonlimiting example, computing hardware <NUM>, storage hardware <NUM>, and network hardware <NUM>. Hardware resources <NUM> operate a virtualization layer <NUM>, which then provides virtual resources <NUM> to NFVI <NUM>. Virtual resources <NUM> may include virtual computing <NUM>, virtual storage <NUM>, and virtual network <NUM>. Using these virtual resources <NUM>, NFVI <NUM> can instantiate one or more virtual machines, which can provide one or more VNFs according to network needs.

NFVI <NUM> may provide a VNF architecture <NUM>. VNF architecture <NUM> may include, by way of nonlimiting example, a plurality of EMSs <NUM>. These may include, namely, EMS <NUM><NUM>-<NUM>, EMS <NUM><NUM>-<NUM>, and EMS <NUM><NUM>-<NUM>. VNF architecture <NUM> may also provide a plurality of VNFs <NUM>, in this example namely VNF <NUM><NUM>-<NUM>, VNF <NUM><NUM>-<NUM>, and VNF <NUM><NUM>-<NUM>.

NFVI <NUM> may be managed by virtualized infrastructure managers <NUM>. Note that while virtualized infrastructure managers <NUM> are illustrated in this example as being part of NFV MANO <NUM>, virtualized infrastructure managers <NUM> may not necessarily be co-located with the rest of NFV MANO <NUM>. Instead, virtualized infrastructure managers <NUM> may be local software services that provide localized infrastructure management. In contrast, VNF managers <NUM> and orchestrator <NUM> may in some cases be remote resources of NFV MANO <NUM>. There is also shown here an operational support system/business support system (OSS/BSS) <NUM>, which can provide service, VNF, and infrastructure descriptions <NUM> to NFV MANO <NUM>.

For purposes of this specification, OSS/BSS <NUM> and NFV MANO <NUM> may be considered jointly to be a management system, and NFVI <NUM> communicatively couples to the management system via a management interface. This management interface may be, by way of nonlimiting example, a dedicated hardware management interface, an Ethernet or other network connection, or any other data or communication interface that allows NFVI <NUM> to communicate with the management system.

The management system may have several responsibilities, including MANO and NFV MANO, and in this example is broken up into three functional blocks.

Orchestrator <NUM> is responsible for onboarding of new network services and VNF packages. According to examples of the present specification, orchestrator <NUM> may also handle network service lifecycle management, global resource management, and validation and authorization of NFVI resource requests.

VNF managers <NUM> may be responsible for overseeing the lifecycle management of VNF instances, coordinating and adapting roles for configuration, and event reporting between NFVI and enterprise network management system (E/NMS).

Virtualized infrastructure manager (VIM) <NUM> may be responsible for controlling and managing the NFVI compute, storage, and network resources.

As discussed above, difficulties may occur when NFVI <NUM> is unable to reach the management system via the management interface. Furthermore, even when NFVI <NUM> can reach the management system via the management interface, response to a fault may be relatively slow.

For example, an error flow is shown in <FIG>. At operation <NUM>, a fault occurs within NFVI <NUM>. In this example, the fault is illustrated as a failure of network hardware <NUM>, but the fault may be any fault, including within hardware resources <NUM>, virtual resources <NUM>, or within any of VNFs <NUM>.

NFVI <NUM> notifies virtualized infrastructure manager <NUM> of the fault. At operation <NUM>, virtualized infrastructure manager <NUM> queries VNF managers <NUM> for a solution to the fault identified in operation <NUM>.

At operation <NUM>, VNF managers <NUM> query orchestrator <NUM> for a solution for the fault.

At operation <NUM>, orchestrator <NUM> consults its internal resiliency policy to identify a solution for the encountered fault. Orchestrator <NUM> then provides the solution to VNF managers <NUM>.

VNF managers <NUM>, acting on the resiliency policy provided by orchestrator <NUM>, then operates on VNF architecture <NUM> to effect a resiliency action to mitigate the issue.

<FIG> is a flowchart of providing service resiliency without a localized resiliency service, according to one or more examples of the present specification.

Method <NUM> illustrates remote resiliency as may be provided in the illustration of <FIG> when a local resiliency service is not available.

In block <NUM>, a VNF is providing VNF services.

However, the VNF may encounter a problem. For example, the VNF may have a particular reliability, availability, and serviceability (RAS) target. An RAS target may include, for example, a requirement to produce correct output within a temporal window, a requirement to be operational a certain percentage of time within a temporal window, and simplicity or speed of repair or maintenance. In block <NUM>, for example, this may be a hardware fault, a RAS fault, or connectivity fault. Alternatively, in block <NUM>, the VNF may encounter a virtual resource/virtual switch/virtual network/virtual compute or storage fault.

The operations of blocks <NUM> and <NUM> then take place on the management system, which may be remote from the local system.

In block <NUM>, the NFVI notifies the management and analytics system.

In block <NUM>, the management system identifies a preferred resiliency action to mitigate the issue, then instructs the local platform to take corrective action based on the type of service and fault severity.

In block <NUM>, the VNF service may be restored. Following on page connector <NUM>, returning to block <NUM>, the VNF may then continue to provide its service.

<FIG> is a block diagram of a local platform with a local NFV resiliency controller, according to one or more examples of the present specification.

In the example of <FIG>, local platform <NUM> includes a hardware platform <NUM>, physical resources such as network, compute, and storage resources <NUM>, virtual resources <NUM>, such as virtual network, compute, and storage resources, a VNF <NUM> and a virtual switch (vSwitch) <NUM>. In addition, a telemetry agent <NUM> is provided. Telemetry agent <NUM> collects telemetry and fault information about local platform <NUM> and provides telemetry to resiliency controller <NUM>.

Telemetry for local platform <NUM> may include, by way of nonlimiting example, information about hardware and/or software failures, bandwidth, congestion, RAS factors, uptime, downtime, boot time, bottlenecks, oversubscription of resources, undersubscription of resources, or any other relevant factor that may be valuable in applying a resiliency policy.

Local platform <NUM> also provides NFV resiliency services via NFV resiliency controller <NUM>.

NFV resiliency controller <NUM> is a local resiliency controller, which may include, by way of nonlimiting example, software to operate on hardware platform <NUM>, software that may operate on a dedicated co-processor, firmware, or a hardware device such as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other dedicated hardware block. In some examples, all or part of local platform <NUM> may be provided in an integrated circuit or system-on-a-chip (SoC), in which case NFV resiliency controller <NUM> may be provided as a dedicated intellectual property (IP) block.

In some embodiments of the present disclosure, the resiliency controller may be implemented in software on the central processing unit (CPU) complex, on the Intel@ Innovation Engine (IE), the Intel@ management engine (ME), an FPGA, an ASIC, or a dedicated hardware IP block.

Hardware implementations outside of the CPU complex have some advantages, including that the resiliency controller is isolated from software faults, and the resiliency controller is protected from unrecoverable machine check architecture (MCA) events that may halt the CPU. In the case that resiliency controller <NUM> is implemented in software on the operating core, other steps may be taken to protect resiliency controller <NUM> from software faults. These include placing resiliency controller <NUM> in a highly privileged program privilege ring, or even placing it in software that operates either independently or outside of the main operating system software, by way of nonlimiting example.

In cases where resiliency controller <NUM> is instantiated in the CPU complex, recoverable MCA events may be handled by the resiliency controller. However, to protect against "<NUM>-strike" or unrecoverable MCA events, an off complex resiliency controller may be used.

Examples of recoverable MCA events include, by way of nonlimiting example:.

Furthermore, in some cases, hardware platform <NUM> may include one or more cores, and NFV resiliency controller <NUM> may include hooks into a core architecture, such as microcode in the core that is designed specifically to handle functions of NFV resiliency controller <NUM>.

NFV resiliency controller <NUM> may be configured to scan the hardware of local platform <NUM> to establish a hardware topology connected graph. An illustration of a hardware topology connected graph is shown in <FIG>. NFV resiliency controller <NUM> may receive telemetry and fault data from VNF <NUM> and vSwitch <NUM>. It may also receive telemetry and fault data from physical resources <NUM> and virtual resources <NUM>. From management system <NUM>, NFV resiliency controller <NUM> may receive a resiliency policy, as well as VNF descriptors and network service descriptors. At appropriate times, NFV resiliency controller <NUM> may communicate with management system <NUM> via management interface <NUM>. In some cases, communication with management system <NUM> on management interface <NUM> may include one-directional communications, meaning communications that may receive but do not require an acknowledgement or response from management system <NUM>. Such one-directional communications can be important in cases where connectivity with management system <NUM> via management interface <NUM> is uncertain, allowing resiliency controller <NUM> to provide notifications to management system <NUM>, while preventing those communications from becoming a bottleneck as resiliency controller <NUM> waits for a response that may or may not come.

By way of illustrative example, NFV resiliency controller <NUM> receives, stores, and executes on the following:.

NFV resiliency controller <NUM> also carries out local corrective actions for multiple fault domain physical resources, virtual resources, virtual switching, and management interfaces (regardless of connectivity to management system <NUM>). This corrective action may include recovery from denial of service attacks and software faults.

Advantageously, a local NFV resiliency controller <NUM> provides immediate remediation of failures, or applications of management policies within time frames that better meet requirements, including telecommunications uptime requirements. Thus, service downtime is minimized by quickly applying VNF management policies. This becomes an even greater advantage when an operator wishes to outsource aspects of OSS/BSS into a cloud domain. Note that many mission-critical telecommunication services require a fault detection and correction time in the range of <NUM> to <NUM> milliseconds. This can be difficult to achieve when resiliency control resides off-site.

In the case of failure of management system <NUM>, or failure of connection to management system <NUM>, full network failures can be avoided because managed network elements can continue to operate with appropriate policies and actions according to resiliency controller <NUM>. Local platform <NUM> may also be configured with secondary or backup MANO interfaces, should the primary MANO fail.

This also enables service-specific failsafe behavior on the platform. When NFV resiliency controller <NUM> takes a resiliency action, for example on VNF <NUM>, it can inspect the result to ensure that the action succeeded. If the action does not succeed, then resiliency controller <NUM> can place VNF <NUM> into a failsafe mode, which may include disabling one or more egress ports to ensure that VNF <NUM> is not spewing garbage packets out to the network and possibly bringing down other nodes. This can prevent downstream chatter and enable faster system recovery by putting platforms and services in good recoverable states.

NFV resiliency controller <NUM> may also be configured to survive or recover from events such as network failure or managing power failure.

In the event of a network failure, resiliency controller <NUM> may be notified by the telemetry and faults provided by the network interface card (NIC), as illustrated in <FIG> between block <NUM> and block <NUM>. Resiliency controller <NUM> may then take action defined by the network failure policy previously provided by management system <NUM>. For example, the network failure recovery action may be to switch network ports and allow a secondary port to act as management interface <NUM>.

In the event of a power failure recovery, resiliency controller <NUM> may be notified of the unexpected shutdown by reading the operating system (OS) syslog on recovery. This log can contain the crash indication on power outage, or an unexpected shutdown status indication. Resiliency controller <NUM> may then take action as defined by the NFVI power failure recovery policy previously provided by management system <NUM> via management interface <NUM>.

Note that in various embodiments, management system <NUM> and resiliency controller <NUM> may have distinct responsibilities.

Management system <NUM> may work with templates for standard VNFs, and gives users the ability to pick and choose from existing NFVI resources to deploy their platforms or elements. Management system <NUM> informs resiliency controller <NUM> of VNF deployments and lifecycle changes (e.g., VNFs added or removed from the platform).

Management system <NUM> also selects and deploys resiliency policy, VNF policy, and NFVI policy, and provides these to resiliency controller <NUM> via management interface <NUM>. These may be selected to match the service level agreements (SLAs) of the VNFs selected and deployed on local platform <NUM>.

Resiliency controller <NUM> is responsible for generating and maintaining a topology hardware dependency graph. NFVI resiliency controller <NUM> receives and stores the resiliency policy, VNF policy, and NFVI policy provided by management system <NUM> via management interface <NUM>, and associates hardware resources to VNFs deployed as part of the management system lifecycle management. Resiliency controller <NUM> continually monitors the fault domains on local platform <NUM>. On detection of a fault in one of the fault domains, resiliency controller <NUM> triggers the associated resiliency policy action for the associated VNF.

In the case of a fault associated with an NFVI domain, such as a vSwitch or management interface, resiliency controller <NUM> triggers the associated NFVI resiliency policy for that fault domain.

<FIG> and <FIG> are a flowchart of methods of providing localized NFV resiliency responsive to a hardware/reliability, availability and serviceability (RAS)/storage fault or a virtual resource fault, according to one or more examples of the present specification.

Starting in block <NUM>, the VNF is providing a service according to its designated network function.

In block <NUM>, the VNF encounters a hardware, RAS, or storage fault. Alternatively, following off page connector <NUM> to <FIG>, the VNF may encounter a virtual resource fault such as a virtual compute or storage fault.

Following the path of the hardware fault of block <NUM>, in block <NUM>, the resiliency controller identifies the VNF associated with the hardware fault.

In block <NUM>, the resiliency controller triggers a resiliency policy action for the VNF, based for example on the VNF identity, the NFVI for the VNF, the fault type, and the severity of the fault.

In block <NUM>, the resiliency controller takes a resiliency action. In some embodiments, this may include restarting the VNF as necessary.

In block <NUM>, the resiliency controller notifies the management system of the resiliency action taken. As described in previous FIGURES, this may be a one-directional message, which ensures that the notification does not itself become a bottleneck. In decision block <NUM>, the resiliency controller determines whether the attempted resiliency action has successfully repaired the VNF. If the repair was successful, then returning to block <NUM>, the VNF continues providing its service.

Returning to block <NUM>, if the repair was not successful, then in block <NUM>, the resiliency controller may take a VNF-specific failsafe action. This may include, for example, placing the VNF in a failsafe mode in which one or more egress ports are disabled, thus preventing downstream chattering. In block <NUM>, the VNF is in failsafe mode and the method is done.

Returning to off page connector <NUM>, on <FIG>, the VNF has encountered a virtual resource fault such as a virtual compute or virtual storage fault.

In block <NUM>, the resiliency controller identifies the associated VNF.

In block <NUM>, the resiliency controller triggers a resiliency policy action for the VNF, based on, for example, the VNF identity as defined in the NFVI, the fault type, and the fault severity.

In block <NUM>, the resiliency controller takes the resiliency action. As necessary, this may include restarting the VNF. In block <NUM>, the resiliency controller notifies the management system of the resiliency action. This may be done by way of a one-directional message. Control then follows off page connector <NUM> back to block <NUM>. In block <NUM> the management system is notified of the resiliency action, which again may be conveyed via one-directional message. In decision block <NUM>, the resiliency controller determines whether the action was successful. If the action was successful, then in block <NUM>, the VNF continues to provide its service.

If the resiliency action was not successful, then in block <NUM>, the resiliency controller takes a VNF-specific failsafe action. This may include disabling egress ports to avoid downstream chattering.

In block <NUM>, the VNF is in failsafe mode and the method is done.

<FIG> is a flowchart of providing localized resiliency service responsive to a virtual switching fault, according to one or more examples of the present specification.

In block <NUM>, the vSwitch is providing its virtual switching services. In block <NUM>, the vSwitch encounters a virtual switching fault.

In block <NUM>, responsive to the virtual switching fault, the resiliency controller triggers a resiliency policy action for the vSwitch. This may depend on the identity of the vSwitch, the fault type, and the fault severity.

In block <NUM>, the resiliency controller takes the resiliency action, which may include as necessary restarting the vSwitch.

In block <NUM>, the resiliency controller notifies the management system of the resiliency action. As in previous examples, this may include a one-directional message.

In decision block <NUM>, the resiliency controller determines whether the resiliency action was successful. If the resiliency action was successful, then in block <NUM>, the vSwitch continues providing its virtual switching services.

In block <NUM>, if the resiliency action was not successful, then in block <NUM>, the resiliency controller takes a vSwitch-specific failsafe action. This can include disabling ingress or egress ports of the vSwitch, and may also include marking the vSwitch as unavailable so that nodes in the system do not attempt to use the vSwitch. In block <NUM>, the vSwitch is in failsafe mode, and the method is done.

<FIG> is a flowchart of providing localized resiliency service responsive to a platform management connectivity fault, according to one or more examples of the present specification.

In block <NUM>, the resiliency controller has a good management interface connection to the platform management system or other management system.

In block <NUM>, the resiliency controller experiences a platform management connectivity fault.

In block <NUM>, the connectivity fault triggers a resiliency policy action for the platform management interface. This may depend on the fault type and the fault severity.

In block <NUM>, the resiliency controller takes a resiliency action responsive to the failure. As necessary, this may include a port switchover, or other port reconfiguration to regain connectivity to the management system.

In block <NUM>, the resiliency controller attempts to notify the management system of the resiliency action. As before, this may be a one-directional message. The use of a one-directional message in this instance is particularly useful, because the identified fault is in the management interface itself. Thus, there is higher danger in this case of the notification becoming a bottleneck if the system waits for an acknowledgement or response from the management system.

In block <NUM>, the resiliency controller determines whether it has successfully reconnected with the platform management system. If reconnection was successful, then in block <NUM>, the platform management system is connected.

If the reconnection was not successful, then in block <NUM> the resiliency controller may take a management interface-specific failsafe action. This may include disabling ports, or simply reverting to autonomous local mode, wherein the resiliency controller continues to manage VNFs according to the last management policy received, until connectivity is reestablished with the management system.

In block <NUM>, the management interface is in failsafe mode, and the method is done.

<FIG> is a block diagram of a hardware dependency graph, according to one or more examples of the present specification. Hardware dependency graph <NUM> is generated by a resiliency controller such as resiliency controller <NUM> of <FIG>.

Hardware dependency graph <NUM> begins with a root node <NUM>, which may for example represent a local platform such as local platform <NUM> of <FIG>. Root <NUM> may be divided into two or more nonuniform memory access (NUMA) nodes <NUM>, in this case specifically NUMA node <NUM> and NUMA node <NUM>. A NUMA architecture may provide, for example, separate memory for each processor, which can help to ameliorate the performance hit that may occur if a plurality of processors simultaneously try to access a single memory address.

Each NUMA node <NUM> may be divided into a plurality of components such as DIMMs <NUM>, PCIe devices <NUM>, CPUs <NUM>, and last level caches (LLCs) <NUM>.

For example, NUMA node <NUM> includes memory banks DIMM <NUM> through DIMM N, as well as PCIe device x. NUMA node <NUM> also includes a plurality of CPUs, namely CPU <NUM> through CPU N. CPU <NUM> through CPU N may share LLC <NUM>. Note that while a single PCIe device <NUM>, namely PCIe device x, is illustrated in NUMA node <NUM>, this is by way of nonlimiting and illustrative example only, and a plurality of PCIe devices may be provided on any NUMA node <NUM> as appropriate to the embodiment.

NUMA node <NUM> includes CPU N+<NUM> through CPU M, which share LLC N. NUMA node <NUM> also includes PCIe device y, as well as memory banks DIMM N+<NUM> through DIMM M.

Hardware dependency graph <NUM> allows resiliency controller <NUM> (<FIG>) to keep track of which hardware devices are dependent on each other. This enables the resiliency controller to trace hardware faults.

<FIG> is a block diagram of virtual network functions hosted on a hardware dependency graph, according to one or more examples of the present specification.

<FIG> illustrates that hardware dependency graph <NUM> may be extended by also graphing which VNFs <NUM> are hosted on which hardware platforms. For example, in the example of <FIG>, VNF <NUM> is hosted on CPU <NUM>, and is also allocated PCIe device x.

Similarly, VNF <NUM> is hosted on CPU M, and is allocated PCIe device y.

Note that in this example, each VNF <NUM> is hosted on a single CPU <NUM>, and is allocated a single PCIe device <NUM>, but this is a nonlimiting and illustrative example only. In many cases, a plurality of VNFs may be hosted on a single CPU, while in other cases, a VNF may be allocated a plurality of CPUs. Furthermore, a VNF <NUM> may have any number of PCIe devices <NUM>, between <NUM> and the maximum number of PCIe devices supported by the platform.

It should also be noted that the illustration of certain hardware elements on hardware dependency graph <NUM> is a nonlimiting and illustrative example. The illustration of certain hardware elements on hardware dependency graph <NUM> does not imply that all of these elements need to be present in every hardware platform, and it should also be understood that some platforms will also host other hardware elements, such as accelerators, coprocessors, and peripheral devices that may be connected via other bus types.

By way of illustration, VNF <NUM> may be a virtual firewall, while VNF <NUM> may be a virtual router.

Under normal operating conditions, the status of both VNF <NUM> and VNF <NUM> is "green. " Each VNF is bound to a socket on the hardware platform. These two VNFs may, for example, pass information via an Intel@ Quick-Path Interconnect (QPI) link between the two sockets. As illustrated in <FIG>, a resiliency controller <NUM> hosted on the local hardware platform (e.g., at the same logical level as root <NUM>) may communicate with both a management system such as a MANO via a management interface, and may also have a local telemetry interface to communicate with a local telemetry agent <NUM>.

Hardware dependency graph <NUM> can be used to solve the issue of detecting and associating physical and virtual faults. For example, a QPI failure between CPU M and CPU <NUM> may result in halving the bandwidth between the two VNFs <NUM>. This results in a status "red," resulting in a warning. From the central MANO perspective, it may appear to be simply a degradation in performance of the VNFs. The MANO may not be able to correlate the hardware QPI failure with the VNF performance degradation or SLA violation. But resiliency controller <NUM> may receive via telemetry agent <NUM> information about the failure of the QPI link. Thus, resiliency controller <NUM> can address the issue with a detailed view or insight into the hardware topology, including mapping the virtual domain (i.e., VNFs <NUM>) to the specific hardware they are running on. Resiliency controller <NUM> also has an understanding of what a healthy platform is and what is not via the NFVI policies that are provided to it by the management system.

The VNF placement policies for VNFs <NUM> instantiated on a platform may be pushed from the management system to resiliency controller <NUM>. Another part of the policies passed to resiliency controller <NUM> may include VNF monitoring parameters. By way of nonlimiting example:
monitoring parameter :{
Name: {presence: required, value: CPU-utilization,},
Value: {presence: required,
Action: [<NUM>, down, CPU_Low, Scale_Down_Action],
Action: [<NUM>, up, CPU_High, Scale_Up_Action]
}
}.

Applications of specific policies are illustrated in <FIG>, <FIG>, and <FIG>.

<FIG>, <FIG>, and <FIG> are examples of resiliency actions taken according to a hardware dependency graph, according to one or more examples of the present specification.

Turning to <FIG>, it is illustrated that resiliency controller <NUM> can use hardware dependency graph <NUM> to address a specific hardware failure. In this case, DIMM N, which is relied on by CPU <NUM>, goes bad. This may include a complete failure of the DIMM, or a degradation in performance, such as by failure of particular memory blocks, or other partial failure. The failure of DIMM N causes a failure of NUMA node <NUM>. This in turn impacts the performance of VNF <NUM>, which in this example is a virtual firewall.

Because resiliency controller <NUM> has access to the placement needs and lifecycle management policies for VNF <NUM>, it is able to isolate the failure to NUMA node <NUM>. Resiliency controller <NUM> may also know the capacity of the hardware platform, and may determine that it is feasible to switch VNF <NUM> to NUMA node <NUM>. Advantageously, resiliency controller <NUM> may be able to migrate VNF <NUM> to NUMA node <NUM> before any noticeable performance impact occurs for VNF <NUM>.

<FIG> and <FIG> illustrate an example of a scale up action according to one or more examples of the present specification. The scale up action of <FIG> and <FIG> may take place when resiliency controller <NUM> identifies that VNF <NUM> has exceeded a threshold for CPU usage on CPU <NUM>. Specifically, resiliency controller <NUM> determines that CPU <NUM> is running at <NUM>% capacity. Resiliency controller <NUM> previously received a management policy via the management interface. The management policy specified that when a CPU exceeds <NUM>% utilization, the resiliency controller <NUM> is to take a scale up action. Thus, as illustrated in <FIG>, resiliency controller <NUM> allocates both CPU <NUM> and CPU N to VNF <NUM>. Because VNF <NUM> now has two CPUs allocated to it, each CPU is utilized at only <NUM>%. Note that the management policy also specified that a CPU with under <NUM>% utilization is underutilized. But resiliency policy <NUM> includes sufficient intelligence to determine that, although both CPUs are underutilized at <NUM>%, deallocating one of the CPUs would cause the remaining CPU to be overutilized at <NUM>%. Thus, resiliency controller <NUM> does not deallocate either CPU for VNF <NUM>. However, resiliency controller <NUM> may determine that one or both CPUs have sufficient remaining capacity that they may host some other VNF or other virtual function. Particularly, a lightweight VNF may be additionally hosted on either CPU, which allows both CPUs to realize optimal utilization.

<FIG> is a block diagram of selected components of a data center with connectivity to network <NUM> of a CSP <NUM>, according to one or more examples of the present specification. Embodiments of network <NUM> disclosed herein may be adapted or configured to provide the method of providing localized service resiliency according to the teachings of the present specification. CSP <NUM> may be, by way of nonlimiting example, a traditional enterprise data center, an enterprise "private cloud," or a "public cloud," providing services such as infrastructure as a service (IaaS), platform as a service (PaaS), or software as a service (SaaS). In some cases, CSP <NUM> may provide, instead of or in addition to cloud services, high-performance computing (HPC) platforms or services. Indeed, while not expressly identical, HPC clusters ("supercomputers") may be structurally similar to cloud data centers, and unless and except where expressly specified, the teachings of this specification may be applied to either.

CSP <NUM> may provision some number of workload clusters <NUM>, which may be clusters of individual servers, blade servers, rackmount servers, or any other suitable server topology. In this illustrative example, two workload clusters, <NUM>-<NUM> and <NUM>-<NUM> are shown, each providing rackmount servers <NUM> in a chassis <NUM>.

In this illustration, workload clusters <NUM> are shown as modular workload clusters conforming to the rack unit ("U") standard, in which a standard rack, <NUM> inches wide, may be built to accommodate <NUM> units (42U), each <NUM> inches high and approximately <NUM> inches deep. In this case, compute resources such as processors, memory, storage, accelerators, and switches may fit into some multiple of rack units from one to <NUM>.

Each server <NUM> may host a standalone operating system and provide a server function, or servers may be virtualized, in which case they may be under the control of a virtual machine manager (VMM), hypervisor, and/or orchestrator, and may host one or more virtual machines, virtual servers, or virtual appliances. These server racks may be collocated in a single data center, or may be located in different geographic data centers. Depending on the contractual agreements, some servers <NUM> may be specifically dedicated to certain enterprise clients or tenants, while others may be shared.

The various devices in a data center may be connected to each other via a switching fabric <NUM>, which may include one or more high speed routing and/or switching devices. Switching fabric <NUM> may provide both "north-south" traffic (e.g., traffic to and from the wide area network (WAN), such as the internet), and "east-west" traffic (e.g., traffic across the data center). Historically, north-south traffic accounted for the bulk of network traffic, but as web services become more complex and distributed, the volume of east-west traffic has risen. In many data centers, east-west traffic now accounts for the majority of traffic.

Furthermore, as the capability of each server <NUM> increases, traffic volume may further increase. For example, each server <NUM> may provide multiple processor slots, with each slot accommodating a processor having four to eight cores, along with sufficient memory for the cores. Thus, each server may host a number of VMs, each generating its own traffic.

To accommodate the large volume of traffic in a data center, a highly capable switching fabric <NUM> may be provided. Switching fabric <NUM> is illustrated in this example as a "flat" network, wherein each server <NUM> may have a direct connection to a top-of-rack (ToR) switch <NUM> (e.g., a "star" configuration), and each ToR switch <NUM> may couple to a core switch <NUM>. This two-tier flat network architecture is shown only as an illustrative example. In other examples, other architectures may be used, such as three-tier star or leaf-spine (also called "fat tree" topologies) based on the "Clos" architecture, hub-and-spoke topologies, mesh topologies, ring topologies, or <NUM>-D mesh topologies, by way of nonlimiting example.

The fabric itself may be provided by any suitable interconnect. For example, each server <NUM> may include an Intel@ Host Fabric Interface (HFI), a NIC, a host channel adapter (HCA), or other host interface. For simplicity and unity, these may be referred to throughout this specification as a "host fabric interface" (HFI), which should be broadly construed as an interface to communicatively couple the host to the data center fabric. The HFI may couple to one or more host processors via an interconnect or bus, such as PCI, PCIe, or similar. In some cases, this interconnect bus, along with other "local" interconnects (e.g., core-to-core Ultra Path Interconnect) may be considered to be part of fabric <NUM>. In other embodiments, the Ultra Path Interconnect (UPI) or other local coherent interconnect may be treated as part of the secure domain of the processor complex, and thus not part of the fabric.

The interconnect technology may be provided by a single interconnect or a hybrid interconnect, such as where PCIe provides on-chip communication, 1Gb or 10Gb copper Ethernet provides relatively short connections to a ToR switch <NUM>, and optical cabling provides relatively longer connections to core switch <NUM>. Interconnect technologies that may be found in the data center include, by way of nonlimiting example, Intel® Omni-Path™ Architecture (OPA), TrueScale™, UPI (formerly called QPI or KTI), FibreChannel, Ethernet, FibreChannel over Ethernet (FCoE), InfiniBand, PCI, PCIe, or fiber optics, to name just a few. The fabric may be cache- and memory-coherent, cache- and memory-non-coherent, or a hybrid of coherent and non-coherent interconnects. Some interconnects are more popular for certain purposes or functions than others, and selecting an appropriate fabric for the instant application is an exercise of ordinary skill. For example, OPA and Infiniband are commonly used in HPC applications, while Ethernet and FibreChannel are more popular in cloud data centers. But these examples are expressly nonlimiting, and as data centers evolve fabric technologies similarly evolve.

Note that while high-end fabrics such as OPA are provided herein by way of illustration, more generally, fabric <NUM> may be any suitable interconnect or bus for the particular application. This could, in some cases, include legacy interconnects like local area networks (LANs), token ring networks, synchronous optical networks (SONET), asynchronous transfer mode (ATM) networks, wireless networks such as WiFi and Bluetooth, "plain old telephone system" (POTS) interconnects, or similar. It is also expressly anticipated that in the future, new network technologies will arise to supplement or replace some of those listed here, and any such future network topologies and technologies can be or form a part of fabric <NUM>.

In certain embodiments, fabric <NUM> may provide communication services on various "layers," as originally outlined in the Open Systems Interconnection (OSI) seven-layer network model. In contemporary practice, the OSI model is not followed strictly. In general terms, layers <NUM> and <NUM> are often called the "Ethernet" layer (though in some data centers or supercomputers, Ethernet may be supplanted or supplemented by newer technologies). Layers <NUM> and <NUM> are often referred to as the transmission control protocol/internet protocol (TCP/IP) layer (which may be further subdivided into TCP and IP layers). Layers <NUM> - <NUM> may be referred to as the "application layer. " These layer definitions are disclosed as a useful framework, but are intended to be nonlimiting.

<FIG> is a block diagram of selected components of an end-user computing device <NUM>, according to one or more examples of the present specification. Embodiments of computing device <NUM> disclosed herein may be adapted or configured to provide the method of providing localized service resiliency according to the teachings of the present specification. As above, computing device <NUM> may provide, as appropriate, cloud service, HPC, telecommunication services, enterprise data center services, or any other compute services that benefit from a computing device <NUM>.

In this example, a fabric <NUM> is provided to interconnect various aspects of computing device <NUM>. Fabric <NUM> may be the same as fabric <NUM> of <FIG>, or may be a different fabric. As above, fabric <NUM> may be provided by any suitable interconnect technology. In this example, Intel@ Omni-Path™ is used as an illustrative and nonlimiting example.

As illustrated, computing device <NUM> includes a number of logic elements forming a plurality of nodes. It should be understood that each node may be provided by a physical server, a group of servers, or other hardware. Each server may be running one or more virtual machines as appropriate to its application.

Node <NUM><NUM> is a processing node including a processor socket <NUM> and processor socket <NUM>. The processors may be, for example, Intel® Xeon™ processors with a plurality of cores, such as <NUM> or <NUM> cores. Node <NUM><NUM> may be configured to provide network or workload functions, such as by hosting a plurality of virtual machines or virtual appliances.

Onboard communication between processor socket <NUM> and processor socket <NUM> may be provided by an onboard uplink <NUM>. This may provide a very high speed, short-length interconnect between the two processor sockets, so that virtual machines running on node <NUM><NUM> can communicate with one another at very high speeds. To facilitate this communication, a virtual switch (vSwitch) may be provisioned on node <NUM><NUM>, which may be considered to be part of fabric <NUM>.

Node <NUM><NUM> connects to fabric <NUM> via an HFI <NUM>. HFI <NUM> may connect to an Intel® Omni-Path™ fabric. In some examples, communication with fabric <NUM> may be tunneled, such as by providing UPI tunneling over Omni-Path™.

Because computing device <NUM> may provide many functions in a distributed fashion that in previous generations were provided onboard, a highly capable HFI <NUM> may be provided. HFI <NUM> may operate at speeds of multiple gigabits per second, and in some cases may be tightly coupled with node <NUM><NUM>. For example, in some embodiments, the logic for HFI <NUM> is integrated directly with the processors on a system-on-a-chip. This provides very high speed communication between HFI <NUM> and the processor sockets, without the need for intermediary bus devices, which may introduce additional latency into the fabric. However, this is not to imply that embodiments where HFI <NUM> is provided over a traditional bus are to be excluded. Rather, it is expressly anticipated that in some examples, HFI <NUM> may be provided on a bus, such as a PCIe bus, which is a serialized version of PCI that provides higher speeds than traditional PCI. Throughout computing device <NUM>, various nodes may provide different types of HFIs <NUM>, such as onboard HFIs and plug-in HFIs. It should also be noted that certain blocks in an SoC may be provided as IP blocks that can be "dropped" into an integrated circuit as a modular unit. Thus, HFI <NUM> may in some cases be derived from such an IP block.

Note that in "the network is the device" fashion, node <NUM><NUM> may provide limited or no onboard memory or storage. Rather, node <NUM><NUM> may rely primarily on distributed services, such as a memory server and a networked storage server. Onboard, node <NUM><NUM> may provide only sufficient memory and storage to bootstrap the device and get it communicating with fabric <NUM>. This kind of distributed architecture is possible because of the very high speeds of contemporary data centers, and may be advantageous because there is no need to over-provision resources for each node. Rather, a large pool of high speed or specialized memory may be dynamically provisioned between a number of nodes, so that each node has access to a large pool of resources, but those resources do not sit idle when that particular node does not need them.

In this example, a node <NUM> memory server <NUM> and a node <NUM> storage server <NUM> provide the operational memory and storage capabilities of node <NUM><NUM>. For example, memory server node <NUM><NUM> may provide remote direct memory access (RDMA), whereby node <NUM><NUM> may access memory resources on node <NUM><NUM> via fabric <NUM> in a direct memory access fashion, similar to how it would access its own onboard memory. The memory provided by memory server <NUM> may be traditional memory, such as double data rate type <NUM> (DDR3), DRAM, which is volatile, or may be a more exotic type of memory, such as a persistent fast memory (PFM) like Intel® 3D Crosspoint™ (3DXP), which operates at DRAM-like speeds, but is nonvolatile.

Similarly, rather than providing an onboard hard disk for node <NUM><NUM>, a storage server node <NUM><NUM> may be provided. Storage server <NUM> may provide a networked bunch of disks (NBOD), PFM, redundant array of independent disks (RAID), redundant array of independent nodes (RAIN), network-attached storage (NAS), optical storage, tape drives, or other nonvolatile memory solutions.

Thus, in performing its designated function, node <NUM><NUM> may access memory from memory server <NUM> and store results on storage provided by storage server <NUM>. Each of these devices couples to fabric <NUM> via a HFI <NUM>, which provides fast communication that makes these technologies possible.

By way of further illustration, node <NUM><NUM> is also depicted. Node <NUM><NUM> also includes a HFI <NUM>, along with two processor sockets internally connected by an uplink. However, unlike node <NUM><NUM>, node <NUM><NUM> includes its own onboard memory <NUM> and storage <NUM>. Thus, node <NUM><NUM> may be configured to perform its functions primarily onboard, and may not be required to rely upon memory server <NUM> and storage server <NUM>. However, in appropriate circumstances, node <NUM><NUM> may supplement its own onboard memory <NUM> and storage <NUM> with distributed resources similar to node <NUM><NUM>.

Computing device <NUM> may also include accelerators <NUM>. These may provide various accelerated functions, including hardware or co-processor acceleration for functions such as packet processing, encryption, decryption, compression, decompression, network security, or other accelerated functions in the data center. In some examples, accelerators <NUM> may include deep learning accelerators that may be directly attached to one or more cores in nodes such as node <NUM><NUM> or node <NUM><NUM>. Examples of such accelerators can include, by way of nonlimiting example, Intel® QuickData Technology (QDT), Intel® QuickAssist Technology (QAT), Intel® Direct Cache Access (DCA), Intel® Extended Message Signaled Interrupt (MSI-X), Intel® Receive Side Coalescing (RSC), and other acceleration technologies.

In other embodiments, an accelerator could also be provided as an ASIC, FPGA, co-processor, graphics processing unit (GPU), digital signal processor (DSP), or other processing entity, which may optionally be tuned or configured to provide the accelerator function.

The basic building block of the various components disclosed herein may be referred to as "logic elements. " Logic elements may include hardware (including, for example, a software-programmable processor, an ASIC, or an FPGA), external hardware (digital, analog, or mixed-signal), software, reciprocating software, services, drivers, interfaces, components, modules, algorithms, sensors, components, firmware, microcode, programmable logic, or objects that can coordinate to achieve a logical operation. Furthermore, some logic elements are provided by a tangible, non-transitory computer-readable medium having stored thereon executable instructions for instructing a processor to perform a certain task. Such a non-transitory medium could include, for example, a hard disk, solid state memory or disk, read-only memory (ROM), PFM (e.g., Intel@ 3D Crosspoint™), external storage, RAID, RAIN, NAS, optical storage, tape drive, backup system, cloud storage, or any combination of the foregoing by way of nonlimiting example. Such a medium could also include instructions programmed into an FPGA, or encoded in hardware on an ASIC or processor.

<FIG> is a block diagram of a software defined network <NUM>, according to one or more examples of the present specification. Embodiments of software defined network <NUM> disclosed herein may be adapted or configured to provide the method of providing localized service resiliency according to the teachings of the present specification. In software defined networking (SDN), a single configuration utility (often a graphical interface or browser interface) may be used to manage network resources at a high level, with very little manual human intervention into the details of the network. SDN may provide a data plane that is separate from a control plane, to separate management functions from data functions. Another benefit of SDNs is that they may be based on open standards, thus providing portability between systems, and alleviating issues of vendor lock-in.

SDN <NUM> is controlled by an SDN controller <NUM>, which may include, for example, a VM controller <NUM>, a storage controller <NUM>, and a network controller <NUM>. Other SDN controller functions may also be provided in other embodiments, and not every embodiments is required to have the foregoing elements. SDN controller <NUM> provides an orchestration layer <NUM>. The orchestration layer may employ an open orchestration protocol, such as the OpenStack cloud operating system.

Orchestration layer <NUM> may include various plug-in components that can be used as interfaces to control data center resources. These plugins may interact with orchestration layer <NUM> via a set of standardized and open APIs, thus enabling different vendors to provide different plugins. In many cases, data center resources can all be managed via a single graphical interface provided by orchestration layer <NUM>. For example, OpenStack currently provides a dashboard called "Horizon," which provides a monolithic interface that enables an administrator to fully configure and administer a data center.

In some embodiments, orchestration layer <NUM> provides a northbound API, on which may be instantiated various applications or functions, such as applications <NUM>-<NUM> - <NUM>-N illustrated here. Each application <NUM> may be provided on a discrete virtual network device or container (referred to herein generically as "network devices"). Applications <NUM> may be configured to perform a variety of network functions, such as, by way of nonlimiting example, load-balancing, firewall, DPI, DNS, antivirus, or any other suitable network function. The particular arrangement of interconnections between network devices <NUM> and from network devices <NUM> to host devices <NUM> may be determined by the particular network configuration and needs. Thus, the specific configuration of <FIG> should be understood to be an illustrative example only.

Orchestration layer may provide communication protocols, including in some embodiments OpenFlow. OpenFlow centralizes networking intelligence into an SDN controller such as SDN controller <NUM>. Using OpenFlow, switches and routers do not need to use prior data exchange protocols to learn about the network topology and other environmental factors. These topology data are replicated across all switches and routers in the network, and each one maintains forwarding tables. In contrast, an OpenFlow controller (which may be, for example, a function of network controller <NUM>) provides centralized network management that maintains the network state in a central repository. Network controller <NUM> can then update forwarding tables as the network topology evolves or changes, generate a forwarding table for each switch or router, and push them out to network devices as necessary. This realizes separation of the control plane and the data plane. Individual network devices may implement the OpenFlow API by adding an OpenFlow shim layer that translates OpenFlow API function calls into native function calls for the switch or router.

Note that while this illustration shows a single SDN controller <NUM>, in large data centers, an orchestration controller may control a number of distributed SDN controllers. In some embodiments, SDN controllers and orchestrators can communicate with one another to remain synchronization and state information via a sideband, such as a separate, lower-speed Ethernet connection.

<FIG> is a block diagram of a network function virtualization (NFV) infrastructure <NUM>, according to one or more examples of the present specification. Embodiments of NFV infrastructure <NUM> disclosed herein may be adapted or configured to provide the method of providing localized service resiliency according to the teachings of the present specification. NFV is an aspect of network virtualization that is generally considered distinct from, but that can still interoperate with SDN. For example, virtual network functions (VNFs) may operate within the data plane of an SDN deployment. NFV was originally envisioned as a method for providing reduced capital expenditure (Capex) and operating expenses (Opex) for telecommunication services. One feature of NFV is replacing proprietary, special-purpose hardware appliances with virtual appliances running on commercial off-the-shelf (COTS) hardware within a virtualized environment. In addition to Capex and Opex savings, NFV provides a more agile and adaptable network. As network loads change, VNFs can be provisioned ("spun up") or removed ("spun down") to meet network demands. For example, in times of high load, more load balancer VNFs may be spun up to distribute traffic to more workload servers (which may themselves be virtual machines). In times when more suspicious traffic is experienced, additional firewalls or DPI appliances may be needed.

Because NFV started out as a telecommunications feature, many NFV instances are focused on telecommunications. However, NFV is not limited to telecommunication services. In a broad sense, NFV includes one or more VNFs running within a network function virtualization infrastructure (NFVI), such as NFVI <NUM>. Often, the VNFs are in-line service functions that are separate from workload servers or other nodes. These VNFs can be chained together into a service chain, which may be defined by a virtual subnetwork, and which may include a serial string of network services that provide behind-the-scenes work, such as security, logging, billing, and similar.

Like SDN, NFV is a subset of network virtualization. Thus, the network as illustrated in <FIG> may be defined instead of or in addition to the network of <FIG>. In other words, certain portions of the network may rely on SDN, while other portions (or the same portions) may rely on NFV.

In the example of <FIG>, an NFV orchestrator <NUM> manages a number of the VNFs <NUM> running on an NFVI <NUM>. NFV requires nontrivial resource management, such as allocating a very large pool of compute resources among appropriate numbers of instances of each VNF, managing connections between VNFs, determining how many instances of each VNF to allocate, and managing memory, storage, and network connections. This may require complex software management, thus making NFV orchestrator <NUM> a valuable system resource. Note that NFV orchestrator <NUM> may provide a browser-based or graphical configuration interface, and in some embodiments may be integrated with SDN orchestration functions.

Note that NFV orchestrator <NUM> itself may be virtualized (rather than a special-purpose hardware appliance). NFV orchestrator <NUM> may be integrated within an existing SDN system, wherein an operations support system (OSS) manages the SDN. This may interact with cloud resource management systems (e.g., OpenStack) to provide NFV orchestration. An NFVI <NUM> may include the hardware, software, and other infrastructure to enable VNFs to run. This may include a hardware platform <NUM> on which one or more VMs <NUM> may run. For example, hardware platform <NUM>-<NUM> in this example runs VMs <NUM>-<NUM> and <NUM>-<NUM>. Hardware platform <NUM>-<NUM> runs VMs <NUM>-<NUM> and <NUM>-<NUM>. Each hardware platform may include a hypervisor <NUM>, VMM, or similar function, which may include and run on a native (bare metal) operating system, which may be minimal so as to consume very few resources.

Hardware platforms <NUM> may be or comprise a rack or several racks of blade or slot servers (including, e.g., processors, memory, and storage), one or more data centers, other hardware resources distributed across one or more geographic locations, hardware switches, or network interfaces. An NFVI <NUM> may also include the software architecture that enables hypervisors to run and be managed by NFV orchestrator <NUM>.

Running on NFVI <NUM> are a number of VMs <NUM>, each of which in this example is a VNF providing a virtual service appliance. Each VM <NUM> in this example includes an instance of the Data Plane Development Kit (DVDK), a virtual operating system <NUM>, and an application providing the VNF <NUM>.

Virtualized network functions could include, as nonlimiting and illustrative examples, firewalls, intrusion detection systems, load balancers, routers, session border controllers, DPI services, network address translation (NAT) modules, or call security association.

The illustration of <FIG> shows that a number of VNFs <NUM> have been provisioned and exist within NFVI <NUM>. This figure does not necessarily illustrate any relationship between the VNFs and the larger network, or the packet flows that NFVI <NUM> may employ.

The illustrated Data Plane Development Kit (DPDK) instances <NUM> provide a set of highly-optimized libraries for communicating across a virtual switch (vSwitch) <NUM>. Like VMs <NUM>, vSwitch <NUM> is provisioned and allocated by a hypervisor <NUM>. The hypervisor uses a network interface to connect the hardware platform to the data center fabric (e.g., an HFI). This HFI may be shared by all VMs <NUM> running on a hardware platform <NUM>. Thus, a vSwitch may be allocated to switch traffic between VMs <NUM>. The vSwitch may be a pure software vSwitch (e.g., a shared memory vSwitch), which may be optimized so that data are not moved between memory locations, but rather, the data may stay in one place, and pointers may be passed between VMs <NUM> to simulate data moving between ingress and egress ports of the vSwitch. The vSwitch may also include a hardware driver (e.g., a hardware network interface IP block that switches traffic, but that connects to virtual ports rather than physical ports). In this illustration, a distributed vSwitch <NUM> is illustrated, wherein vSwitch <NUM> is shared between two or more physical hardware platforms <NUM>.

<FIG> is a block diagram of components of a computing platform 1502A, according to one or more examples of the present specification. Embodiments of computing platform 1502A disclosed herein may be adapted or configured to provide the method of providing localized service resiliency according to the teachings of the present specification. In the embodiment depicted, platforms 1502A, 1502B, and 1502C, along with a data center management platform <NUM> and data analytics engine <NUM> are interconnected via network <NUM>. In other embodiments, a computer system may include any suitable number of (i.e., one or more) platforms. In some embodiments (e.g., when a computer system only includes a single platform), all or a portion of the system management platform <NUM> may be included on a platform <NUM>. A platform <NUM> may include platform logic <NUM> with one or more CPUs <NUM>, memories <NUM> (which may include any number of different modules), chipsets <NUM>, communication interfaces <NUM>, and any other suitable hardware and/or software to execute a hypervisor <NUM> or other operating system capable of executing workloads associated with applications running on platform <NUM>. In some embodiments, a platform <NUM> may function as a host platform for one or more guest systems <NUM> that invoke these applications. Platform 1502A may represent any suitable computing environment, such as an HPC environment, a data center, a communications service provider infrastructure (e.g., one or more portions of an Evolved Packet Core), an in-memory computing environment, a computing system of a vehicle (e.g., an automobile or airplane), an Internet of Things environment, an industrial control system, other computing environment, or combination thereof.

In various embodiments of the present disclosure, accumulated stress and/or rates of stress accumulated of a plurality of hardware resources (e.g., cores and uncores) are monitored and entities (e.g., system management platform <NUM>, hypervisor <NUM>, or other operating system) of computer platform 1502A may assign hardware resources of platform logic <NUM> to perform workloads in accordance with the stress information. In some embodiments, self-diagnostic capabilities may be combined with the stress monitoring to more accurately determine the health of the hardware resources. Each platform <NUM> may include platform logic <NUM>. Platform logic <NUM> comprises, among other logic enabling the functionality of platform <NUM>, one or more CPUs <NUM>, memory <NUM>, one or more chipsets <NUM>, and communication interfaces <NUM>. Although three platforms are illustrated, computer platform 1502A may be interconnected with any suitable number of platforms. In various embodiments, a platform <NUM> may reside on a circuit board that is installed in a chassis, rack, or other suitable structure that comprises multiple platforms coupled together through network <NUM> (which may comprise, e.g., a rack or backplane switch).

CPUs <NUM> may each comprise any suitable number of processor cores and supporting logic (e.g., uncores). The cores may be coupled to each other, to memory <NUM>, to at least one chipset <NUM>, and/or to a communication interface <NUM>, through one or more controllers residing on CPU <NUM> and/or chipset <NUM>. In particular embodiments, a CPU <NUM> is embodied within a socket that is permanently or removably coupled to platform 1502A. Although four CPUs are shown, a platform <NUM> may include any suitable number of CPUs.

Memory <NUM> may comprise any form of volatile or nonvolatile memory including, without limitation, magnetic media (e.g., one or more tape drives), optical media, random access memory (RAM), ROM, flash memory, removable media, or any other suitable local or remote memory component or components. Memory <NUM> may be used for short, medium, and/or long term storage by platform 1502A. Memory <NUM> may store any suitable data or information utilized by platform logic <NUM>, including software embedded in a computer-readable medium, and/or encoded logic incorporated in hardware or otherwise stored (e.g., firmware). Memory <NUM> may store data that is used by cores of CPUs <NUM>. In some embodiments, memory <NUM> may also comprise storage for instructions that may be executed by the cores of CPUs <NUM> or other processing elements (e.g., logic resident on chipsets <NUM>) to provide functionality associated with the manageability engine <NUM> or other components of platform logic <NUM>. A platform <NUM> may also include one or more chipsets <NUM> comprising any suitable logic to support the operation of the CPUs <NUM>. In various embodiments, chipset <NUM> may reside on the same die or package as a CPU <NUM> or on one or more different dies or packages. Each chipset may support any suitable number of CPUs <NUM>. A chipset <NUM> may also include one or more controllers to couple other components of platform logic <NUM> (e.g., communication interface <NUM> or memory <NUM>) to one or more CPUs. In the embodiment depicted, each chipset <NUM> also includes a manageability engine <NUM>. Manageability engine <NUM> may include any suitable logic to support the operation of chipset <NUM>. In a particular embodiment, a manageability engine <NUM> (which may also be referred to as an Intel@ Innovation Engine) is capable of collecting real-time telemetry data from the chipset <NUM>, the CPU(s) <NUM> and/or memory <NUM> managed by the chipset <NUM>, other components of platform logic <NUM>, and/or various connections between components of platform logic <NUM>. In various embodiments, the telemetry data collected includes the stress information described herein.

In various embodiments, a manageability engine <NUM> operates as an out-of-band asynchronous compute agent which is capable of interfacing with the various elements of platform logic <NUM> to collect telemetry data with no or minimal disruption to running processes on CPUs <NUM>. For example, manageability engine <NUM> may comprise a dedicated processing element (e.g., a processor, controller, or other logic) on chipset <NUM>, which provides the functionality of manageability engine <NUM> (e.g., by executing software instructions), thus conserving processing cycles of CPUs <NUM> for operations associated with the workloads performed by the platform logic <NUM>. Moreover the dedicated logic for the manageability engine <NUM> may operate asynchronously with respect to the CPUs <NUM> and may gather at least some of the telemetry data without increasing the load on the CPUs.

A manageability engine <NUM> may process telemetry data it collects (specific examples of the processing of stress information will be provided herein). In various embodiments, manageability engine <NUM> reports the data it collects and/or the results of its processing to other elements in the computer system, such as one or more hypervisors <NUM> or other operating systems and/or system management software (which may run on any suitable logic such as system management platform <NUM>). In particular embodiments, a critical event such as a core that has accumulated an excessive amount of stress may be reported prior to the normal interval for reporting telemetry data (e.g., a notification may be sent immediately upon detection).

Additionally, manageability engine <NUM> may include programmable code configurable to set which CPU(s) <NUM> a particular chipset <NUM> will manage and/or which telemetry data will be collected.

Chipsets <NUM> also each include a communication interface <NUM>. Communication interface <NUM> may be used for the communication of signaling and/or data between chipset <NUM> and one or more I/O devices, one or more networks <NUM>, and/or one or more devices coupled to network <NUM> (e.g., system management platform <NUM>). For example, communication interface <NUM> may be used to send and receive network traffic such as data packets. In a particular embodiment, a communication interface <NUM> comprises one or more physical network interface controllers (NICs), also known as network interface cards or network adapters. A NIC may include electronic circuitry to communicate using any suitable physical layer and data link layer standard such as Ethernet (e.g., as defined by a IEEE <NUM> standard), Fibre Channel, InfiniBand, Wi-Fi, or other suitable standard. A NIC may include one or more physical ports that may couple to a cable (e.g., an Ethernet cable). A NIC may enable communication between any suitable element of chipset <NUM> (e.g., manageability engine <NUM> or switch <NUM>) and another device coupled to network <NUM>. In various embodiments a NIC may be integrated with the chipset (i.e., may be on the same integrated circuit or circuit board as the rest of the chipset logic) or may be on a different integrated circuit or circuit board that is electromechanically coupled to the chipset.

In particular embodiments, communication interfaces <NUM> may allow communication of data (e.g., between the manageability engine <NUM> and the data center management platform <NUM>) associated with management and monitoring functions performed by manageability engine <NUM>. In various embodiments, manageability engine <NUM> may utilize elements (e.g., one or more NICs) of communication interfaces <NUM> to report the telemetry data (e.g., to system management platform <NUM>) in order to reserve usage of NICs of communication interface <NUM> for operations associated with workloads performed by platform logic <NUM>.

Switches <NUM> may couple to various ports (e.g., provided by NICs) of communication interface <NUM> and may switch data between these ports and various components of chipset <NUM> (e.g., one or more Peripheral Component Interconnect Express (PCIe) lanes coupled to CPUs <NUM>). Switches <NUM> may be a physical or virtual (i.e., software) switch.

Platform logic <NUM> may include an additional communication interface <NUM>. Similar to communication interfaces <NUM>, communication interfaces <NUM> may be used for the communication of signaling and/or data between platform logic <NUM> and one or more networks <NUM> and one or more devices coupled to the network <NUM>. For example, communication interface <NUM> may be used to send and receive network traffic such as data packets. In a particular embodiment, communication interfaces <NUM> comprise one or more physical NICs. These NICs may enable communication between any suitable element of platform logic <NUM> (e.g., CPUs <NUM> or memory <NUM>) and another device coupled to network <NUM> (e.g., elements of other platforms or remote computing devices coupled to network <NUM> through one or more networks).

Platform logic <NUM> may receive and perform any suitable types of workloads. A workload may include any request to utilize one or more resources of platform logic <NUM>, such as one or more cores or associated logic. For example, a workload may comprise a request to instantiate a software component, such as an I/O device driver <NUM> or guest system <NUM>; a request to process a network packet received from a virtual machine <NUM> or device external to platform 1502A (such as a network node coupled to network <NUM>); a request to execute a process or thread associated with a guest system <NUM>, an application running on platform 1502A, a hypervisor <NUM> or other operating system running on platform 1502A; or other suitable processing request.

A virtual machine <NUM> may emulate a computer system with its own dedicated hardware. A virtual machine <NUM> may run a guest operating system on top of the hypervisor <NUM>. The components of platform logic <NUM> (e.g., CPUs <NUM>, memory <NUM>, chipset <NUM>, and communication interface <NUM>) may be virtualized such that it appears to the guest operating system that the virtual machine <NUM> has its own dedicated components.

A virtual machine <NUM> may include a virtualized NIC (vNIC), which is used by the virtual machine as its network interface. A vNIC may be assigned a media access control (MAC) address or other identifier, thus allowing multiple virtual machines <NUM> to be individually addressable in a network.

VNF <NUM> may comprise a software implementation of a functional building block with defined interfaces and behavior that can be deployed in a virtualized infrastructure. In particular embodiments, a VNF <NUM> may include one or more virtual machines <NUM> that collectively provide specific functionalities (e.g., WAN optimization, virtual private network (VPN) termination, firewall operations, load-balancing operations, security functions, etc.). A VNF <NUM> running on platform logic <NUM> may provide the same functionality as traditional network components implemented through dedicated hardware. For example, a VNF <NUM> may include components to perform any suitable NFV workloads, such as virtualized evolved packet core (vEPC) components, mobility management entities, 3rd Generation Partnership Project (3GPP) control and data plane components, etc..

SFC <NUM> is a group of VNFs <NUM> organized as a chain to perform a series of operations, such as network packet processing operations. Service function chaining may provide the ability to define an ordered list of network services (e.g. firewalls, load balancers) that are stitched together in the network to create a service chain.

A hypervisor <NUM> (also known as a virtual machine monitor) may comprise logic to create and run guest systems <NUM>. The hypervisor <NUM> may present guest operating systems run by virtual machines with a virtual operating platform (i.e., it appears to the virtual machines that they are running on separate physical nodes when they are actually consolidated onto a single hardware platform) and manage the execution of the guest operating systems by platform logic <NUM>. Services of hypervisor <NUM> may be provided by virtualizing in software or through hardware assisted resources that require minimal software intervention, or both. Multiple instances of a variety of guest operating systems may be managed by the hypervisor <NUM>. Each platform <NUM> may have a separate instantiation of a hypervisor <NUM>.

Hypervisor <NUM> may be a native or bare metal hypervisor that runs directly on platform logic <NUM> to control the platform logic and manage the guest operating systems. Alternatively, hypervisor <NUM> may be a hosted hypervisor that runs on a host operating system and abstracts the guest operating systems from the host operating system. Hypervisor <NUM> may include a virtual switch <NUM> that may provide virtual switching and/or routing functions to virtual machines of guest systems <NUM>. The virtual switch <NUM> may comprise a logical switching fabric that couples the vNICs of the virtual machines <NUM> to each other, thus creating a virtual network through which virtual machines may communicate with each other.

Virtual switch <NUM> may comprise a software element that is executed using components of platform logic <NUM>. In various embodiments, hypervisor <NUM> may be in communication with any suitable entity (e.g., a SDN controller) which may cause hypervisor <NUM> to reconfigure the parameters of virtual switch <NUM> in response to changing conditions in platform <NUM> (e.g., the addition or deletion of virtual machines <NUM> or identification of optimizations that may be made to enhance performance of the platform).

Hypervisor <NUM> may also include resource allocation logic <NUM>, which may include logic for determining allocation of platform resources based on the telemetry data (which may include stress information). Resource allocation logic <NUM> may also include logic for communicating with various components of platform logic <NUM> entities of platform 1502A to implement such optimization, such as components of platform logic <NUM>.

Any suitable logic may make one or more of these optimization decisions. For example, system management platform <NUM>; resource allocation logic <NUM> of hypervisor <NUM> or other operating system; or other logic of computer platform 1502A may be capable of making such decisions. In various embodiments, the system management platform <NUM> may receive telemetry data from and manage workload placement across multiple platforms <NUM>. The system management platform <NUM> may communicate with hypervisors <NUM> (e.g., in an out-of-band manner) or other operating systems of the various platforms <NUM> to implement workload placements directed by the system management platform.

The elements of platform logic <NUM> may be coupled together in any suitable manner. For example, a bus may couple any of the components together. A bus may include any known interconnect, such as a multi-drop bus, a mesh interconnect, a ring interconnect, a point-to-point interconnect, a serial interconnect, a parallel bus, a coherent (e.g. cache coherent) bus, a layered protocol architecture, a differential bus, or a Gunning transceiver logic (GTL) bus.

As used in the present specification, cache coherency is a memory architecture that provides uniform sharing and mapping between a plurality of caches. For example, the caches may map to the same address space. If two different caches have cached the same address in the shared address space, a coherency agent provides logic (hardware and/or software) to ensure the compatibility and uniformity of shared resource. For example, if two caches have cached the same address, when the value stored in that address is updated in one cache, the coherency agent ensures that the change is propagated to the other cache. Coherency may be maintained, for example, via "snooping," wherein each cache monitors the address lines of each other cache, and detects updates. Cache coherency may also be maintained via a directory-based system, in which shared data are placed in a shared directory that maintains coherency. Some distributed shared memory architectures may also provide coherency, for example by emulating the foregoing mechanisms.

Coherency may be either "snoopy" or directory-based. In snoopy protocols, coherency may be maintained via write-invalidate, wherein a first cache that snoops a write to the same address in a second cache invalidates its own copy. This forces a read from memory if a program tries to read the value from the first cache. Alternatively, in write-update, a first cache snoops a write to a second cache, and a cache controller (which may include a coherency agent) copies the data out and updates the copy in the first cache.

By way of nonlimiting example, current cache coherency models include MSI (modified, shared, invalid), MESI (modified, exclusive, shared, invalid), MOSI (modified, owned, shared, invalid), MOESI (modified, owned, exclusive, shared, invalid), MERSI (modified, exclusive, read-only or recent, shared, invalid), MESIF (modified, exclusive, shared, invalid, forward), write-once, Synapse, Berkeley, Firefly, and Dragon protocols. Furthermore, ARM processors may use advanced microcontroller bus architecture (AMBA), including AMBA <NUM> ACE, to provide cache coherency in systems-on-a-chip (SoCs) or elsewhere.

Elements of the computer platform 1502A may be coupled together in any suitable manner such as through one or more networks <NUM>. A network <NUM> may be any suitable network or combination of one or more networks operating using one or more suitable networking protocols. A network may represent a series of nodes, points, and interconnected communication paths for receiving and transmitting packets of information that propagate through a communication system. For example, a network may include one or more firewalls, routers, switches, security appliances, antivirus servers, or other useful network devices.

The foregoing outlines features of one or more embodiments of the subject matter disclosed herein. These embodiments are provided to enable a person having ordinary skill in the art (PHOSITA) to better understand various aspects of the present disclosure. Certain well-understood terms, as well as underlying technologies and/or standards may be referenced without being described in detail. It is anticipated that the PHOSITA will possess or have access to background knowledge or information in those technologies and standards sufficient to practice the teachings of the present specification.

In the foregoing description, certain aspects of some or all embodiments are described in greater detail than is strictly necessary for practicing the appended claims. These details are provided by way of nonlimiting example only, for the purpose of providing context and illustration of the disclosed embodiments. Such details should not be understood to be required, and should not be "read into" the claims as limitations. The phrase may refer to "an embodiment" or "embodiments. " These phrases, and any other references to embodiments, should be understood broadly to refer to any combination of one or more embodiments. Furthermore, the several features disclosed in a particular "embodiment" could just as well be spread across multiple embodiments. For example, if features <NUM> and <NUM> are disclosed in "an embodiment," embodiment A may have feature <NUM> but lack feature <NUM>, while embodiment B may have feature <NUM> but lack feature <NUM>.

This specification may provide illustrations in a block diagram format, wherein certain features are disclosed in separate blocks. These should be understood broadly to disclose how various features interoperate, but are not intended to imply that those features must necessarily be embodied in separate hardware or software. Furthermore, where a single block discloses more than one feature in the same block, those features need not necessarily be embodied in the same hardware and/or software. For example, a computer "memory" could in some circumstances be distributed or mapped between multiple levels of cache or local memory, main memory, battery-backed volatile memory, and various forms of persistent memory such as a hard disk, storage server, optical disk, tape drive, or similar. In certain embodiments, some of the components may be omitted or consolidated. In a general sense, the arrangements depicted in the figures may be more logical in their representations, whereas a physical architecture may include various permutations, combinations, and/or hybrids of these elements. Countless possible design configurations can be used to achieve the operational objectives outlined herein. Accordingly, the associated infrastructure has a myriad of substitute arrangements, design choices, device possibilities, hardware configurations, software implementations, and equipment options.

References may be made herein to a computer-readable medium, which may be a tangible and non-transitory computer-readable medium. As used in this specification and throughout the claims, a "computer-readable medium" should be understood to include one or more computer-readable mediums of the same or different types. A computer-readable medium may include, by way of nonlimiting example, an optical drive (e.g., CD/DVD/Blu-Ray), a hard drive, a solid state drive, a flash memory, or other nonvolatile medium. A computer-readable medium could also include a medium such as a ROM, an FPGA or ASIC configured to carry out the desired instructions, stored instructions for programming an FPGA or ASIC to carry out the desired instructions, an IP block that can be integrated in hardware into other circuits, or instructions encoded directly into hardware or microcode on a processor such as a microprocessor, DSP, microcontroller, or in any other suitable component, device, element, or object where appropriate and based on particular needs. A non-transitory storage medium herein is expressly intended to include any non-transitory special-purpose or programmable hardware configured to provide the disclosed operations, or to cause a processor to perform the disclosed operations.

Various elements may be "communicatively," "electrically," "mechanically," or otherwise "coupled" to one another throughout this specification and the claims. Such coupling may be a direct, point-to-point coupling, or may include intermediary devices. For example, two devices may be communicatively coupled to one another via a controller that facilitates the communication. Devices may be electrically coupled to one another via intermediary devices such as signal boosters, voltage dividers, or buffers. Mechanically-coupled devices may be indirectly mechanically-coupled.

Any "module" or "engine" disclosed herein may refer to or include software, a software stack, a combination of hardware, firmware, and/or software, a circuit configured to carry out the function of the engine or module, or any computer-readable medium as disclosed above. Such modules or engines may, in appropriate circumstances, be provided on or in conjunction with a hardware platform, which may include hardware compute resources such as a processor, memory, storage, interconnects, networks and network interfaces, accelerators, or other suitable hardware. Such a hardware platform may be provided as a single monolithic device (e.g., in a PC form factor), or with some or part of the function being distributed (e.g., a "composite node" in a high-end data center, where compute, memory, storage, and other resources may be dynamically allocated and need not be local to one another).

There may be disclosed herein flow charts, signal flow diagram, or other illustrations showing operations being performed in a particular order. Unless otherwise expressly noted, or unless required in a particular context, the order should be understood to be a nonlimiting example only. Furthermore, in cases where one operation is shown to follow another, other intervening operations may also occur, which may be related or unrelated. Some operations may also be performed simultaneously or in parallel. In cases where an operation is said to be "based on" or "according to" another item or operation, this should be understood to imply that the operation is based at least partly on or according at least partly to the other item or operation. This should not be construed to imply that the operation is based solely or exclusively on, or solely or exclusively according to the item or operation.

All or part of any hardware element disclosed herein may readily be provided in an SoC, including a CPU package. An SoC represents an integrated circuit (IC) that integrates components of a computer or other electronic system into a single chip. Thus, for example, client devices or server devices may be provided, in whole or in part, in an SoC. The SoC may contain digital, analog, mixed-signal, and radio frequency functions, all of which may be provided on a single chip substrate. Other embodiments may include a multichip module (MCM), with a plurality of chips located within a single electronic package and configured to interact closely with each other through the electronic package.

In a general sense, any suitably-configured circuit or processor can execute any type of instructions associated with the data to achieve the operations detailed herein. Any processor disclosed herein could transform an element or an article (for example, data) from one state or thing to another state or thing. Furthermore, the information being tracked, sent, received, or stored in a processor could be provided in any database, register, table, cache, queue, control list, or storage structure, based on particular needs and implementations, all of which could be referenced in any suitable timeframe. Any of the memory or storage elements disclosed herein, should be construed as being encompassed within the broad terms "memory" and "storage," as appropriate.

Computer program logic implementing all or part of the functionality described herein is embodied in various forms, including, but in no way limited to, a source code form, a computer executable form, machine instructions or microcode, programmable hardware, and various intermediate forms (for example, forms generated by an assembler, compiler, linker, or locator). In an example, source code includes a series of computer program instructions implemented in various programming languages, such as an object code, an assembly language, or a high level language such as OpenCL, FORTRAN, C, C++, JAVA, or HTML for use with various operating systems or operating environments, or in hardware description languages such as Spice, Verilog, and VHDL. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form, or converted to an intermediate form such as byte code. Where appropriate, any of the foregoing may be used to build or describe appropriate discrete or integrated circuits, whether sequential, combinatorial, state machines, or otherwise.

Claim 1:
A method performed by a resiliency controller (<NUM>) for providing local resiliency control on a local platform (<NUM>) of a computing apparatus that includes a central processing unit, CPU, the method comprising:
communicatively coupling to a management controller (<NUM>) via a management controller interface (<NUM>), the management controller being located external to the local platform;
communicatively coupling to a local virtualized function (<NUM>; <NUM>) via a virtualization infrastructure interface, the local virtualized function being located within the local platform;
generating and maintaining a topology hardware dependency graph (<NUM>) to allow the resiliency controller to track interdependencies of hardware resources (<NUM>) of a local hardware platform (<NUM>) of the local platform to thereby enable tracing of at least physical faults at the hardware resources;
associating the hardware resources (<NUM>) with the local virtualized function;
receiving a preloaded resiliency policy from the management controller via the management controller interface (<NUM>), the resiliency policy comprising information to handle a fault in the local virtualized function;
monitoring for a plurality of fault domains on the local hardware platform (<NUM>), wherein a fault domain is associated with one of: a physical fault at one of the hardware resources of the local hardware platform, or a virtual fault at the local virtualized function;
detecting a fault in at least one of the plurality of fault domains being monitored; and
in response to detecting the fault, taking a resiliency action, based on the received resiliency policy,
wherein the resiliency controller is located within the local platform and is configured to be implemented external to the CPU to thereby be isolated from unrecoverable machine check architecture, MCA, events that may halt the CPU.