Patent Description:
Various technical specifications define the way in which network functions and services are deployed and managed by network operators and service providers worldwide. For example, specifications define the use of virtualized platforms to deliver services and, oftentimes, components within a service may be "chained" together. Such technical specifications include, for example, the European Telecommunication Standards Institute's standard for Network Functions Virtualization (ETSI NFV). When a network operator runs the network functions and services on a virtual network function model as currently defined by ETSI NFV, well-defined interfaces traditionally available to physical networking systems are no longer available for inter-flow packet analysis. As such, the system's ability to ensure threats are detected and responded to (e.g., preventing a subscriber from accessing a service on a network function reserved for subscribers with a higher privilege level) may be significantly inhibited.

While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

Additionally, it should be appreciated that items included in a list in the form of "at least one A, B, and C" can mean (A); (B); (C): (A and B); (B and C); (A and C); or (A, B, and C). Similarly, items listed in the form of "at least one of A, B, or C" can mean (A); (B); (C): (A and B); (B and C); (A and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

Referring now to <FIG>, a system <NUM> for distributed detection of security anomalies illustratively includes a backbone network system <NUM>, a backhaul network system <NUM>, one or more tower systems <NUM>, and one or more subscriber devices <NUM>. In the illustrative embodiment, the subscriber devices <NUM> communicate with the backhaul network system <NUM> by virtue of the tower systems <NUM>, and the backhaul network system <NUM> ensures the appropriate data packets are routed to the backbone network system <NUM> for processing and/or further routing. It should be appreciated that each of the backbone network system <NUM>, the backhaul network system <NUM>, the tower systems <NUM>, and the subscriber devices <NUM> may be embodied as any suitable device or collection of devices for performing the functions described herein. In the illustrative embodiment, each of the backbone network system <NUM>, the backhaul network system <NUM>, and the tower systems <NUM> enable telecommunication between the subscriber devices <NUM> and/or other devices (e.g., over the Internet). Further the backbone network system <NUM>, the backhaul network system <NUM>, and the tower systems <NUM> may include any number of devices, networks, routers, switches, computers, and/or other intervening devices to facilitate their corresponding functions depending on the particular implementation.

In some embodiments, the backbone network system <NUM> may be embodied as a Network Function Virtualization (NFV) -based Long-Term Evolution (LTE) backbone network having a Virtual Evolved Packet Core (vEPC) architecture. It should be appreciated that the backbone network system <NUM> may serve as a centralized network and, in some embodiments, may be communicatively coupled to another network (e.g., the Internet). In the illustrative embodiment, the backhaul network system <NUM> includes one or more devices that communicatively couple (e.g., via intermediate links) the backbone network system <NUM> to the tower systems <NUM>, subnetworks, and/or edge networks. In some embodiments, the backhaul network system <NUM> may be embodied as an LTE backhaul network system and may include a variety of networks including, for example, T1, IP, optical, ATM, leased, and/or other networks.

The tower systems <NUM> include hardware configured to permit communication devices, for example, mobile computing devices (e.g., mobile phones) and/or other subscriber devices <NUM>, to communicate with one another and/or other remote devices. In doing so, the tower systems <NUM> enable the subscriber devices <NUM> to communicate with the backhaul network system <NUM>. In some embodiments, one or more of tower systems <NUM> may include or otherwise be embodied as an evolved node (eNodeB) configured to communicate directly or indirectly with one or more of the subscriber devices <NUM> (e.g., mobile computing device handsets). Further, the tower systems <NUM> may include or serve as, for example, a base transceiver station (BTS) or another station/system depending on the particular embodiment. The subscriber devices <NUM> may be embodied as any type of computing device capable of performing the functions described herein. For example, in embodiments in which an LTE backhaul and backbone system are utilized, the subscriber devices <NUM> may be embodied as mobile computing devices (e.g., smartphones) and may be configured to utilize a cellular network.

As described in detail below, the system <NUM> may utilize various virtual network functions while ensuring that threats are detected and acted upon (e.g., via inter-flow packet analysis). Additionally, the system <NUM> may provide enhanced and fine-grain security inspection capabilities on virtual platforms using a Trusted Execution Environment (TEE). As described below, in the illustrative embodiment, the TEE is established as a secure enclave such as Intel® Software Guard Extensions (SGX). However, in other embodiments, the TEE may be otherwise established or embodied as, for example, a Manageability Engine (ME), trusted platform module (TPM), Innovation Engine (IE), secure partition, separate processor core, and/or otherwise established.

It should be appreciated that in a Network Function Virtualization (NVF) environment, the traditional well-defined interfaces of non-virtualized environments are generally unavailable and the NFV system may include multiple Virtual Network Functions (VNFs), each of which may include one or more Virtual Network Function Components (VNFCs). The VNFs and/or VNFCs may communicate with one another using various different mechanisms including, for example, shared memory, OS- or Hypervisor-specific Application Programming Interfaces (APIs) that are closed, network virtual switch test access points (TAPs), and/or other mechanisms. Further, in some embodiments, the intra-VNF and/or intra-VNF traffic may be encrypted using, for example, Internet Protocol security (IPsec) or Secure Sockets Layer (SSL). As such, it should be appreciated that, the traditional mechanisms may not offer a consistent way for a traditional Network Inspection System to operate efficiently with clear visibility to all traffic in a virtualized environment.

However, in the illustrative embodiment, the system <NUM> is configured to inspect packets and/or flows across virtualized systems using the capability of the TEE (e.g., in conjunction with microcode (ucode), hardware instructions, and/or other mechanisms). For example, as described below, each server or platform of the system <NUM> may include a platform-specific TEE that assumes the role of platform security policy inspector. In particular, the platform-specific TEE may inspect all packets coming (i.e., ingress and/or egress) from the Network MAC/Ethernet and/or other network/communication interfaces (e.g., through inter-IP side-channel mechanisms). Additionally, the platform-specific TEE may inspect shared memory and/or proprietary APIs based on hypervisor (e.g., virtual machine monitor) privileges and communication with the TEE using defined APIs (e.g., a HECI interface). The platform-specific TEE may, additionally or alternatively, inspect local and shared processor (e.g., CPU) and SoC cache memory based on higher privileges invoked on signed and anti-rollback protected microcode (ucode) patches. In some embodiments, the platform-specific TEE uses the TEE-based inter-VNFC tunnel keys for monitoring protected inter-VNFC and inter-VNF traffic. Additionally or alternatively, the platform-specific TEE may use hypervisor access into the various virtual switch interfaces and into TAPs to access traffic data.

It should be appreciated that, in some embodiments, the TEE may collect information from multiple sources on the platform and may do so in a far more detailed manner than done in traditional systems. For example, the TEE may be configurable by a policy to monitor all or selected packets, network flows, track packet modifications, and/or perform other monitoring functions. The TEE may run advanced heuristics on the data collected and, depending on the particular policy, retain threat information. Further, the TEE may take one or more remedial actions based on the policy and/or received remediation instructions (e.g., blocking certain flows, copying packets, etc.). In some embodiments, the TEE may convey exceptions and/or threat heuristics to a nominated TEE (e.g., on an NFV distributed threat detection security system), which may execute system-wide security threat heuristics/analysis. It should be appreciated that, in some embodiments, the TEE is "nominated" in the sense that the distributed threat detection system is designed such that other TEEs transmit security information to the nominated TEE for further (e.g., wider-scale) analysis. As described below, in some embodiments, the nominated TEE may be included in a security server and/or a distributed threat detection security system. Further, in some embodiments, multiple TEEs may be nominated to perform a system-wide or subsystem-wide security threat analysis, and the TEEs may be arranged hierarchically. For example, in an embodiment, a first nominated TEE may perform a security threat analysis of a first subsystem based on information received from corresponding TEEs of servers in the first subsystem, and a second nominated TEE may perform a security threat analysis of a second subsystem based on information received from corresponding TEEs of servers in the second subsystem, and so on. Each of those subsystem TEEs (e.g., the first and second nominated TEEs) may provide their analyses and/or additional information to another nominated TEE at a "higher" hierarchical level to perform a full system-wide (or larger subsystem-wide) security threat analysis based on the information received from the lower level nominated TEEs. Of course, the number of nominated TEEs and/or hierarchical levels may vary depending on the particular embodiment.

It should be appreciated that the hierarchical ability of the TEE may allow a local remediation action to be enacted and simultaneously enable system-wide threat detection and remediation for flows that span VNFs and VNFCs across multiple platforms. In some embodiments, the TEE may be protected, including all code and data, and loaded only upon signature verification and measurements (e.g., using a TPM or virtual TPM). Further, the TEE may have the ability to run signature-verified third party verification (TPV) code authorized by the root keys for enabling TPVs and/or other vendors. It should be appreciated that the interfaces for communication described herein may include, for example, inter-IP communication (IPC) within a SoC or processor), device-driver model (e.g., HECI interface), virtual LAN attachment, existing protocols for inter-component interaction (e.g., PECI, SMBUS, etc.). In other embodiments, the components may communication over, for example, TLS-protected HTTPS web-based REST APIs. It should further be appreciated that, in some embodiments, the system <NUM> may be implemented in a platform-, hypervisor-, and cloud OS-neutral manner.

Referring now to <FIG>, in the illustrative embodiment, the backbone network system <NUM> includes one or more VNFs <NUM>, one or more servers <NUM>, and a security server <NUM>. Additionally, in some embodiments, the backbone network system <NUM> includes a remediation server <NUM> and/or an orchestrator <NUM>. Although only one security server <NUM>, one remediation server <NUM>, and one orchestrator <NUM> are illustratively shown in <FIG>, the backbone network system <NUM> may include any number of security servers <NUM>, remediation servers <NUM>, and/or orchestrators <NUM> in other embodiments. For example, several security servers <NUM> may be included, each of which may include a nominated TEE as described herein for hierarchical and distributed threat detection. It should be appreciated that, in some embodiments, each of the servers <NUM> and the security server <NUM> may include similar hardware, software, and/or firmware components. Further, in some embodiments, the security server <NUM> may be embodied as one of the servers <NUM> except that the security server <NUM> includes a nominated TEE as described herein.

Referring now to <FIG>, an illustrative embodiment of the servers <NUM>, <NUM> of the system <NUM> is shown. As shown, the illustrative server <NUM>, <NUM> includes a processor <NUM>, an input/output ("I/O" subsystem) <NUM>, a memory <NUM>, a data storage <NUM>, a communication circuitry <NUM>, and one or more peripheral devices <NUM>. Additionally, in some embodiments, the server <NUM>, <NUM> may include a security co-processor <NUM>. Of course, the server <NUM>, <NUM> may include other or additional components, such as those commonly found in a typical computing device (e.g., various input/output devices and/or other components), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory <NUM>, or portions thereof, may be incorporated in the processor <NUM> in some embodiments.

The processor <NUM> may be embodied as any type of processor capable of performing the functions described herein. For example, the processor <NUM> may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. As shown, the processor <NUM> may include one or more cache memories <NUM>. It should be appreciated that the memory <NUM> may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory <NUM> may store various data and software used during operation of the server <NUM>, <NUM> such as operating systems, applications, programs, libraries, and drivers. The memory <NUM> is communicatively coupled to the processor <NUM> via the I/O subsystem <NUM>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor <NUM>, the memory <NUM>, and other components of the server <NUM>, <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <NUM> may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor <NUM>, the memory <NUM>, and other components of the server <NUM>, <NUM>, on a single integrated circuit chip.

The data storage <NUM> may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage <NUM> and/or the memory <NUM> may store various data during operation of the server <NUM>, <NUM> useful for performing the functions described herein.

The communication circuitry <NUM> may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the server <NUM>, <NUM> and other remote devices over a network. The communication circuitry <NUM> may be configured to use any one or more communication technologies (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication. In some embodiments, the communication circuitry <NUM> includes cellular communication circuitry and/or other long-ranged wireless communication circuitry.

The peripheral devices <NUM> may include any number of additional peripheral or interface devices, such as speakers, microphones, additional storage devices, and so forth. The particular devices included in the peripheral devices <NUM> may depend on, for example, the type and/or intended use of the server <NUM>, <NUM>.

The security co-processor <NUM>, if included, may be embodied as any hardware component(s) or circuitry capable of performing security functions, cryptographic functions, and/or establishing a trusted execution environment. For example, in some embodiments, the security co-processor <NUM> may be embodied as a trusted platform module (TPM) or an out-of-band processor. Additionally, in some embodiments, the security co-processor <NUM> may establish an out-of-band communication link with remote devices (e.g., corresponding security co-processors <NUM> of other servers <NUM>, <NUM>).

Referring back to <FIG>, as shown, the backbone network system <NUM> includes one or more virtual network functions (VNFs) <NUM>, each of which may include one or more virtual network function components (VNFCs) <NUM>. It should be appreciated that the VNFs <NUM> may be embodied as any suitable virtual network functions; similarly, the VNFCs <NUM> may be embodied as any suitable VNF components. For example, in some embodiments, the VNFs <NUM> may include a security gateway (SGW), a packet data network gateway (PNG), a billing function, and/or other virtual network functions. In some embodiments, a particular VNF <NUM> may have multiple sub-instances, which could be executing on the same server <NUM>, <NUM> or different servers <NUM>, <NUM>. In other words, when virtualized, network functions traditionally handled by physical hardware co-located with a particular server <NUM>, <NUM> may be distributed as VNFs <NUM> across one or more of the servers <NUM>, <NUM>. In the illustrative embodiment, the VNFCs <NUM> are processes and/or instances that cooperate to deliver the functionality of one or more VNFs <NUM>. For example, in some embodiments, the VNFCs <NUM> are sub-modules of the VNFs <NUM>. Similar to the VNFs <NUM>, it should be appreciated that the VNFCs <NUM> may be distributed across one or more servers <NUM>, <NUM>. Further, it should be appreciated that a particular VNFC <NUM> may be distributed across multiple servers <NUM>, <NUM> and still form a part of a VNF <NUM> established on a single server <NUM>, <NUM>.

As described herein, in the illustrative embodiment, the VNFs <NUM> of one or more servers <NUM>, <NUM> may communicate with one another, for example, over an inter-VNF communication network <NUM> via one or more inter-VNF communication mechanisms. Similarly, the VNFCs <NUM> of one or more servers <NUM>, <NUM> may communicate with one another, for example, over an inter-VNFC communication network <NUM> via one or more inter-VNFC communication mechanisms. It should be appreciated that the inter-VNF and inter-VNFC communication mechanisms may be embodied as any suitable mechanisms configured to enable inter-VNF and/or inter-VNFC communication. For example, in some embodiments, the VNFs <NUM> and/or VNFCs <NUM> may communicate with one another using an open switch with a hypervisor and packet parsing, formatted packets based on a standard format, shared memory (e.g., physical/virtual memory reserved by the hypervisor), and/or other suitable mechanisms. In the illustrative embodiment, the TEE of the server <NUM>, <NUM> on which a particular VNF <NUM> or VNFC <NUM> is executing is configured to read (directly or indirectly) inter-VNF and inter-VNFC communication associated with the particular VNF <NUM> or VNFC <NUM>.

It should be appreciated that the VNFs <NUM> may process packets into a service chain. However, during operation, one or more runtime threats may be injected into the system, which may circumvent a set of packets or flows from being processed by the entire service chain as required by a particular policy. As such, the TEE of the server <NUM>, <NUM> may be utilized to identify such anomalies and abnormal VNF runtime behavior including, for example, malicious TCP sync floods, packet drops, flow disconnections, violation of application-level policies, and other potential security threats. As such, the TEE may assume a role as the server's security policy inspector.

In the illustrative embodiment of <FIG>, each of the servers <NUM> includes a hypervisor <NUM>, a memory <NUM>, a cache <NUM>, one or more engines <NUM>, one or more network interfaces <NUM>, and a trusted execution environment <NUM>. Additionally, the hypervisor <NUM> includes one or more APIs <NUM>, a virtual switch (vSwitch) <NUM>, one or more encryption tunnels <NUM>, and a shared memory <NUM>. Of course, the servers <NUM> may include additional components in some embodiments, which are omitted for clarity of the description.

The hypervisor <NUM> or virtual machine monitor runs one or more virtual machines (VMs) on the corresponding server <NUM>. As such, the hypervisor <NUM> may establish and/or utilize various virtualized hardware resources (e.g., virtual memory, virtual operating systems, virtual networking components, etc.). The particular APIs <NUM> included in the hypervisor <NUM> and/or the server <NUM> generally may vary depending on the particular server <NUM>. In some embodiments, the APIs <NUM> include one or more proprietary APIs. In some embodiments, the APIs <NUM> may provide access to packets (e.g., associated with a particular VNF <NUM>) such that they may be analyzed by the TEE <NUM>. The virtual switch <NUM> may be utilized to enforce network policies and/or enforce actions (e.g., drop packets, monitor flows, perform deep inspection, perform remediation actions, etc.). For example, the virtual switch <NUM> may permit the networking of virtual machines (VMs) in the system <NUM>. As described below, in some embodiments, the server <NUM> may establish encryption tunnels <NUM> for secure communication (e.g., for communication with the security server <NUM>, between VNFs <NUM>, and/or between VNFCs <NUM>). In some embodiments, the encryption tunnels <NUM> may be read by the TEE <NUM> of the server <NUM> (e.g., in encrypted form or in unencrypted form by virtue of access to the corresponding encryption keys). Additionally, in some embodiments, one or more VMs, VNFs <NUM>, and/or VNFCs <NUM> may utilize the shared memory <NUM>. For example, in some embodiments, the VNFs <NUM> and VNFCs <NUM> may utilize the shared memory <NUM> to communicate with one another. It should be appreciated that the shared memory <NUM> may include physical memory and/or virtual memory depending on the particular embodiment. In the illustrative embodiment, the TEE <NUM> of a particular server <NUM> may access each of the APIs <NUM>, the virtual switch <NUM>, the encryption tunnels <NUM>, and the shared memory <NUM> of that server <NUM> to retrieve data for a security threat analysis of one or more packets/flows. Additionally, the TEE <NUM> may access the inter-VNF and inter-VNFC communications for such an analysis.

As described above, the server <NUM> includes the memory <NUM>, the cache <NUM>, the engines <NUM>, the network interfaces <NUM>, and the TEE <NUM>. It should be appreciated that the memory <NUM> may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. Further, in some embodiments, the memory <NUM> may include software-defined storage. The one or more engines <NUM> may embodied as any hardware, firmware, and/or software components that generate data useful to the TEE <NUM> and/or the security server <NUM> in preparing a security assessment. For example, the engines <NUM> may include a SoC, graphics engine, security engine, audio engine, cryptographic module, TPM, co-processor, communication link or channel, switch, and/or another engine configured to process or otherwise handle data. The network interfaces <NUM> may be embodied as any interface associated with networking processes of data packets. For example, in some embodiments, the network interfaces <NUM> include a Network MAC/Ethernet interface, a software-defined networking module, and/or another network interface.

As indicated above, in the illustrative embodiment, the TEE <NUM> is established as a secure enclave such as Intel® Software Guard Extensions (SGX). However, in other embodiments, the TEE <NUM> may be otherwise established, for example, as a Manageability Engine (ME), trusted platform module (TPM), Innovation Engine (IE), secure partition, separate processor core, and/or otherwise established. For example, in some embodiments, the TEE <NUM> may be embodied as or established by virtue of the security co-processor <NUM>. As discussed herein, the TEE <NUM> is configured to retrieve data from the various components of the server <NUM>, which may be used to perform a security analysis. In some embodiments, the TEE <NUM> may perform a local security analysis based on the retrieved data. Further, in the illustrative embodiment, the TEE <NUM> transmits the security threat assessment data (i.e., the collected data and/or analytic results) to a corresponding TEE <NUM> of the security server <NUM> (i.e., to the nominated TEE <NUM>). It should be appreciated that, in the illustrative embodiment, the TEEs <NUM> may communicate with one another over an out-of-band communication network.

As discussed herein, the nominated TEE <NUM> of the security server <NUM> performs a system-wide (or larger subsystem-wide) security assessment. In some embodiments, the security server <NUM> may communicate with the remediation server <NUM> to request a remediation instruction (i.e., a suitable action to be performed by the server <NUM>) associated with the security assessment. As shown in <FIG>, the remediation server <NUM> may be included within a cloud computing environment <NUM> in which case the remediation server <NUM> may consult with the orchestrator <NUM> to determine an appropriate remediation action/instruction. The remediation server <NUM> and the orchestrator <NUM> may be embodied as any server or computing device capable of performing the functions described herein. Further, the remediation server <NUM> and the orchestrator <NUM> may include components similar to the components of the servers <NUM>, <NUM> described above and/or components commonly found in a server such as a processor, memory, I/O subsystem, data storage, peripheral devices, and so forth, which are not illustrated in <FIG> for clarity of the description.

Referring now to <FIG>, in use, one or more of the servers <NUM>, <NUM> establishes an environment <NUM> for distributed detection of security anomalies. The illustrative environment <NUM> of the server <NUM>, <NUM> includes a security module <NUM>, a trusted execution environment module <NUM>, a communication module <NUM>, a security threat database <NUM>, one or more policies <NUM> (e.g., security and/or configuration policies), and heuristic code <NUM>. Each of the modules of the environment <NUM> may be embodied as hardware, software, firmware, or a combination thereof. Additionally, in some embodiments, one or more of the illustrative modules may form a portion of another module and/or one or more of the illustrative modules may be embodied as a standalone or independent module. For example, each of the modules, logic, and other components of the environment <NUM> may form a portion of, or otherwise be established by, the processor <NUM> of the server <NUM>, <NUM>.

The security module <NUM> is configured to perform various security functions for the server <NUM>. For example, the security module <NUM> may handle the generation and verification of cryptographic keys, signatures, hashes, and/or perform other cryptographic functions.

The trusted execution environment module <NUM> establishes a trusted execution environment (e.g., the TEE <NUM>) or otherwise secure environment within the server <NUM>, <NUM>. As described above, the TEE <NUM> may establish a trusted relationship with a corresponding TEE <NUM> of another server <NUM>, <NUM>. For example, in doing so, the TEEs <NUM> may perform a cryptographic key exchange. In some embodiments, the TEEs <NUM> may communicate with one another over established encrypted and/or otherwise secure tunnels. As described above, in some embodiments, the TEEs <NUM> may communicate with one another over an out-of-band communication channel (i.e., a communication channel separate from a common communication channel between the corresponding servers <NUM>, <NUM>). For example, the TEE <NUM> of one of the servers <NUM> may establish a trusted relationship with the TEE <NUM> of the security server <NUM>. Further, as described above, the TEE <NUM> may read packets of VNFC-VNFC and VNF-VNF networks, retrieve data from the memory <NUM>, the cache <NUM>, the engines <NUM>, and/or the network interfaces <NUM>. Further, in some embodiments, the TEE module <NUM> reads fuses, the memory <NUM>, the data storage <NUM>, and/or other hardware components of the server <NUM>, <NUM> to determine a particular policy <NUM> (e.g., a configuration or security policy) of the server <NUM>, <NUM>. Additionally, the TEE module <NUM> may perform a security assessment of one or more packets of the server <NUM>, <NUM> based on the retrieved information to determine, for example, whether the packets pose a security threat. In doing so, the TEE module <NUM> may retrieve data from a security threat database <NUM> or otherwise correlate retrieved security threat assessment data with the security threat database <NUM>. It should be appreciated that one of the servers <NUM> may perform a local security threat assessment and the security server <NUM> may perform a system-wide (or larger subsystem-wide) security threat assessment. As such, the security threat databases <NUM> of those servers <NUM>, <NUM> may include corresponding data. In some embodiments, the TEE module <NUM> may utilize heuristic code <NUM> in assessing the security of one or more packets. In some embodiments, the heuristic code <NUM> identifies parameters and/or a context in which questionable instructions should be executed (e.g., in a VM or secure container). Additionally or alternatively, the heuristic code <NUM> may identify malicious code signatures, white lists, black lists, and/or otherwise include data useful by the TEE module in assessing the security of one or more packets/instructions.

The communication module <NUM> handles the communication between the server <NUM>, <NUM> and remote devices through a suitable network. For example, as discussed above, the TEEs <NUM> of the servers <NUM>, <NUM> may communicate with one another over an out-of-band communication channel or via encrypted tunnels.

Referring now to <FIG>, in use, the server <NUM> may execute a method <NUM> for distributed detection of security anomalies. The illustrative method <NUM> begins with block <NUM> in which the server establishes a trusted relationship with the security server <NUM>. As discussed above, in some embodiments, the security server <NUM> may be embodied as one of the servers <NUM> that includes a TEE <NUM> that has been selected or "nominated" to perform system-wide or subsystem-wide security analytics. In other embodiments, the security server <NUM> may be embodied as a server separate from the servers <NUM>. It should be appreciated that, in establishing the trusted relationship, the server <NUM> may exchange cryptographic keys with the security server <NUM> in block <NUM> and/or may use a root of trust and/or fuse keys in block <NUM>. For example, the server <NUM> and/or the security server <NUM> may include a cryptographic key or identification bound (e.g., cryptographically) to the server <NUM>, <NUM> or, more particularly, a hardware component of the server <NUM>, <NUM> (e.g., the security co-processor <NUM>).

In block <NUM>, the server <NUM> securely boots. In doing so, the server <NUM> retrieves its configuration policy in block <NUM> (e.g., from secure non-volatile memory of the server <NUM>). In some embodiments, the configuration policy may indicate the execution parameters, contextual information, and/or other information associated with the operation of the server <NUM>. For example, in some embodiments, the configuration policy may be utilized to notify the TEE <NUM> regarding various hardware, firmware, and/or software components of the server <NUM>.

In block <NUM>, the server <NUM> establishes a trusted tunnel with the security server <NUM>. In doing so, the server <NUM> may advertise its aliveness in block <NUM>. To do so, the server <NUM> may communicate with the security server <NUM> to inform the security server <NUM> that the server <NUM> is operational. For example, the server <NUM> may transmit a heartbeat signal to the security server <NUM>. Further, in some embodiments, the server <NUM> may periodically or continuously advertise its aliveness. Additionally or alternatively, the server <NUM> may transmit its security policy and/or heuristic code (e.g., for use in applying a heuristic security algorithm to analyze packet data) to the security server <NUM> in block <NUM>. In some embodiments, the server <NUM> may transmit the entire security policy, whereas in other embodiments, the security server <NUM> may maintain security policies for various servers <NUM>, so that the server <NUM> may just provide the security server <NUM> with recent updates to the security policy rather than the entire security policy. Further, in some embodiments, the security server <NUM> may transmit heuristic code to the server <NUM> for use in assessing security.

In block <NUM>, the server <NUM> determines the runtime posture (e.g., contextual and/or state information) of the server <NUM>. In doing so, in block <NUM>, the server <NUM> may determine the runtime posture of one or more VNFs <NUM> of the server <NUM>. For example, the server <NUM> may determine a current context of the server <NUM> as a function of the VNFs <NUM>, the VNFCs <NUM>, and/or VMs. In block <NUM>, the server <NUM> reads one or more packets of the VNFC-VNFC and/or VNF-VNF networks through the hypervisor <NUM>. In particular, the server <NUM> may read one or more packets of the VNFC-VNFC and/or VNF-VNF networks through the virtual switch <NUM> and/or the network interfaces <NUM>. In block <NUM>, the server <NUM> reads one or more packets associated with the VNFC and/or VNF process execution state from the memory <NUM>, <NUM> and/or the cache <NUM> of the server <NUM>. In block <NUM> of <FIG>, the server <NUM> enables server accesses through microcode (ucode) and/or BIOS of the server <NUM>. In doing so, in block <NUM>, the server <NUM> may read fuses and/or a state of the server <NUM> to determine a policy (e.g., security policy) of the server <NUM>.

In block <NUM>, the server <NUM> may perform a local threat assessment of the server <NUM>. It should be appreciated that the server <NUM> may utilize the policies, heuristic code, runtime posture, packets, and/or other information retrieved or otherwise accessible to the server <NUM>. In some embodiments, the server <NUM> executes one or more heuristic algorithms to perform a security threat assessment. In block <NUM>, the server <NUM> reports the security threat assessment data to the security server <NUM>. In doing so, the server <NUM> may transmit the raw data collected by the server <NUM>, the local security assessment data, and/or intermediate data generated by the server <NUM>.

Depending on the particular embodiment, the server <NUM> may receive remediation action instructions for a network flow/packet from the security server <NUM> or the remediation server <NUM> in block <NUM>. For example, as discussed herein, the security server <NUM> may perform a system-wide threat analysis and/or request assistance from the remediation server <NUM> to determine whether any particular remediation action should be performed by the server <NUM>. If none, the server <NUM> may not receive a response from the security server <NUM> in some embodiments. Of course, in some embodiments, the server <NUM> may independently determine whether to perform a security remediation action. In block <NUM>, the server <NUM> enforces the network policy and/or any remediation action. In some embodiments, the server <NUM> may do so by virtue of the virtual switch <NUM> and/or the network interfaces <NUM>. The particular remediation actions may vary depending on the particular security threat and/or the particular embodiment. For example, in block <NUM>, the server <NUM> may drop one or more network packets based on the remediation instruction. In block <NUM>, the server <NUM> may monitor one or more network flows. For example, in some embodiments, the security server <NUM> or the remediation server <NUM> may instruct the server <NUM> to monitor a particular class of network flows that may pose a security risk based on the threat analysis. Further, in block <NUM>, the server <NUM> may perform a deep packet inspection of one or more network packets based on the remediation instruction. Of course, the server <NUM> may perform a wide variety of other remediation actions depending on the particular embodiment.

Referring now to <FIG>, in use, the security server <NUM> may execute a method <NUM> distributed detection of security anomalies. The illustrative method <NUM> begins with block <NUM> of <FIG> in which the security server <NUM> establishes a trusted relationship one of the servers <NUM>. As described above, in doing so, the security server <NUM> may perform a cryptographic key exchange with the server <NUM> in block <NUM> and/or utilize a root of trust and/or fuse key in block <NUM>. For example, in embodiments in which bilateral trust is established, both the server <NUM> and the security server <NUM> includes a root of trust (e.g., a cryptographically bound key or identifier). In block <NUM>, the security server <NUM> establishes trusted tunnel with the server <NUM> as described above. In doing so, the security server <NUM> may receive a security policy update and/or heuristics code from the server <NUM> in block <NUM>. Additionally or alternatively, the security server <NUM> may transmit heuristic code to the server <NUM> (e.g., for use in assessing the security). Further, in block <NUM>, the security server <NUM> receives security threat assessment data from the server <NUM> based on the received information. As discussed above, the server <NUM> may transmit the raw data collected by the server <NUM>, the local security assessment data, and/or intermediate data generated by the server <NUM> to the security server <NUM> to enable the security server <NUM> to perform a system-wide or subsystem-wide security assessment.

In block <NUM>, the security server <NUM> correlates the security threat assessment data with the security threat database <NUM> to determine whether the analyzed packet(s) pose a security threat to the server <NUM>. In some embodiments, the security server <NUM> may simulate the execution of the packets (e.g., in a VM) based on the posture, security and configuration policies, context, heuristic code, and/or other information regarding the operations of the server <NUM>. Additionally or alternatively, the security server <NUM> may compare the packets to various malware (e.g., virus) signatures, white lists, black lists, and/or other data to determine whether the packets are secure.

Claim 1:
One or more machine-readable storage media having stored therein a plurality of instructions that, when executed, cause a compute device to:
establish a virtual network function (<NUM>) on the compute device to perform a network function, wherein the virtual network function comprises a plurality of virtual network function components (<NUM>) that communicate with each other;
read one or more packets communicated between the virtual network function components;
perform a security threat assessment on the one or more packets communicated between the virtual network function components; and
transmit the security threat assessment to a security compute device;
wherein the plurality of instructions, when executed, further cause the compute device to perform a remediation action on the one or more packets in response to the security threat assessment, wherein the remediation action varies based on the security threat.