Patent Description:
Referring to <FIG>, a packet switched computer networking architecture <NUM> is illustrated. In a typical computer network, a network node is either data communication equipment (DCE) <NUM> or data terminal equipment (DTE) <NUM>. Example DCEs <NUM> include, but are not limited to, routers, switches, and bridges. Example DTEs <NUM> include, but are not limited to, host computers and peripherals, such as printers. In a network, nodes are connected by a transmission medium, such as wired or wireless, and other elements of a communications network <NUM>.

Packet switched computer network operation across DCEs <NUM> can be described as occurring across three "planes" - the data plane <NUM>, the control plane <NUM>, and the management plane <NUM>. The data plane <NUM>, also referred to as the "forwarding plane," is responsible for the actual switching/forwarding of packets over the network by a DCE <NUM> to either another DCE <NUM> or a DTE <NUM> based on a forwarding information base (FIB). The control plane <NUM> is responsible for i) defining the network topology and the network routing protocols, and ii) building/maintaining the FIB in accordance with the topology and routing protocols. The management plane <NUM> is responsible for both user and programmatic interfaces to the control plane <NUM> along with other network housekeeping tasks. Typically, each of both data plane <NUM> functions and control plane <NUM> functions are tightly coupled to the DCEs <NUM>. Document Etsi Group Specification: "<NPL>, discloses network functions virtualisation (NFV) and virtual network functions architectures. Document <NPL>, discloses NETCONF Call Home and RESTCONF Call Home. Aspects of the invention are recited in the independent claims and preferred features are recited in the dependent claims.

Referring to <FIG>, and continuing to refer to <FIG> for context, a software defined network (SDN) architecture <NUM> is illustrated. In an SDN, control plane <NUM> functions can be separated from data plane functions <NUM> and hosted in one or more SDN "controllers" <NUM> outside the data communication equipment (DCEs) <NUM> - thus decoupling the control plane from the DCEs <NUM> and from the data plane <NUM>. For simplicity, the remainder of the disclosure will not explicitly discuss the management plane <NUM>.

Referring to <FIG>, and continuing to refer to prior figures for context, an architecture <NUM> for an SDN employing network function virtualization (NFV) is illustrated. Similar to the manner in which the evolution from architecture <NUM> to an SDN <NUM> decouples the control plane <NUM> from the DCEs <NUM> and the data plane <NUM>, an SDN <NUM> employing NFV as in architecture <NUM> decouples the data plane <NUM> from DCE <NUM> hardware - allowing general purpose compute elements <NUM> (including some DCEs <NUM>) to host virtual network functions (VNFs) <NUM>. NFV uses the approaches of information technology virtualization, such as hypervisors in a network function virtualization infrastructure (NFVI) <NUM>, to virtualize classes of DCE <NUM> functions into building blocks that may connect, or chain together, to create communication services.

In a typical SDN/NFV environment <NUM> a need often exists to change the compute element <NUM> image (the collection of files and data that, when installed, implement the NFV infrastructure) from one version to another - or merely to install such an image on a compute element <NUM>. Such image upgrade often involves a reboot/restart (hereinafter "reboot") to the compute element <NUM> for the changes to take effect. Additionally, the SDN controller <NUM> has the responsibility to check the status of the image upgrade by connecting to the compute element <NUM>.

In a typical SDN/NFV environment <NUM>, the controller <NUM> imports an image to an image management service that is part of, or managed by, the controller <NUM>. The image management service distributes archived versions of the image, for example, "*. gz" or "qcow2" archives, to the compute elements <NUM>. The controller <NUM> then activates the distributed archives by executing a set of configuration commands on the compute element <NUM> that does the installation of the image on the compute element <NUM>. On its own schedule, the compute element <NUM> typically reboots after installation of the image; while independently the controller <NUM> begins polling the compute element <NUM> at some time after the activation, for example, using the PING command. Once the compute element <NUM> finishes rebooting and responds to the PING command from the controller <NUM> (thereby confirming the reachability of the rebooted compute element with respect to the controller <NUM>, the controller <NUM> can issue command line interface (CLI) commands (as a CLI client) to the compute element <NUM> (as a CLI server) to determine the status of the upgrade.

There are several drawbacks to the above approach to managing image upgrades on SDN/NFV compute elements <NUM>. First, the compute element <NUM> may take considerable time to reboot after an image installation - requiring that the controller <NUM> employ a properly-tested and synchronized timeout and retry mechanism.

Second, in the microservice environment employed by some controllers <NUM>, certain microservices are designated to execute the CLI commands on the compute element <NUM>. For example, a network programmer microservice can do a write on the compute element <NUM> and an inventory microservice can do a read on the compute element <NUM> to determine the image upgrade status. The controller <NUM> invokes multiple calls to such microservices. These calls can introduce non-trivial latency, especially at the scale of networks with many compute elements <NUM>.

Third, some controllers <NUM> distribute their functionality across subsystems each dedicated to specific tasks. For example, image distribution may be handled by a subsystem of the controller <NUM> that is separate from the subsystem of the controller <NUM> responsible for monitoring VNFs <NUM>. The polling logic that determines whether a compute element <NUM> device is up and running has to be duplicated across different subsystems that interact with the compute element <NUM> during the upgrade.

Fourth, the rebooted compute element <NUM> might obtain a new Internet Protocol (IP) address from a Dynamic Host Configuration Protocol (DHCP) server. In such cases, a controller <NUM> polling the compute element's <NUM> old IP address will report an upgrade failure.

Fifth, leaving a port open on the rebooted compute element <NUM> listening for controller <NUM> polling may present a security risk.

Sixth, the compute element <NUM> could be behind a firewall with respect to the controller <NUM> - requiring approaches such as a change in firewall settings or the configuration of a virtual private network between the rebooted compute element <NUM> and the controller <NUM>.

The technology disclosed herein provides computer-implemented methods, systems, and computer program products to manage compute element images that support virtual network functions (VNFs) on compute elements in software defined networks. In some examples, a network function virtualization (NFV) compute element installs an image supporting at least one VNF on the compute element. The image includes instructions and data to initiate a Transmission Control Protocol (TCP) connection between the compute element and a Software Defined Network (SDN) controller over a network upon a restart of the compute element.

Upon rebooting the compute element after the installation, the rebooted compute element establishes, as TCP client in accordance with the instructions and using the data, a TCP connection with the controller over the network. The compute element then accepts, as a cryptographic network protocol server, a cryptographic network protocol connection via the TCP connection from the controller as a cryptographic network protocol client in accordance with the instructions. Next, the compute element accepts, as a network management protocol server, a network management protocol connection via the cryptographic network protocol connection from the controller as network management protocol client in accordance with the instructions.

The compute element receives, from the controller over the network management protocol connection, network management commands regarding the status of the virtual network function, and then transmits, to the controller over the network management protocol connection, responses to the received commands in accordance with the instructions.

In some examples, the cryptographic network protocol is one of a Secure Shell (SSH) cryptographic network protocol or a Blocks Extensible Exchange Protocol (BEEP) compliant cryptographic network protocol.

In some examples, the network management protocol is a Network Configuration Protocol (NETCONF) network management protocol. In some such examples, the received network management commands include one or more commands to create a NETCONF remote procedure call (RPC) subscription for the controller with the compute element, and transmitting responses to the received commands includes transmitting responses in accordance with the RPC subscription.

In some examples, the controller is characterized by first Internet Protocol (IP) addresses for monitoring reachability of the compute element and a second IP address for image management and the data comprises the first IP address and the second IP address. In such examples, establishing a TCP connection includes establishing, by the rebooted compute element a first TCP connection to monitor reachability with the first IP address, and a second TCP connection to monitor image installation with the second IP address. In such examples, transmitting includes transmitting reachability responses to the first IP address, and transmitting image installation responses to the second IP address.

In some examples, the rebooted compute element also transmits, to the controller, a public key certificate of the rebooted compute element including an IP address of the compute element. In such examples, at least one of the TCP connection, the cryptographic protocol connection, and the network management protocol connection are conditioned on the authentication of the rebooted compute element by the controller based on the public key certificate.

By using and relying on the methods, systems, and computer program products described herein, the technology disclosed herein provides for management of compute element <NUM> images implementing network function virtualization. As such, the technologies described herein may be employed to determine whether an image installation on a compute element <NUM> was successfully completed.

These and other aspects, objects, features, and advantages of the example embodiments will become apparent to those having ordinary skill in the art upon consideration of the following detailed description of illustrated example embodiments. Turning now to the drawings, in which like numerals represent like (but not necessarily identical) elements throughout the figures, example embodiments are described in detail.

In example architectures for the technology, while some items (such as servers, systems, and devices) shown in the architecture may be represented by one instance of the server, system, or device, multiple instances of each can be used. Further, while certain aspects of operation of the technology are presented in examples related to the figures to facilitate enablement of the claimed invention, additional features of the technology, also facilitating enablement of the claimed invention, are disclosed elsewhere herein.

Referring again to <FIG>, while each element shown in the architecture <NUM> may be represented by one instance of the element, multiple instances of each can be used. Further, while certain aspects of operation of the present technology are presented in examples related to <FIG> to facilitate enablement of the claimed invention, additional features of the present technology, also facilitating enablement of the claimed invention, are disclosed elsewhere herein.

Further network <NUM> includes one or more of a local area network (LAN), a wide area network (WAN), an intranet, an Internet, a storage area network (SAN), a personal area network (PAN), a metropolitan area network (MAN), a wireless local area network (WLAN), a virtual private network (VPN), a cellular or other mobile communication network, a BLUETOOTH ® wireless technology connection, any combination thereof, and any other appropriate architecture or system that facilitates the communication of signals, data, and/or messages. Throughout the discussion of example embodiments, it should be understood that the terms "data" and "information" are used interchangeably herein to refer to text, images, audio, video, or any other form of information that can exist in a computer-based environment.

The network connections illustrated are examples and other approaches for establishing a communications link between the computers and devices can be used. Additionally, those having ordinary skill in the art and having the benefit of this disclosure will appreciate that the network devices illustrated in the figures may have any of several other suitable computer system configurations, and may not include all the components described above.

In example embodiments, the network computing devices, and any other computing machines associated with the technology presented herein, may be any type of computing machine such as, but not limited to, those discussed in more detail with respect to Figure <NUM>. Furthermore, any functions, applications, or components associated with any of these computing machines, such as those described herein or any others (for example, scripts, web content, software, firmware, hardware, or modules) associated with the technology presented herein may by any of the components discussed in more detail with respect to Figure <NUM>. The computing machines discussed herein may communicate with one another, as well as with other computing machines or communication systems over one or more networks, such as network <NUM>. Each network may include various types of data or communications network, including any of the network technology discussed with respect to Figure <NUM>.

The examples illustrated in the following figures are described hereinafter with respect to the components of the example operating environment and example architectures <NUM>, <NUM>, and <NUM> described elsewhere herein. The example embodiments may also be practiced with other systems and in other environments. The operations described with respect to the example processes can be implemented as executable code stored on a computer or machine readable non-transitory tangible storage medium (e.g., floppy disk, hard disk, ROM, EEPROM, nonvolatile RAM, CD-ROM, etc.) that are completed based on execution of the code by a processor circuit implemented using one or more integrated circuits. The operations described herein also can be implemented as executable logic that is encoded in one or more non-transitory tangible media for execution (e.g., programmable logic arrays or devices, field programmable gate arrays, programmable array logic, application specific integrated circuits, etc.).

Referring to <FIG>, and continuing to refer to prior figures for context, a block flow diagram depicting methods <NUM> to upgrade network function virtualization (NFV) images on compute elements <NUM> is shown, in accordance with certain examples of the technology disclosed herein. In such methods, an NFV compute element <NUM> installs an image implementing at least the NFV infrastructure (NFVI) to support at least one virtualized network function (VNF) <NUM> on the compute element <NUM> - Block <NUM>. The image includes instructions and data (including a configuration file) to initiate an Internet protocol suite transport layer connection between the compute element <NUM> and its controller <NUM> over network <NUM> upon a reboot of the compute element <NUM>.

Consider, as a continuing example, a Cisco Digital Network Architecture (DNA) Center (DNAC) as controller <NUM>, with integrated Software Image Management (SWIM). The DNAC <NUM> can be used to build a SDN hierarchy across sites, buildings, and floors; define device credentials, discover devices, import software images using SWIM functions (for example, tagging an existing image as "golden," or downloading an image from an online server or local host); and distribute the image to a compute element <NUM> for later or immediate activation in support of one or more VNFs <NUM>. Upon activation the compute element <NUM> installs the image. After the image has been installed in the compute element <NUM>, the compute element <NUM> reboots - Block <NUM>.

From the perspective of the controller <NUM>, after activation of the image on the compute element <NUM>, the controller <NUM> listens for Internet Protocol suite transport layer connections on a port designated for such. In the continuing example, the Internet Protocol transport layer connection is a Transmission Control Protocol (TCP) connection and the port is <NUM>, corresponding to the NETwork CONFiguration (NETCONF) protocol "callhome" Secure Shell (SSH) port. TABLE <NUM> presents an example command string for opening such a port in the controller <NUM>.

In other examples, the NETCONF callhome Transport Layer Security (TLS) port <NUM> can be used. In yet other examples, the technology uses the REpresentational State Transfer (REST) CONFiguration (RESTCONF) callhome port <NUM>.

The rebooted compute element <NUM>, as an Internet protocol suite transport layer client, establishes an Internet protocol suite transport layer connection over the network <NUM> with the controller <NUM> in accordance with the instructions and using the data - Block <NUM>. In the continuing example the compute element <NUM>, on a successful reboot after image installation, initiates a TCP connection with the controller <NUM> via the controller's port <NUM> using a daemon process and information from a configuration file - each of which is included in the image. The information can include the SWIM IP address (and other controller <NUM> IP addresses, for example, if the controller <NUM> distributes functions across multiple IP addresses) along with the compute element <NUM> credentials.

Once the TCP connection is established, the compute element <NUM> is reachable by the controller <NUM> over network <NUM>. This portion of the technology differs from conventional operation in that the controller <NUM>, not the compute element <NUM>, is typically the "client" initiator of communication connections. In the continuing example, a NETCONF protocol approach is used not only in the Internet protocol suite transport layer, but also in the Internet protocol suite application layer as described below.

This approach has several advantages over the polling mechanism described above. For example, the controller <NUM> does not require a properly-tested and synchronized timeout and retry mechanism for polling each compute element <NUM>, given that the compute element <NUM> is the TCP connection initiator. As another example, the controller <NUM> will not have to proliferate calls to microservices (which introduces latency and requires processing resources) in order to poll/ping possibly-rebooted compute elements <NUM>. In controllers <NUM> with functionality distributed across different nodes or clusters, each of those clusters will not have to spend processing resources, memory resources, and bandwidth to initiate interaction with a potentially large number of rebooted compute elements <NUM>. Further, each compute element <NUM> does not have to open a port to listen for polls/pings from the controller <NUM>.

The compute element <NUM>, as a cryptographic network protocol server, accepts a cryptographic network protocol connection via the established transport layer connection from the controller <NUM> as a cryptographic network protocol client in accordance with the instructions - Block <NUM>. In the continuing example, the compute element <NUM> acts as a Secure Shell (SSH) server, and accepts an SSH connection initiated by the controller <NUM>. SSH provides a secure channel over an unsecured network in a client-server architecture, connecting an SSH client application (in this case an application running on the controller <NUM>) with an SSH server application running on the compute element <NUM>. In some examples, the cryptographic network protocol is a Blocks Extensible Exchange Protocol (BEEP) compliant cryptographic network protocol.

Referring to <FIG>, and continuing to refer to prior figures for context, a block flow diagram depicting methods <NUM> to upgrade images on compute elements <NUM> is shown, in accordance with certain examples of the technology disclosed herein. In such methods <NUM>, as a part of establishing an SSH connection, Block <NUM>, the compute element <NUM> sends its host key or certificate, for example, an X. <NUM> public key certificate, to the controller <NUM> - Block <NUM>. In the continuing example, the controller <NUM> uses the certificate to validate the identity of the rebooted compute element <NUM>. In some examples, the compute element <NUM> has been assigned a trusted key from the compute element <NUM> manufacturer that is common across a plurality of compute elements <NUM> and that can be verified by a third party certificate authority. In such cases controller <NUM> has to trust only a single certificate that validates the authenticity of the various compute elements <NUM>. In some examples, the compute element <NUM>, receives a new IP address upon reboot from a DHCP server. The new IP address can be incorporated into the compute element's <NUM> certificate, from which the controller <NUM> can extract the new IP address.

Establishing the SSH session can also include the authentication of the controller <NUM> credentials which the compute element <NUM> can read from a configuration file that was part of the image installation. The compute element <NUM> can implement a persistent SSH connection with the controller. As the connection initiator, the compute element <NUM> can actively test the aliveness of the established connection using a keep-alive mechanism.

Returning to <FIG>, the compute element <NUM> accepts, as a network management protocol server, a network management protocol connection via the cryptographic network protocol connection from the controller <NUM> as network management protocol client in accordance with the instructions - Block <NUM>. In the continuing example, the compute element <NUM> acts as a NETCONF server and accepts a NETCONF connection on top of the SSH connection from controller <NUM>. NETCONF is a network management protocol that provides mechanisms to install, manipulate, and delete the configuration of network devices. NETCONF's operations are realized on top of a Remote Procedure Call (RPC) layer. The NETCONF protocol uses an Extensible Markup Language (XML) based data encoding for the configuration data as well as the protocol messages.

The compute element <NUM> receives, from the controller <NUM> over the network management protocol connection, network management commands regarding the status of the upgrade - Block <NUM>. In the continuing example, the compute element <NUM> receives NETCONF instructions creating an RPC subscription that sends upgrade event reports from the compute element <NUM> to the controller <NUM> over the layered TCP/SSH/NETCONF connection. TABLE <NUM> illustrates a NETCONF command from the controller <NUM> to the compute element for creating such an RPC subscription. In some examples, using the NETCONF client session, the controller <NUM> can also use GET RPC calls instead of the notification to determine the status of the image upgrade.

The compute element <NUM> transmits, to the controller <NUM> over the network management protocol connection, responses to the received commands in accordance with the instructions - Block <NUM>. Referring to <FIG>, and continuing to refer to prior figures for context, a user interface <NUM> of the controller <NUM> reporting events from the RPC subscription created by the command of TABLE <NUM> is shown, in accordance with certain examples of the technology disclosed herein. The user interface <NUM> includes a temporal indication <NUM>, event type <NUM>, event description <NUM>, and event status <NUM> for each event, along with interface controls for search <NUM>, the number of entries per page <NUM>, which entries are being displayed <NUM>, and page status/control <NUM>. TABLE <NUM> presents example NETCONF syntax message from the compute element <NUM> to the controller <NUM> as a response indicating an "In process" upgrade - entry <NUM> in <FIG>. Other event descriptions <NUM> include "restoring VMs" and "successfully upgraded.

<FIG> depicts a computing machine <NUM> and a module <NUM> in accordance with certain example embodiments. The computing machine <NUM> may correspond to any of the various computers, servers, mobile devices, embedded systems, or computing systems presented herein. The module <NUM> may comprise one or more hardware or software elements configured to facilitate the computing machine <NUM> in performing the various methods and processing functions presented herein. The computing machine <NUM> may include various internal or attached components, for example, a processor <NUM>, system bus <NUM>, system memory <NUM>, storage media <NUM>, input/output interface <NUM>, and a network interface <NUM> for communicating with a network <NUM>.

The computing machine <NUM> may be implemented as a conventional computer system, an embedded controller, a laptop, a server, a mobile device, a smartphone, a set-top box, a kiosk, a vehicular information system, one more processors associated with a television, a customized machine, any other hardware platform, or any combination or multiplicity thereof. The computing machine <NUM> may be a distributed system configured to function using multiple computing machines interconnected via a data network or bus system.

The processor <NUM> may be configured to execute code or instructions to perform the operations and functionality described herein, manage request flow and address mappings, and to perform calculations and generate commands. The processor <NUM> may be configured to monitor and control the operation of the components in the computing machine <NUM>. The processor <NUM> may be a general purpose processor, a processor core, a multiprocessor, a reconfigurable processor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gated logic, discrete hardware components, any other processing unit, or any combination or multiplicity thereof. The processor <NUM> may be a single processing unit, multiple processing units, a single processing core, multiple processing cores, special purpose processing cores, co-processors, or any combination thereof. According to certain embodiments, the processor <NUM> along with other components of the computing machine <NUM> may be a virtualized computing machine executing within one or more other computing machines.

The system memory <NUM> may include non-volatile memories, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash memory, or any other device capable of storing program instructions or data with or without applied power. The system memory <NUM> may also include volatile memories, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), and synchronous dynamic random access memory (SDRAM). Other types of RAM also may be used to implement the system memory <NUM>. The system memory <NUM> may be implemented using a single memory module or multiple memory modules. While the system memory <NUM> is depicted as being part of the computing machine <NUM>, one skilled in the art will recognize that the system memory <NUM> may be separate from the computing machine <NUM> without departing from the scope of the subject technology. It should also be appreciated that the system memory <NUM> may include, or operate in conjunction with, a non-volatile storage device, for example, the storage media <NUM>.

The storage media <NUM> may include a hard disk, a floppy disk, a compact disc read only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray disc, a magnetic tape, a flash memory, other non-volatile memory device, a solid state drive (SSD), any magnetic storage device, any optical storage device, any electrical storage device, any semiconductor storage device, any physical-based storage device, any other data storage device, or any combination or multiplicity thereof. The storage media <NUM> may store one or more operating systems, application programs and program modules, for example, module <NUM>, data, or any other information. The storage media <NUM> may be part of, or connected to, the computing machine <NUM>. The storage media <NUM> may also be part of one or more other computing machines that are in communication with the computing machine <NUM>, for example, servers, database servers, cloud storage, network attached storage, and so forth.

The module <NUM> may comprise one or more hardware or software elements configured to facilitate the computing machine <NUM> with performing the various methods and processing functions presented herein. The module <NUM> may include one or more sequences of instructions stored as software or firmware in association with the system memory <NUM>, the storage media <NUM>, or both. The storage media <NUM> may therefore represent examples of machine or computer readable media on which instructions or code may be stored for execution by the processor <NUM>. Machine or computer readable media may generally refer to any medium or media used to provide instructions to the processor <NUM>. Such machine or computer readable media associated with the module <NUM> may comprise a computer software product. It should be appreciated that a computer software product comprising the module <NUM> may also be associated with one or more processes or methods for delivering the module <NUM> to the computing machine <NUM> via the network <NUM>, any signal-bearing medium, or any other communication or delivery technology. The module <NUM> may also comprise hardware circuits or information for configuring hardware circuits, for example, microcode or configuration information for an FPGA or other PLD.

The input/output (I/O) interface <NUM> may be configured to couple to one or more external devices, to receive data from the one or more external devices, and to send data to the one or more external devices. Such external devices along with the various internal devices may also be known as peripheral devices. The I/O interface <NUM> may include both electrical and physical connections for operably coupling the various peripheral devices to the computing machine <NUM> or the processor <NUM>. The I/O interface <NUM> may be configured to communicate data, addresses, and control signals between the peripheral devices, the computing machine <NUM>, or the processor <NUM>. The I/O interface <NUM> may be configured to implement any standard interface, for example, small computer system interface (SCSI), serial-attached SCSI (SAS), Fibre Channel, peripheral component interconnect (PCI), PCI express (PCIe), serial bus, parallel bus, advanced technology attached (ATA), serial ATA (SATA), universal serial bus (USB), Thunderbolt, FireWire, various video buses, and the like. The I/O interface <NUM> may be configured to implement only one interface or bus technology. Alternatively, the I/O interface <NUM> may be configured to implement multiple interfaces or bus technologies. The I/O interface <NUM> may be configured as part of, all of, or to operate in conjunction with, the system bus <NUM>. The I/O interface <NUM> may include one or more buffers for buffering transmissions between one or more external devices, internal devices, the computing machine <NUM>, or the processor <NUM>.

The I/O interface <NUM> may couple the computing machine <NUM> to various input devices including mice, touch-screens, scanners, electronic digitizers, sensors, receivers, touchpads, trackballs, cameras, microphones, keyboards, any other pointing devices, or any combinations thereof. The I/O interface <NUM> may couple the computing machine <NUM> to various output devices including video displays, speakers, printers, projectors, tactile feedback devices, automation control, robotic components, actuators, motors, fans, solenoids, valves, pumps, transmitters, signal emitters, lights, and so forth.

The computing machine <NUM> may operate in a networked environment using logical connections through the network interface <NUM> to one or more other systems or computing machines across the network <NUM>. The network <NUM> may include wide area networks (WAN), local area networks (LAN), intranets, the Internet, wireless access networks, wired networks, mobile networks, telephone networks, optical networks, or combinations thereof. The network <NUM> may be packet switched, circuit switched, of any topology, and may use any communication protocol. Communication links within the network <NUM> may involve various digital or analog communication media, for example, fiber optic cables, freespace optics, waveguides, electrical conductors, wireless links, antennas, radio-frequency communications, and so forth.

The processor <NUM> may be connected to the other elements of the computing machine <NUM> or the various peripherals discussed herein through the system bus <NUM>. It should be appreciated that the system bus <NUM> may be within the processor <NUM>, outside the processor <NUM>, or both. According to certain example embodiments, any of the processor <NUM>, the other elements of the computing machine <NUM>, or the various peripherals discussed herein may be integrated into a single device, for example, a system on chip (SOC), system on package (SOP), or ASIC device.

Embodiments may comprise a computer program that embodies the functions described and illustrated herein, wherein the computer program is implemented in a computer system that comprises instructions stored in a machine-readable medium and a processor that executes the instructions. However, it should be apparent that there could be many different ways of implementing embodiments in computer programming, and the embodiments should not be construed as limited to any one set of computer program instructions. Further, a skilled programmer would be able to write such a computer program to implement an embodiment of the disclosed embodiments based on the appended flow charts and associated description in the application text. Therefore, disclosure of a particular set of program code instructions is not considered necessary for an adequate understanding of how to make and use embodiments. Further, those skilled in the art will appreciate that one or more aspects of embodiments described herein may be performed by hardware, software, or a combination thereof, as may be embodied in one or more computing systems. Additionally, any reference to an act being performed by a computer should not be construed as being performed by a single computer as more than one computer may perform the act.

The example embodiments described herein can be used with computer hardware and software that perform the methods and processing functions described previously. The systems, methods, and procedures described herein can be embodied in a programmable computer, computer-executable software, or digital circuitry. The software can be stored on computer-readable media. For example, computer-readable media can include a floppy disk, RAM, ROM, hard disk, removable media, flash memory, memory stick, optical media, magneto-optical media, CD-ROM, etc. Digital circuitry can include integrated circuits, gate arrays, building block logic, field programmable gate arrays (FPGA), etc..

The example systems, methods, and acts described in the embodiments presented previously are illustrative, and, in alternative embodiments, certain acts can be performed in a different order, in parallel with one another, omitted entirely, and/or combined between different example embodiments, and/or certain additional acts can be performed, without departing from the scope of the claims. Accordingly, such alternative embodiments are included in the scope of the following claims, which are to be accorded the broadest interpretation so as to encompass such alternate embodiments.

For example, in "high availability" (HA) scenarios the image management service portion of the controller <NUM> may have multiple instances which are load balanced with a HA proxy. The HA proxy address can be stored on the configuration file which the compute element <NUM> uses to initiate the transport layer connection. In such cases the notification is routed to one of the instances in the HA cluster.

As another example, through the use of keep-alive and retry-connection mechanism NETCONF connections can be persisted in case the controller <NUM> misses some transport layer connections so that the compute element <NUM> can retry the connection. This can help on a scale setup where the controller <NUM> manages many devices. In such cases it can listen for the appropriate port (<NUM> in the case of NETCONF) on all the controller <NUM> devices and can open NETCONF client session on each of the devices. If a device transport layer connection is missed, keep alive and retry mechanism at the compute element <NUM> side can be used to persist the connection and retry. If an active NETCONF session is disconnected due to some network connection issues the compute element <NUM> can restart the process by doing a keep alive check on the established connection.

In yet another example, the compute element image may implement operating system (OS) level virtualization in which the kernel allows the existence of multiple isolated user-space instances. OS-level virtualization is also known as "containerization. " In another example, the technology can employ a transport layer protocol other than transport control protocol (TCP) - for example, user datagram protocol (UDP) employed with corresponding cryptographic network and network management protocols.

Network function virtualization can use virtualization technologies to virtualize physical network elements like routers and firewalls. A virtualized network function or VNF can include one or more virtual machines runs different network functions on top of an NFV compute device. Below is a step wise process of how to bring up a VNF on an NFV compute element.

Install a hypervisor (e.g. NFVIS, ESXi, openstack) on a standard X86 based compute servers. As part of this installation hypervisor software is installed on the compute element. The hypervisor helps to represent the hardware elements like storage, network and memory present on the compute machine. An NFV-Orchestrator that is typically part of the hypervisor software helps to orchestrate or create VNFs. Creating a VNF calls for an image, storage, CPU, network and memory - all can be created using the orchestrator. Once the image is uploaded in the compute device, NFV orchestrator will use that image and create a virtual machine with the given storage, CPU, network, and disk.

As we note here there are two images associated with NFV, one is the hypervisor software and the other is the image of the VNF. The image of hypervisor implements the new/upgraded capabilities to bring up a new VNF.

In summary, a network function virtualization (NFV) compute element installs an image supporting a virtualized network function (VNF) on the element. The image includes instructions/data to initiate a TCP connection between the element and a Software Defined Network (SDN) controller upon reboot of the element. Upon rebooting, the element establishes, as client in accordance with the instructions/data, a TCP connection with the controller. The element then accepts, as a cryptographic network protocol server, a connection via the TCP connection from the controller as a client in accordance with the instructions. Next, the element accepts, as a network management protocol server, a connection via the cryptographic network protocol connection from the controller as network management protocol client. The element receives, from the controller over the network management protocol connection, commands regarding the status of the rebooted compute element, and then transmits, to the controller over the network management protocol connection, responses to the commands.

Although specific embodiments have been described above in detail, the description is merely for purposes of illustration.

Claim 1:
A method (<NUM>), comprising:
installing (<NUM>), by a network function virtualization compute element (<NUM>), an image supporting at least one virtualized network function on the compute element, the image comprising instructions and data to initiate a Transmission Control Protocol, TCP, connection between the compute element and a Software Defined Network, SDN, controller (<NUM>) over a network upon a reboot of the compute element;
rebooting (<NUM>) the compute element after the installing;
establishing (<NUM>), by the rebooted compute element as TCP client in accordance with the instructions and using the data, a TCP connection with the controller over the network;
accepting (<NUM>, <NUM>), by the compute element as a cryptographic network protocol server, a cryptographic network protocol connection via the TCP connection from the controller as a cryptographic network protocol client in accordance with the instructions;
accepting (<NUM>), by the compute element as a network management protocol server, a network management protocol connection via the cryptographic network protocol connection from the controller as network management protocol client in accordance with the instructions;
receiving (<NUM>), by the compute element from the controller over the network management protocol connection, network management commands regarding a status of the rebooted compute element; and
transmitting (<NUM>), by the compute element to the controller over the network management protocol connection, responses to the received commands in accordance with the instructions.