Patent Description:
Large enterprises often face the challenge of mass distribution and deployment of network infrastructure. For example, an enterprise may operate a number of geographically distributed facilities (e.g., offices, retail outlets, and the like) that require network connectivity to a central or main office of the enterprise and optionally to each other. In such cases, a challenge arises when the enterprise desires to install or upgrade network devices with each of the many remote facilities. For example, a central information technology (IT) administrative group of the enterprise may coordinate an effort to upgrade computers, firewalls, gateways, routers, VPN appliances, switches or other network equipment in each of the remote facilities. Such operations may require deployment and activation of hundreds or sometimes thousands of devices.

To simplify the process, the enterprise may purchase similar network devices for deployment at the remote facilities in a single mass rollout. By purchasing similar if not the same network devices, the enterprise may ease administrative burdens with respect to deploying and operating these network devices. In such cases, it is common for the enterprise to contract with the manufacturer to ship the units directly to the remote facilities. This saves shipping costs and offers the advantage of alleviating the central IT group from having the burden of physically receiving and reshipping the devices. However, when devices are shipped directly from the manufacturer to the final location at which the devices are to be deployed, it may not always be possible for a trained network administrator to physically manipulate the devices to ensure proper installation and activation. As a result, the person who configures the devices is typically a store manager or other person who does not have experience in configuring network devices. In such cases, it may be difficult to ensure that the devices are correctly deployed and activated in a manner so as to match a centralized device management system often maintained by the IT group for remotely monitoring and managing devices in the enterprise. <CIT> relates to methods of performing a device management bootstrap. <CIT> relates to a chip card with volatile memory for storing algorithm code. <CIT> relates to a method for automatically initializing a network device.

In general, the disclosure describes techniques for performing a secure remote bootstrapping operation of a network device. To reduce the technical complexity of configuring a network device to an unsophisticated user, conventional devices may use the Zero-touch Provisioning (ZTP) to remotely and automatically perform a first-time configuration of a network device for use within an enterprise network. Additional information with respect to the ZTP protocol is described in "Zero Touch Provisioning for Networking Devices," Internet-Draft, Internet Engineering Task Force (IETF), available at https://tools. org/html/draft-ietf-netconf-zerotouch.

However, network devices that are deployed to public places (e.g., kiosks, shopping malls, offices, apartment basements) may still be vulnerable to attack by a malicious actor with physical access to the device. For example, a conventional network device may store sensitive information, such as user data and credentials, network traffic, or the address and identity of other devices within the enterprise network, within a non-volatile memory of the network device. A malicious actor with physical access to the network device potentially could remove the non-volatile memory and glean valuable information from analysis of data stored on the non-volatile memory. One potential solution would be to configure the network device to store such sensitive information within a volatile memory of the network device, such that the sensitive information is lost upon loss of power to the device. However, this may be beyond the ability of an end-user that relies upon ZTP to perform first-time configuration of the network device. Furthermore, some security configurations should be performed in non-volatile memory to prevent loss of power from removing the security configurations, further increasing the technical burden on the end-user.

Accordingly, techniques are disclosed for performing a secure, remote bootstrapping operation of a network device such that sensitive information is stored only on a volatile memory of the network device. For example, a network device as described herein may, during an initial boot cycle, perform a first touchless provisioning operation to retrieve onboarding information. The network device may process the onboarding information to determine whether a first initialization of the network device has occurred. If the first initialization has not yet occurred, the network device performs the first initialization. For example, the network device may perform the first initialization by configuring local user access permissions and configuring itself to mount at least a portion of a file system to a volatile memory of the network device and not a non-volatile memory of the network device. Further, the network device reboots itself.

For each subsequent boot cycle, the network device mounts at least a portion of the file system to the volatile memory and/or encrypts at least a portion of the non-volatile memory. Further, the network device performs a second touchless provisioning operation to request the onboarding information. The network device processes the onboarding information to determine whether the first initialization of the network device has occurred. In response to determining that the first initialization has occurred, the network device performs a bootstrapping operation. As examples of bootstrapping operations, the network device may configure itself for remote management by another network device.

Thus, by using the techniques of the disclosure, the network device may perform a secure remote bootstrapping operation. Further, upon loss of power to the network device, the network device loses information stored on the volatile memory or access to encrypted information stored on the non-volatile memory. Thus, the network device does not retain any information that may be of use to a malicious actor that has physical access to the network device. Accordingly, the techniques of the disclosure provide specific technical improvements to the computer-related field of network device deployment and configuration. For example, the techniques of the disclosure may allow for the deployment to public spaces of network devices that are secured from physical or local attack by a malicious actor. Further, the techniques of the disclosure may allow for the use of touchless provisioning to securely and remotely configure a network device. Additionally, the techniques of the disclosure may reduce the cost and technical burden on the end user of deploying and configuring network devices to operate within an enterprise network.

In one example, this disclosure describes a method comprising: performing, by one or more processors of a network device, a first request for onboarding information for the network device; processing, by the one or more processors, the onboarding information to determine that a first initialization of the network device has not occurred; in response to determining that the first initialization of the network device has not occurred, performing the first initialization by: configuring, by the one or more processors and with the onboarding information, the network device to mount at least a portion of a file system to a volatile memory of the network device and not a non-volatile memory of the network device; and rebooting, by the one or more processors, the network device; after rebooting the network device: performing, by the one or more processors, a second request for the onboarding information for the network device; processing, by the one or more processors, the onboarding information to determine that the first initialization of the network device has occurred; and in response to determining that the first initialization of the network device has occurred, performing, by the one or more processors, a bootstrapping operation of the network device.

In another example, this disclosure describes a network device comprising: a non-volatile memory; a volatile memory; and one or more processors configured to: perform a first request for onboarding information for the network device; process the onboarding information to determine that a first initialization of the network device has not occurred; in response to determining that the first initialization of the network device has not occurred, perform the first initialization by: configuring, with the onboarding information, the network device to mount at least a portion of a file system to the volatile memory and not the non-volatile memory; and rebooting the network device; after rebooting the network device: perform a second request for the onboarding information for the network device; process the onboarding information to determine that the first initialization of the network device has occurred; and in response to determining that the first initialization of the network device has occurred, perform a bootstrapping operation of the network device.

In another example, this disclosure describes a computer-readable medium comprising instructions configured to cause one or more processors of a network device to: perform a first request for onboarding information for the network device; process the onboarding information to determine that a first initialization of the network device has not occurred; in response to determining that the first initialization of the network device has not occurred, perform the first initialization by: configuring, with the onboarding information, the network device to mount at least a portion of a file system to a volatile memory of the network device and not a non-volatile memory of the network device; and rebooting the network device; after rebooting the network device: perform a second request for the onboarding information for the network device; process the onboarding information to determine that the first initialization of the network device has occurred; and in response to determining that the first initialization of the network device has occurred, perform a bootstrapping operation of the network device.

The details of one or more examples of the techniques of this disclosure are set forth in the accompanying drawings and the description below.

<FIG> is a block diagram illustrating example system <NUM> for performing a secure remote bootstrapping operation in accordance with the techniques of the disclosure. In general, this disclosure describes techniques for securely and remotely configuring network device <NUM>. To deploy and manage network device <NUM> with a centralized management system (e.g., orchestrator server <NUM> and voucher server <NUM>), network device <NUM> solicits network settings, management configurations, and security configurations automatically upon power-up. In accordance with the techniques of this disclosure, network device <NUM> receives configuration information and a boot image from, e.g., bootstrap server <NUM>.

System <NUM>, in the example of <FIG>, may include a plurality of sub-networks, e.g., service provider network <NUM> and customer network <NUM>. Service provider network <NUM> provides to customer network <NUM> network services that are available for request and use by customer devices <NUM> within customer network <NUM>. As will be discussed in greater detail below, service provider network <NUM> provides, via voucher server <NUM>, authentication and verification services to customer network <NUM>. In some examples, service provider network <NUM> is an Internet Service Provider (ISP) that provides customer network <NUM> with access to one or more external networks such as, for example, the Internet.

Customer network <NUM> may correspond to, for example, a retail outlet or a corporate division (e.g., legal, engineering, marketing, sales, accounting, etc.). Customer network <NUM> may correspond to a new subnetwork for which new network device <NUM> is to be enabled. Customer network <NUM> includes orchestrator server <NUM> that manages each network element (e.g., router, switch, gateway, VPN appliance, firewall, and the like) within customer network <NUM>, DHCP server <NUM> that provides DHCP services to devices within customer network <NUM>, bootstrap server <NUM> that provides configuration and bootstrap services to devices within customer network <NUM>, one or more customer devices <NUM>, and network device <NUM> that provides network traffic routing and forwarding services to customer devices <NUM>. For ease of discussion, customer network <NUM> is depicted as including a single network device <NUM>, a single DHCP server <NUM>, and a single bootstrap server <NUM>. However, in other examples of the techniques of the disclosure, customer network <NUM> includes a plurality of network devices <NUM>, a plurality of DHCP servers <NUM>, a plurality of bootstrap servers <NUM>, or some combination thereof.

Orchestrator server <NUM> is communicatively coupled to network device <NUM> of customer network <NUM>. Orchestrator server <NUM> may obtain ownership vouchers <NUM> from voucher server <NUM> of service provider network <NUM>, as described in greater detail below. Orchestrator server <NUM> may provide DHCP configuration <NUM> to DHCP server <NUM> to enable DHCP server <NUM> to provide DHCP services to devices within customer network <NUM>, such as network device <NUM>. Additionally, orchestrator server <NUM> may provide bootstrap configuration information <NUM> to bootstrap server <NUM> to enable bootstrap server <NUM> to provide bootstrapping services to network device <NUM>. Once network device <NUM> is deployed and activated, orchestrator server <NUM> may manage network device <NUM> using a communications protocol, such as NETCONF. Managed network device <NUM> is also referred to herein as a network "element. " In common practice, orchestrator server <NUM> and network device <NUM> managed by orchestrator server <NUM> are centrally maintained by an IT group of the customer and are collective referred to as an element management system (EMS) or a network management system (NMS). An administrator may interact with orchestrator server <NUM> to remotely monitor and configure network device <NUM>. For example, the administrator may receive alerts from orchestrator server <NUM> regarding network device <NUM>, view configurations or management data of network device <NUM>, modify the configurations or management data of network device <NUM>, add new network devices to customer network <NUM>, remove existing network device <NUM> from customer network <NUM>, or otherwise manipulate customer network <NUM> and network device <NUM>. In some examples, orchestrator server <NUM> provides a device management interface (DMI) for an administrator to manage network device <NUM> once network device <NUM> becomes active. The DMI may comprise an interface, such as a graphical user interface (GUI) or command line interface, by which the administrator dynamically adjusts configuration data for network device <NUM> or other devices managed by orchestrator server <NUM>.

DHCP server <NUM> provides DHCP services to devices within customer network <NUM>, such as network device <NUM>, bootstrap server <NUM>, and customer devices <NUM>. As described herein, network device <NUM> may request bootstrapping data from DHCP server <NUM> via a touchless provisioning operation to perform automatic configuration of network device <NUM>. For example, network device <NUM> may request a network assignment from DHCP server <NUM> as well as redirect information that redirects network device <NUM> to bootstrap server <NUM> for onboarding information. In some examples, the touchless provisioning operation is a ZTP operation.

Bootstrap server <NUM> may be used as a source of onboarding information for network device <NUM>. As described herein, network device <NUM> may request onboarding information from bootstrap server <NUM> via a touchless provisioning operation to perform automatic configuration of network device <NUM>. In some examples, the touchless provisioning operation is a ZTP operation. In some examples, Bootstrap server <NUM> is a RESTCONF server implementing a YANG module.

Network device <NUM> generally is a network device that may be difficult for inexperienced, non-technical users to operate. Network device <NUM>, for example, typically does not include a keyboard, a monitor, or other conventional user interfaces. Accordingly, a console is not generally accessible when starting network device <NUM>. Network device <NUM> may correspond, for example, to a router, switch, gateway, hub, server, computing device, computing terminal, printer, firewall, intrusion detection and/or prevention device, wireless Access Point (AP), or other type of network device. An inexperienced user, such as a store manager, may have difficulty manually configuring network device <NUM>. Accordingly, the techniques of this disclosure may simplify the process of configuring network devices <NUM> through the use of a secure touchless provisioning operation, which is used by network device <NUM> to configure itself during a boot cycle, at which time an input console is typically not available.

A boot cycle is generally a process during which a processor of a device, such as one of network devices <NUM>, "bootstraps" loading of an operating system kernel. Generally, the processor includes hard-coded instructions to retrieve a boot loader from a defined memory address following an initial receipt of power, that is, after the device is initially turned on. The boot loader includes bootstrapping instructions for loading the kernel, as well as for initializing variables, such as various register values. In some examples, the boot loader may include instructions for performing a touchless provisioning operation to retrieve onboarding information used for loading the kernel and/or mounting a file system for network device <NUM>.

Customer devices <NUM> may be, for example, personal computers, laptop computers or other types of computing devices associated with users of customer network <NUM>. Additional examples of customer devices <NUM> include mobile telephones, laptop or desktop computers having, e.g., a <NUM> or <NUM> wireless card, wireless-capable netbooks, video game devices, pagers, smart phones, personal data assistants (PDAs), Internet of Things (IoT) devices such as televisions, refrigerators, light bulbs, thermostats, home security systems, baby monitors, or the like. Each of customer devices <NUM> may run a variety of software applications, such as word processing and other office support software, web browsing software, software to support voice calls, video games, videoconferencing, and email, among others. Customer devices <NUM> connect to customer network <NUM> via wired and/or wireless communication links. The term "communication link," as used herein, comprises any form of transport medium, wired or wireless, and can include intermediate nodes such as network devices, such as network device <NUM>.

In accordance with the techniques of the disclosure, a secure, remote bootstrapping operation of a network device is disclosed such that sensitive information is stored only on a volatile memory of network device <NUM>. To implement the techniques of the disclosure, network device <NUM> perform a touchless provisioning operation to retrieve onboarding information during each boot cycle of network device <NUM>. Network device <NUM> processes the onboarding information to determine whether a first initialization of network device <NUM> has occurred. If the first initialization has not yet occurred, network device <NUM> performs the first initialization by configuring itself to mount at least a portion of a file system to a volatile memory of the network device and not a non-volatile memory of the network device and reboots itself.

For each subsequent boot cycle, network device <NUM> mounts at least a portion of the file system to the volatile memory and/or encrypts at least a portion of the non-volatile memory, such as, e.g., a swap portion of the non-volatile memory. Further, network device <NUM> performs another touchless provisioning operation to request the onboarding information. Network device <NUM> processes the onboarding information to determine whether the first initialization has occurred. In response to determining that the first initialization has occurred, network device <NUM> performs a bootstrapping operation. As examples of bootstrapping operations, the network device may configure itself for remote management by another network device, e.g., by obtaining an IP address for a remote network device or security credentials for authenticating orchestrator server <NUM> or by establishing a local administrator account with which orchestrator server <NUM> can log into network device <NUM>, etc.).

An example operation in accordance with the techniques of the disclosure is set forth in the algorithm below:
# check if running first time
if [running first time]; then
do first-time initialization:
- set non-volatile configuration (e.g., disable console port or set root password). - download and install a package that includes
rc scripts to initialize a memory-based
Master File System (MFS) to a volatile
memory with encrypted swap, mount filesystems,
and initialize filesystems, causing
the MFS to mount to the volatile memory on boot. - reboot the network device (this restarts
the network device and also the secure touchless
provisioning process)
else
do normal configuration operations, per user's discretion
- actions performed here, depending on what they are, may or may not survive a power
loss, depending on whether the actions
affect files in filesystems moved to the MFS in
the volatile memory.

In one example of the techniques of the disclosure, upon startup, network device <NUM> performs first touchless provisioning operation <NUM> to request, from DHCP server <NUM>, address information for bootstrap server <NUM>. Typically, network device <NUM> makes this request using an unsecured protocol and blindly trusts the response. In some examples, the address information is a list of one or more bootstrap servers <NUM> from which network device <NUM> may obtain configuration information. In some examples, the list is a tuple data structure that specifies a hostname and a port for each bootstrap server <NUM>. The address information is redirect information which redirects a request for configuration information from network device <NUM> to bootstrap server <NUM>.

Upon receiving the address information for bootstrap server <NUM>, network device <NUM> performs second touchless provisioning operation <NUM> to request, from bootstrap server <NUM>, onboarding information for network device <NUM>. In some examples, network device <NUM> makes the request using an unsecured connection. Network device <NUM> may blindly trust a TLS certificate of bootstrap server <NUM>. Bootstrap server <NUM> may authenticate network device <NUM> via a TLS-level client certificate such as IDevID. Network device <NUM> receives the onboarding information from bootstrap server <NUM>. Typically, the configuration information received from bootstrap server <NUM> is signed.

Network device <NUM> verifies the signature of bootstrap server <NUM> in the configuration information. After verifying the signature, network device <NUM> processes the onboarding information to determine whether a first initialization of the network device has occurred. If the first initialization has not yet occurred, network device <NUM> performs the first initialization. For example, network device <NUM> may perform the first initialization by configuring local user access permissions to limit local user access and configuring itself to mount at least a portion of a file system to a volatile memory of network device <NUM> and not a non-volatile memory of network device <NUM>. Further, network device <NUM> reboots itself.

The onboarding information may include instructions to perform the configuring of the local user access permissions. For example, network device <NUM> may configure, based on the onboarding information, the local user access permissions by performing a secure hardening of network device <NUM> so as to limit local user access to network device <NUM>. For example, network device <NUM> may disable one or more console ports and/or open management ports of network device <NUM> or set a root access password to network device <NUM>. Typically, network device <NUM> stores the local user access permissions in a non-volatile memory of network device <NUM> such that the local user access permissions are not lost upon power loss to network device <NUM>.

In some examples, the onboarding information specifies a particular boot image that network device <NUM> is to use, an initial configuration that network device <NUM> should use, and one or more scripts for execution by network device <NUM>. In some examples, the onboarding information specifies a particular operating system type and version. In some examples, network device <NUM> uses the onboarding information to configure one or more remote management protocols, such as NETCONF over SSH, and to configure whether network device <NUM> initiates an outbound SSH connection, or opens a port enabling inbound SSH connections. In some examples, network device <NUM> uses the onboarding information to configure whether orchestrator server <NUM> or another user may access network device <NUM> via a root or other login. In some examples, network device <NUM> uses the onboarding information to configure how SSH authentication may be performed (e.g., via password, public-key encryption, RADIUS, tacplus, etc.).

For each subsequent boot cycle, network device <NUM> mounts at least a portion of the file system to the volatile memory and/or encrypts at least a portion of the non-volatile memory. Further, network device <NUM> performs third touchless provisioning operation <NUM> to request the onboarding information. Network device <NUM> processes the onboarding information to determine whether the first initialization of network device <NUM> has occurred. In response to determining that the first initialization has occurred, network device <NUM> performs a bootstrapping operation. As examples of bootstrapping operations, network device <NUM> may configure itself for remote management by another network device (e.g., orchestrator server <NUM>). Further, network device <NUM> may perform volatile configuration of network device <NUM>. As examples of volatile configuration, network device <NUM> may disable the console on system ports, enable access of system services via SSH, disable the use of passwords to network device <NUM> via SSH, allow root login via SSH, configure a root account for SSH public-key authentication, and configure network device <NUM> to initiate an outbound SSH session. These configurations may be lost upon loss of power to or reboot of network device <NUM>. After completing the bootstrapping operation, network device <NUM> may establish secure management connections (e.g., NETCONF) to orchestrator server <NUM>. Further, network device <NUM> begins normal operation. For example, in the case where network device <NUM> is a router, network device <NUM> may begin the processing and routing of network traffic <NUM>.

Typically all control plane and management traffic is encrypted. While secure zero touch provisioning (e.g., secure ZTP) and NETCONF over SSH are built on top of secure transport layers, not all protocols are such (e.g., syslog). Thus, a VPN may be configured between network device <NUM> and bootstrap server <NUM> as an aspect of the bootstrapping process. Different technologies may be used to implement these VPNs. For instance, they may be implemented in software or hardware. In one example, an MS-MPC line card may be used to perform hardware encryption of all control plane and management plane traffic. In some examples, the secure touchless provisioning operation described herein may be used to configure the MS-MPC line cards. In other examples, a user configures the MS-MPC line cards as an interactive step that occurs over a management connection (e.g., NETCONF over SSH), either through access via network device <NUM> or orchestrator server <NUM>.

In some examples, the bootstrapping process may be further split into two operations: the first being an operation where network device <NUM> is configured with onboarding information and the second being an operation where orchestrator server <NUM> or another network management system configures network device <NUM> with additional software applications after orchestrator server <NUM> establishes a connection to network device <NUM> for the first time. In some examples, each touchless provisioning operation concludes with network device <NUM> sending a "bootstrap-complete" progress report to bootstrap server <NUM>. Bootstrap server <NUM> may propagate this report to orchestrator server <NUM>, thus providing a signal for when orchestrator server <NUM> may perform first-time operations on network device <NUM>.

In some examples, orchestrator server <NUM> may obtain ownership vouchers <NUM> from voucher server <NUM> of service provider network <NUM>. For example, voucher server <NUM> may provide a REST-based API that authenticates operator credentials. Voucher server <NUM> verifies that a device within customer network <NUM>, such as network device <NUM>, is owned by an operator of customer network <NUM>. Voucher server <NUM> encodes an owner certificate of orchestrator server <NUM> into the ownership voucher; network device <NUM> may use the owner certificate to verify onboarding information signed by the owner during a touchless provisioning operation. Network device <NUM> may, for example, use a preconfigured trust anchor for verifying ownership vouchers were generated by a trusted authority. Network device <NUM> may also, for example, examine the ownership voucher to ensure that it contains the network device's serial number, and therefore knows that the ownership voucher applies to the network device <NUM>. Voucher server <NUM> may issue a voucher signed by a signing authority trusted by a manufacturer of network device <NUM> (or the manufacturer itself) to each authorized device accessing customer network <NUM>.

As one example, orchestrator <NUM> generates a PKI for owner certificates. Orchestrator <NUM> requests, from voucher server <NUM> to provide vouchers, containing a supplied domain certificate, for a list of devices (identified by their serial numbers) authorized to access customer network <NUM>. Voucher server <NUM> provides a list of ownership vouchers for the devices authorized to access customer network <NUM>. Orchestrator <NUM> stores the device-specific ownership vouchers and owner certificates for subsequent use. Orchestrator <NUM> may use the stored device-specific ownership vouchers and owner certificates to sign bootstrapping data for network device <NUM> during touchless provisioning operations to configure network device as described above.

Thus, by using the techniques of the disclosure, the network device may maintain configuration information obtained during the bootstrapping process within a volatile memory of the network device. Upon loss of power to the network device, the network device loses any information that may be of use to a malicious actor that has physical access to the network device, such as user data, configuration information, or boot images. Each boot cycle, network device <NUM> may retrieve a new onboarding information upon startup using a touchless provisioning operation. Further, after the first initialization, local access to network device <NUM>, such as via console ports, is disabled.

Accordingly, the techniques of the disclosure provide specific improvements to the computer-related field of network device deployment and configuration. For example, the techniques of the disclosure may allow for the deployment to public spaces of network devices that are secured from physical or local attack by a malicious actor. Further, the techniques of the disclosure may allow for the use of touchless provisioning to securely and remotely configure a network device. Additionally, the techniques of the disclosure may reduce the cost and technical burden on the end user of deploying and configuring network devices to operate within an enterprise network. Further, the techniques of the disclosure may reduce the complexity of deployment, maintenance, and upgrading of network devices.

<FIG> is a block diagram illustrating an example network device <NUM> in accordance with the techniques of the disclosure. In some examples. network device <NUM> is an example of network device <NUM> of <FIG>. For example, network device <NUM> may correspond to a router, a bridge, a hub, a switch, a server, a printer, a gateway, a firewall, an IDP device, or other network device. In the example of <FIG>, network device <NUM> includes user interface module <NUM>, control unit <NUM>, and network interface <NUM>.

User interface <NUM> is configured to send/receive data to/from a user, such as a network administrator. Typically, user interface <NUM> comprises a console port that enables local access by a user, such as an administrator. However, in some examples, user interface <NUM> is or otherwise includes a workstation, a keyboard, pointing device, voice responsive system, video camera, biometric detection/response system, button, sensor, mobile device, control pad, microphone, presence-sensitive screen, network, or any other type of device for detecting input from a human or machine. In some examples, user interface <NUM> further includes a display for displaying an output to the user. The display may function as an output device using technologies including liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating tactile, audio, and/or visual output. In other examples, user interface <NUM> may produce an output to a user in another fashion, such as via a sound card, video graphics adapter card, speaker, presence-sensitive screen, one or more USB interfaces, video and/or audio output interfaces, or any other type of device capable of generating tactile, audio, video, or other output. In some examples, user interface <NUM> may include a presence-sensitive display that may serve as a user interface device that operates both as one or more input devices and one or more output devices.

Control unit <NUM> comprises hardware for performing the techniques of this disclosure. Processors <NUM> may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry. Network device further includes non-volatile memory <NUM> and volatile memory <NUM>. As described herein, "non-volatile memory" refers to a storage device which retains data even if power to the non-volatile memory is lost. In contrast, as described herein, "volatile memory" refers to a storage device which loses data if power to the volatile memory is lost. Alternatively, control unit <NUM> may comprise dedicated hardware, such as one or more integrated circuits, one or more Application Specific Integrated Circuits (ASICs), one or more Application Specific Special Processors (ASSPs), one or more Field Programmable Gate Arrays (FPGAs), or any combination of the foregoing examples of dedicated hardware, for performing the techniques described herein.

Non-volatile memory <NUM> may include a disk drive, an optical drive, memory, such as random-access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, etc., comprising executable instructions for causing processors <NUM> to perform the actions attributed to them.

Volatile memory <NUM> may include memory, such as random-access memory (RAM) or flash memory, etc., comprising executable instructions for causing processors <NUM> to perform the actions attributed to them.

In the example of <FIG>, control unit <NUM> comprises device modules <NUM> and protocols <NUM>, which may comprise software modules executed by control unit <NUM> or discrete, independent hardware units of control unit <NUM>. When any or all of device modules <NUM> and protocols <NUM> comprise software, instructions executable by a processor for device modules <NUM> and protocols <NUM> may be encoded in a computer-readable medium of network device <NUM>, such as non-volatile memory <NUM> or volatile memory <NUM>.

Network interface <NUM> may comprise any interface for connecting to devices of a computer network, such as DHCP server <NUM>, bootstrap server <NUM>, or orchestrator server <NUM> of <FIG>. For example, network interface <NUM> may comprise an Ethernet interface, a gigabit Ethernet interface, a telephone modem, a cable modem, a satellite modem, or other network interface. In some examples, network interface <NUM> comprises one or more network interface cards.

Device modules <NUM> generally correspond to components specific to network device <NUM>. For example, when network device <NUM> comprises a router, device modules <NUM> may comprise a control plane that maintains a routing information base, a forwarding engine that maintains a forwarding information base, one or more routing protocols, or other modules required to route packets through a network. As another example, when network device <NUM> comprises a security device, device modules <NUM> may comprise a protocol decoder module, an application identification module, and an attack detection module, or other network security modules. Protocols <NUM> comprise one or more communication protocols for communicating with management device <NUM> and/or other network devices. For example, protocols <NUM> may comprise a touchless provisioning protocol such as ZTP <NUM>. Protocols <NUM> may also comprise one or more routing protocols, security protocols, or other protocols, depending upon the type of device to which network device <NUM> corresponds.

In accordance with the techniques of the disclosure, network device <NUM> performs a secure, remote bootstrapping operation. Upon startup, processors <NUM> perform a first touchless provisioning operation to request, from DHCP server <NUM>, address information for bootstrap server <NUM>. Upon receiving the address information for bootstrap server <NUM>, processors <NUM> perform a second touchless provisioning operation to request, from bootstrap server <NUM>, onboarding information for network device <NUM>.

Processors <NUM> process the onboarding information to determine whether a first initialization of network device <NUM> has occurred. If the first initialization has not yet occurred, processors <NUM> perform the first initialization. For example, processors <NUM> may perform the first initialization by configuring local user access permissions <NUM> of network device <NUM> and configuring network device <NUM> to mount at least a portion of file system <NUM> to volatile memory <NUM> and not non-volatile memory <NUM>. Further, processors <NUM> cause network device <NUM> to reboot.

To configure local user access permissions <NUM>, processors <NUM> may perform a secure hardening of network device <NUM> by disabling one or more console ports and/or open management ports of network device <NUM> or by setting a root access password to network device <NUM>. Typically, processors <NUM> store local user access permissions <NUM> in non-volatile memory <NUM> such that local user access permissions <NUM> are not lost upon power loss to network device <NUM>.

For each subsequent boot cycle, processors <NUM> perform an operating-system level Processors <NUM> mount at least a portion of file system <NUM> to volatile memory <NUM>. As another example, processors <NUM> encrypt at least a portion of non-volatile memory <NUM>, such as swap portion <NUM>. Further, processors <NUM> perform a touchless provisioning-level initialization. As an example, processors <NUM> perform a touchless provisioning operation to retrieve onboarding information from bootstrap server <NUM>. Processors <NUM> store the onboarding information in volatile memory <NUM> such that the onboarding information is not retained upon loss of power to or reboot of network device <NUM>. Processors <NUM> process the onboarding information to determine whether the first initialization of network device <NUM> has occurred. In response to determining that the first initialization has occurred, processors <NUM> perform a bootstrapping operation. As examples of bootstrapping operations, processors <NUM> may configure network device <NUM> for remote management by another network device (e.g., orchestrator server <NUM>). For example, processors <NUM> may assign a hostname, IP address, or port for communication with orchestrator server <NUM>, processors <NUM> may define a trust anchor certificate to authenticate orchestrator server <NUM>, and processors <NUM> may establish an admin account with which orchestrator server <NUM> may access network device <NUM>. As further examples, processors <NUM> may disable the console on system ports, enable access of system services via SSH, disable the use of passwords to network device <NUM> via SSH, allow root login via SSH, configure a root account for SSH public-key authentication, and configure network device <NUM> to initiate an outbound SSH session (e.g., with orchestrator <NUM>). Typically, processors <NUM> store any configuration that is performed at this stage (e.g., after the first initialization has occurred) in volatile memory <NUM> such that the configuration is not retained upon loss of power to or reboot of network device <NUM>. After completing the bootstrapping operation, processors <NUM> may establish secure management connections (e.g., NETCONF) to orchestrator server <NUM> and commence operation of network device <NUM>. For example, in the case where network device <NUM> is a router, processors <NUM> may begin the processing and routing of network traffic.

Thus, by using the techniques of the disclosure, network device <NUM> maintains at least a portion of filesystem <NUM> within volatile memory <NUM>. Upon loss of power to network device <NUM>, volatile memory <NUM> loses the portion of filesystem <NUM>. Further, network device <NUM> may lose a cryptographic cypher for accessing encrypted swap <NUM>. Thus, network device <NUM> does not retain any information in filesystem <NUM> or swap <NUM> that may be of use to a malicious actor that has physical access to network device <NUM>. Each boot cycle, network device <NUM> performs a new touchless provisioning operation to perform bootstrapping of network device <NUM> and so remount the portion of filesystem <NUM> to volatile memory <NUM>. Further, after the first initialization, local access to network device <NUM>, such as via console ports of user interface <NUM>, is disabled.

<FIG> is a block diagram illustrating example server <NUM> in accordance with the techniques of the disclosure. Server <NUM> is a computing device that may implement, for example, one or more of DHCP server <NUM>, bootstrap server <NUM>, orchestrator server <NUM>, or voucher server <NUM> of <FIG>. In the example of <FIG>, server <NUM> includes user interface <NUM>, control unit <NUM>, and network interface <NUM>.

User interface <NUM> is configured to send/receive data to/from a user, such as a network administrator. In some examples, user interface <NUM> is or otherwise includes a workstation, a keyboard, pointing device, voice responsive system, video camera, biometric detection/response system, button, sensor, mobile device, control pad, microphone, presence-sensitive screen, network, or any other type of device for detecting input from a human or machine. In some examples, user interface <NUM> further includes a display for displaying an output to the user. The display may function as an output device using technologies including liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating tactile, audio, and/or visual output. In other examples, user interface <NUM> may produce an output to a user in another fashion, such as via a sound card, video graphics adapter card, speaker, presence-sensitive screen, one or more USB interfaces, video and/or audio output interfaces, or any other type of device capable of generating tactile, audio, video, or other output. In some examples, user interface <NUM> may include a presence-sensitive display that may serve as a user interface device that operates both as one or more input devices and one or more output devices.

Network interface <NUM> may comprise any interface for connecting to devices of a computer network, such as network device <NUM> of customer network <NUM>. For example, network interface <NUM> may comprise an Ethernet interface, a gigabit Ethernet interface, a telephone modem, a cable modem, a satellite modem, or other network interface. In some examples, network interface <NUM> comprises one or more network interface cards.

Control unit <NUM> comprises hardware for performing the techniques of this disclosure. Processors <NUM> may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry. Storage device <NUM> may include a disk drive, an optical drive, memory, such as random-access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, comprising executable instructions for causing the processors <NUM> to perform the actions attributed to them. Alternatively, control unit <NUM> may comprise dedicated hardware, such as one or more integrated circuits, one or more Application Specific Integrated Circuits (ASICs), one or more Application Specific Special Processors (ASSPs), one or more Field Programmable Gate Arrays (FPGAs), or any combination of the foregoing examples of dedicated hardware, for performing the techniques described herein.

In the example of <FIG>, control unit <NUM> comprises device manager <NUM>, device modules <NUM>, and protocols <NUM>, which may comprise software modules executed by control unit <NUM> or discrete, independent hardware units of control unit <NUM>. When any or all of device manager <NUM>, device modules <NUM>, and protocols <NUM> comprise software, e.g., executable software modules, instructions executable by a processor for device manager <NUM>, device modules <NUM>, and protocols <NUM> may be encoded in a computer-readable medium, such as storage device <NUM>.

Device manager <NUM> interacts with one or more managed devices, e.g., network device <NUM>, to manage the network devices. In one example, device manager <NUM> executes an implementation of NETCONF. Device manager <NUM> sends electrical signals to managed network device <NUM> via network interface <NUM>. Therefore, device manager <NUM> sends and receives packets comprising data for managing the managed network devices <NUM> indirectly via a network, such as the Internet, to, e.g., network devices <NUM>.

Device modules <NUM> generally correspond to components specific to server <NUM>. For example, device modules <NUM> may comprise a control plane that maintains a routing information base, a forwarding engine that maintains a forwarding information base, one or more routing protocols, or other modules required to route packets through a network.

In the example where server <NUM> is an example of DHCP server <NUM> of <FIG>, device modules <NUM> include a DHCP module for providing DHCP services to devices within customer network <NUM>, such as network device <NUM>, bootstrap server <NUM>, and customer devices <NUM>. In some examples, server <NUM> may provide redirect information in response to a touchless provisioning operation by network device <NUM>. For example, in response to a first touchless provision operation in which network device <NUM> requests a network assignment and requests redirect information for onboarding information, server <NUM> provides address information for reaching bootstrap server <NUM>. Typically, server <NUM> provides this information using an unsecured protocol. In some examples, the address information is a list of one or more bootstrap servers <NUM> from which network device <NUM> may obtain configuration information. In some examples, the list is a tuple data structure that specifies a hostname and a port for bootstrap server <NUM>. In some examples, the address information is redirect information which redirects a request for configuration information from network device <NUM> to bootstrap server <NUM>.

In the example where server <NUM> is an example of bootstrap server <NUM> of <FIG>, server <NUM> may be used as a source of onboarding information for network device <NUM>. As described herein, network device <NUM> may request onboarding information from bootstrap server <NUM> via a touchless provisioning operation to perform automatic configuration of network device <NUM>. In some examples, the touchless provisioning operation is a ZTP operation. For example, in response to a second touchless provisioning request from network device <NUM>, server <NUM> provides onboarding information for network device <NUM>. In some examples, the onboarding information specifies a particular boot image that network device <NUM> is to use, an initial configuration that network device <NUM> should use, and one or more scripts for execution by network device <NUM>. In some examples, the onboarding information specifies a particular operating system type and version. In some examples, network device <NUM> uses the onboarding information to configure one or more remote management protocols, such as NETCONF over SSH, and to configure whether network device <NUM> initiates an outbound SSH connection, or opens a port enabling inbound SSH connections. In some examples, network device <NUM> uses the onboarding information to configure whether orchestrator server <NUM> or another user may access network device <NUM> via a root or other login. In some examples, network device <NUM> uses the onboarding information to configure how SSH authentication may be performed (e.g., via password, public-key encryption, RADIUS, tacplus, etc.).

In the example where server <NUM> is an example of orchestrator server <NUM> of <FIG>, server <NUM> may obtain ownership vouchers from voucher server <NUM> of service provider network <NUM>, provide DHCP configuration to DHCP server <NUM> to enable DHCP server <NUM> to provide DHCP services to devices within customer network <NUM>, or provide bootstrap configuration information to bootstrap server <NUM> to enable bootstrap server <NUM> to provide bootstrap services to network device <NUM>. Additionally, once network device <NUM> is deployed and activated, server <NUM> may establish secure management connections to manage network device <NUM> using a communications protocol, such as NETCONF. Further, orchestrator server <NUM> may forward network traffic to network device <NUM> for processing and routing.

In the example where server <NUM> is an example of voucher server <NUM> of <FIG>, server <NUM> may provide a REST-based API that authenticates operator credentials. For example, server <NUM> verifies that a device within customer network <NUM>, such as network device <NUM>, is owned by an operator of customer network <NUM>. Server <NUM> encodes an owner certificate of orchestrator server <NUM> into the ownership voucher; network device <NUM> may use the owner certificate to verify onboarding information signed by the owner during a touchless provisioning operation. Server <NUM> may issue a voucher signed by a signing authority trusted by a manufacturer of network device <NUM> (or the manufacturer itself) to each authorized device accessing customer network <NUM>.

Protocols <NUM> comprise one or more network communication protocols for communicating over a network. For example, protocols <NUM> may comprise one or more routing protocols, security protocols, or other protocols, e.g., a touchless provisioning protocol such as ZTP <NUM>, DHCP <NUM>, or other network protocols for communicating over a network not expressly depicted in <FIG>, such as SSH, PPP, PPPoE, PPPoA, MPLS, BGP, SNMP, NETCONF, etc..

<FIG> is a flowchart illustrating an example secure remote bootstrapping operation in accordance with the techniques of the disclosure. For convenience, <FIG> is described with respect to <FIG>.

In one example, upon startup, network device <NUM> performs a first touchless provisioning operation to request, from DHCP server <NUM>, address information for bootstrap server <NUM> (<NUM>). In response, DHCP server <NUM> provides the address information as a list of one or more bootstrap servers <NUM> from which network device <NUM> may obtain configuration information (<NUM>). In some examples, the list is a tuple data structure that specifies a hostname and a port for bootstrap server <NUM>. In some examples, the address information is redirect information which redirects a request for configuration information from network device <NUM> to bootstrap server <NUM>.

Upon receiving the address information for bootstrap server <NUM>, network device <NUM> performs second touchless provisioning operation <NUM> to request, from bootstrap server <NUM>, onboarding information for network device <NUM> (<NUM>). Bootstrap server <NUM> provides the boot configuration information to network device <NUM> (<NUM>). Network device <NUM> receives the onboarding information from bootstrap server <NUM>. Typically, the configuration information received from bootstrap server <NUM> is signed.

Network device <NUM> processes the onboarding information to determine whether a first initialization of the network device has occurred. In response to determining that the first initialization has not yet occurred, network device <NUM> performs the first initialization (<NUM>). For example, network device <NUM> may perform the first initialization by configuring local user access permissions and configuring itself to mount at least a portion of a file system to a volatile memory of network device <NUM> and not a non-volatile memory of network device <NUM>. For example, network device <NUM> may perform a secure hardening of network device <NUM> by disabling one or more console ports and/or open management ports of network device <NUM> or by setting a root access password to network device <NUM>. Typically, network device <NUM> stores the local user access permissions in a non-volatile memory of network device <NUM> such that the local user access permissions are not lost upon power loss to network device <NUM>. Further, network device <NUM> reboots itself.

For each subsequent boot cycle, network device <NUM> mounts at least a portion of the file system to the volatile memory. In some examples, network device <NUM> encrypt at least a portion of the non-volatile memory, such as a swap portion of the file system. Further, network device <NUM> performs a third touchless provisioning operation to request the onboarding information (<NUM>). Network device <NUM> processes the onboarding information to determine whether the first initialization of network device <NUM> has occurred. In response to determining that the first initialization has occurred, network device <NUM> performs a bootstrapping operation.

As examples of bootstrapping operations, network device <NUM> configure itself for remote management by another network device (e.g., orchestrator server <NUM>). Further, network device <NUM> may perform volatile configuration of network device <NUM>. As examples of volatile configuration, network device <NUM> may disable the console on system ports, enable access of system services via SSH, disable the use of passwords to network device <NUM> via SSH, allow root login via SSH, configure a root account for SSH public-key authentication, and configure network device <NUM> to initiate an outbound SSH session.

After completing the bootstrapping operation, orchestrator server <NUM> and network device <NUM> may establish secure management connections (e.g., NETCONF) with one another (<NUM>). Further, network device <NUM> begins normal operation (<NUM>). For example, in the case where network device <NUM> is a router, network device <NUM> commences the processing and routing of network traffic <NUM>.

The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media.

Claim 1:
A method comprising:
performing (<NUM>), by one or more processors of a network device, a first request for onboarding information for the network device;
processing, by the one or more processors, the onboarding information to determine that a first initialization of the network device has not occurred;
in response to determining that the first initialization of the network device has not occurred, performing (<NUM>) the first initialization by:
configuring, by the one or more processors and with the onboarding information, the network device to mount at least a portion of a file system to a volatile memory of the network device and not a non-volatile memory of the network device, such that the at least a portion of the file system will not be retained in the volatile memory upon rebooting of the network device; and
rebooting, by the one or more processors, the network device;
after rebooting the network device:
performing (<NUM>), by the one or more processors, a second request for the onboarding information for the network device;
processing, by the one or more processors, the onboarding information to determine that the first initialization of the network device has occurred; and
in response to determining that the first initialization of the network device has occurred, performing (<NUM>), by the one or more processors, a bootstrapping operation of the network device.