Patent Publication Number: US-2023132853-A1

Title: Extending supervisory services into trusted cloud operator domains

Description:
BACKGROUND 
     A cloud architecture may include a computer system domain of multiple physical server nodes (e.g., blade servers or rack mounted servers), or “domain nodes.” The domain nodes define a computer system, physical infrastructure domain. The physical infrastructure domain may be managed by a central management system, which orchestrates a software-defined logical infrastructure and services (e.g. software-defined compute (SDC) services, software-defined storage (SDS) services and software-defined networking (SDN) services), which are hosted on the domain nodes. The software-defined logical infrastructure relies on an abstraction infrastructure layer (e.g., a hypervisor for virtual machine (VM) abstractions or an OS for container abstractions) to provide the central management system control of the services. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic diagram of a cloud computer system according to an example implementation. 
         FIG.  2    depicts an example power-up sequence for a smart input/output (I/O) peripheral of a domain node of the cloud computer system of  FIG.  1    according to an example implementation. 
         FIGS.  3 A,  3 B,  3 C and  3 D  are illustrations of example architectures associated with host supervisory services of the domain node according to example implementations. 
         FIG.  4    is a flow diagram depicting a bus enumeration process used by a host supervisory service according to an example implementation. 
         FIG.  5    is a flow diagram depicting a process used by a host supervisory service responsive to a runtime container instance change event according to an example implementation. 
         FIG.  6    is a flow diagram depicting a process used by a smart I/O peripheral to manage ready state indicators according to an example implementation. 
         FIG.  7    is a flow diagram depicting a process to extend a supervisory host service into a trusted cloud operator domain according to an example implementation. 
         FIG.  8    is a schematic diagram of a system that includes a host to perform a supervisory service for a smart I/O peripheral according to an example implementation. 
         FIG.  9    is an illustration of a non-transitory storage medium that stores machine-readable instructions that, when executed by a machine, cause the machine to manage a ready state indicator of a smart I/O peripheral according to an example implementation. 
     
    
    
     DETAILED DESCRIPTION 
     A computer platform, such as a server (e.g., a blade server or a rack-mounted server), may have an architecture (e.g., a traditional personal computer (PC)-based architecture, or “legacy architecture”) that may be characterized as employing a “host-centric” control model. In the host-centric control model, the server controls are rooted in the host of the server. As used herein, a “host” refers to the software and hardware of the computer platform, which provides one or multiple application operating environments for the computer platform. Stated differently, the server controls are rooted in the main central processing units (CPUs) of the server. 
     The server may be a compute node of a cloud-based computing system, or “cloud computer system.” The software layer of the computer node&#39;s host may provide the logical infrastructure abstraction and control surfaces via a hypervisor software stack (for virtual machines (VMs)) and/or an operating system software stack (for containers). The result of this host-centric control model is a separation of the administrative control of the physical infrastructure of the compute node from the logical infrastructure of the compute node. 
     A central management system of the cloud computer system may be built around a hierarchical management structure. At the lowest level of the hierarchical management structure the compute node may be divided into two distinct control surfaces: a first control surface for the physical infrastructure of the compute node and a second control surface for the logical infrastructure of the compute node. Each of these infrastructures may be connected to its own central manager so that there may be two distinctly different associated administrators: an administrator for the physical infrastructure and a virtualization administrator for the logical infrastructure. Correspondingly, there may be two distinct central management software stacks, and there may be a cloud infrastructure manager at the top of the hierarchical management structure to unify the software stacks. The cloud infrastructure manager may replace the roles of the traditional infrastructure administrator with the following distinct cloud roles: the operator (for physical &amp; logical infrastructure control), the tenant (for abstracted instance control), and the infrastructure support technician (for support). 
     Cloud-native architectures may look similar to traditional cloud architectures, but cloud-native architectures differ in one significant aspect. Cloud-native architectures blend the physical and logical infrastructure by using intelligent input/output (I/O) subsystems, called “smart I/O peripherals” herein, for purposes of offloading services from the host (i.e., offloading traditional processing performed by the host) and by isolating node management controls within the smart I/O peripherals. The blending of physical and logical infrastructure changes the general-purpose nature of legacy architecture servers by utilizing an independent software stack, which may be managed in segments (e.g., segments aligned to SDC, SDS, and SDN layers). Accordingly, such a server is referred to herein as a segmented server (or “segmented compute node”). This segmented control model presents orchestration and control sequencing challenges for the legacy architecture server related to ensuring that all independent managers are in a “ready state” before backend I/O services provided by the smart I/O peripherals are rendered. This new orchestration and sequencing results in a more vertically-integrated (i.e., tightly-coupled or unified) architecture. While the cloud-native architecture uses a hierarchical control management domain structure, the result is a unified experience around three control roles: the operator, the tenant, and the support technician. 
     As used herein, a “smart I/O peripheral” refers to a device, or component, of the computer platform, which provides one or multiple functions for the host, which, in legacy architectures, have been controlled by the host. In general, a smart I/O peripheral is hardware processor that has been assigned (e.g., programmed with) a certain personality. The smart I/O peripheral may provide one or multiple backend I/O services (or “host offloaded services) in accordance with its personality. The backend I/O services may be non-transparent services or transparent services. An example of a non-transparent host service is a hypervisor virtual switch offloading service using PCIe direct I/O (e.g., CPU input-output memory management unit (IOMMU) mapping of PCIe device physical and/or virtual functions) with no host control. A host transparent backend I/O service does not involve modifying host software. As examples, transparent backend I/O services may include network-related services, such as encryption services, overlay network access services and firewall-based network protection services. In general, the smart I/O peripheral may provide any of a number of transparent and/or non-transparent backend network services for the host. As examples, network-related backend services may include overlay network services, virtual switching services, virtual routing services and network function virtualization services. As examples, storage-related backend services may include backend storage I/O services for the host, such as storage acceleration services (e.g., non-volatile memory express (NVMe)-based services), direct attached storage services, or a Serial Attached SCSI (SAS) storage service. 
     A smart I/O peripheral may coordinate with another network component for purposes of providing one or multiple backend I/O services. For example, the smart I/O peripheral may be a “smart NIC” that is connected to an Ethernet Top-of-the Rack (ToR) switch. This combination may provide host isolation by using the smart NIC in its “lock state” for Peripheral Component Interconnect express (PCIe) physical and virtual functions. Here, the “lock state” refers to restricting host access from full reconfiguring capability of the device based on the cloud operator&#39;s goals. For example, the lock state may prevent the host from reconfiguring certain infrastructure (e.g., turning off storage redundancy, turning off GPUs, and forth), which might affect, for example, the cloud operator&#39;s contractual obligation to the tenant. With this arrangement, the ToR switch may be used to provide such network services as a network protection service (e.g., a firewall service) or an instance overlay service (e.g., a virtual extensible local area network (VxLAN) service). 
     The smart I/O peripheral may take on one of many different physical forms and provide any of a wide variety of backend I/O services. A given smart I/O peripheral may provide network connectivity (e.g., an Ethernet port), provide multiple CPU cores, provide hardware acceleration engines, and provide a rich programming environment (i.e., an environment that enhances the integration of independent software components). Moreover, the smart I/O peripheral may provide endpoint connectivity (e.g., provide one or multiple PCIe ports) to an in-band communication channel (e.g., a PCIe link or bus) that is accessible by the host. As a specific example, the components of a PCIe card-based smart I/O peripheral may be mounted on a circuit card substrate that has a PCIe card edge connector that has a form factor to allow the smart I/O peripheral to be installed in a PCIe card slot connector of the server. In accordance with further implementations, a smart I/O peripheral may be constructed to be installed in a card slot connector other than a PCIe card slot connector, and in accordance with yet further example implementations, components of the smart I/O peripheral may be mounted (e.g., surface mounted) to a motherboard of the server. 
     A cloud-native, segmented compute node that has smart I/O peripherals results in a dual-management control system within the compute node, separating the control of the smart I/O device from the host CPU control by adding a hardware —abstracted interface to the host. The hardware-abstracted interface may be presented to the host as a standard local I/O peripheral (i.e., a non-smart I/O peripheral). 
     The host may access a smart I/O peripheral (e.g., access a PCIe-based smart I/O peripheral using PCIe direct I/O communications) to map physical functions and virtual functions (e.g., PCIe physical and virtual functions) into host-abstracted application operating environments. The backend I/O services that are provided by the smart I/O peripheral may be fully managed by the smart I/O peripheral independently from the host. Moreover, the lifecycles of these services may be controlled by the smart I/O peripheral independently from the host. 
     Therefore, a segmented compute node that has a cloud-native infrastructure architecture with hardware abstraction-isolated smart I/O peripherals differs from a non-segmented compute node that has a legacy architecture that is built around a host-centric control. This difference creates an architecture gap in current server designs, for both the pre-boot and runtime environments. The architecture gap presents challenges in orchestrating control, as the host is no longer the root of control. 
     One approach to address this dilemma and bridge the architecture gap is to modify a legacy server hardware architecture to allow control of the server to be rooted in a service processor of the server. For example, a slot connector (e.g., a PCIe connector) of the legacy server hardware architecture may be modified to support a sideband communication channel with a baseboard management controller, or “BMC.” In general, the BMC is a service processor that is constructed to manage the computer platform. With this approach, control is rooted to the BMC, and the traditional role of the BMC is expanded to manage the smart I/O peripheral by communicating management traffic over the sideband communication channel. Another approach to bridge the architecture gap may be to modify the legacy server hardware architecture to include an additional service processor, a platform controller, and root control in the platform controller so that the platform controller supervises both the BMC and the smart I/O peripheral. 
     The above-described approaches to bridge the architecture gap in a segmented server involve changes to the architectures of next generation servers. Modifying the legacy server hardware architecture to accommodate smart I/O peripherals in next generation servers, however, does not provide a backward compatibility path for existing servers. 
     In accordance with example implementations that are described herein, a trust domain (called the “cloud tenant domain” herein) of the host of a server is blended with a hardware isolated trust domain (called the “cloud operator domain” herein) of a smart I/O peripheral of the server. This blending ensures that services of the host, which may be offloaded to the smart I/O peripheral, are secured and controlled, both prior to the operating environment becoming operational and after the operating environment is operational. Stated differently, the host software stack, in accordance with example implementations, is blended with the offloaded services. This segue of the host-offloaded services with the host software stack may impact the abstracted application operating environments (e.g., bare metal OS environments, hypervisor-VM environments and OS-container applications), which are supported by the smart I/O peripheral. In accordance with example implementations, a server includes a host supervisory service (or “supervisory service”) for each application operating environment. 
     For a bare metal application operating environment (i.e., an environment in which software has full access to the host CPU and other resources of the physical platform, which are exposed to the software), the server provides a host supervisory service that links the backend I/O services that are provided by the smart I/O peripheral into a single security trust domain to the host space. Moreover, in accordance with example implementations, the bare metal operating system may provide hypervisor and container application operating environments; and the server may provide corresponding host supervisory services for these environments. The host supervisory services provide the ability to compensate for the architecture gap that is created by the opaqueness of the smart I/O peripheral&#39;s architecture. 
     Using host supervisory services to manage smart I/O peripherals may encounter challenges due to the smart I/O peripherals behaving differently from traditional peripherals. As an example, a particular host supervisory service may perform PCIe bus enumeration, and the smart I/O peripheral may be a PCIe card. According to the PCIe standard, a PCIe bus device should respond to a valid configuration request within a one second time limit after the PCIe bus device resets or powers on. If the PCIe bus device does not respond to a valid configuration request within the one second time limit, then traditionally, the PCIe bus device is deemed to have failed. 
     A smart I/O peripheral, however, may be relatively more complex than a traditional PCIe bus device that does not provide intelligent I/O services. Due to this complexity, a fully functional, PCIe-based smart I/O peripheral may be unable to respond within the one second time limit after the smart I/O peripheral is powered on or is reset. More specifically, unlike traditional PCIe bus devices, a smart I/O peripheral may have a relatively architecture, such as an architecture that includes a complex multi-core processing system and hardware accelerator engines (e.g., cryptography engines and packet inspection engines). The smart I/O peripheral, responsive to a boot, may further be controlled over the network by a domain manager for purposes of determining a physical infrastructure inventory and setting up logical domain connections, such as SDN, SDC and SDS connections. Due to this complexity, the smart I/O peripheral may take a relatively longer time (as compared to a PCIe non-intelligent I/O peripheral) to be ready to respond after the power on or reset of the smart I/O peripheral. 
     A smart I/O peripheral may also incur a significant, indeterminant delay when configuration changes to the smart I/O peripheral occur, such as changes in which the logical connections are changed or the smart I/O peripheral otherwise makes changes to its backend /O services. For example, the number of processing cores that are assigned to a VM instance may be scaled up; and due to this change, an overlay network connection may be added for a backend I/O service that is used by the VM instance. The addition of the overlay network connection may, for example, involve the smart I/O peripheral configuring or reconfiguring a virtual function that is provided by the smart I/O peripheral. Accordingly, the smart I/O peripheral may incur a relatively long delay before the virtual function is ready to be used. 
     Accordingly, due to its relative complexity, a smart I/O peripheral may not be able to respond in a manner that is expected for legacy I/O peripherals. 
     In accordance with example implementations, a smart I/O peripheral provides a ready state indicator, which may be accessed by a host supervisory service, for purposes of the host supervisory service determining whether or not the smart I/O peripheral is ready. In this context, the “readiness” of the smart I/O peripheral, in accordance with example implementations, generally represents whether the smart I/O peripheral is ready to proceed with the host. As examples, the ready state indicator may represent whether or not a configuration space of the smart I/O peripheral is ready to be configured by a host supervisory service. As another example, the ready state indicator may represent whether or not a particular function of the smart I/O peripheral is available. 
     As a more specific example, in response to a power on or reset of the server, as part of bus enumeration, a host supervisory service may detect the presence of peripherals (e.g., PCIe bus peripherals) and for each detected peripheral, set up a configuration space of the peripheral with the addresses of allocated memory space and/or I/O space. After detecting the presence of the smart I/O peripheral, the host supervisory service may, for example, access a ready state indicator of the smart I/O peripheral for purposes of determining whether or not the smart I/O peripheral is in a state that allows the host supervisory service to proceed with setting up the smart I/O peripheral&#39;s configuration space. The host supervisory service may therefore hold setting up of the configuration until the smart I/O peripheral sets the ready state indicator to a represent a ready state, which allows the host supervisory service to proceed. 
     As another example, a smart I/O peripheral may provide single root-input output virtualization (SR-IOV), which provides sets of virtual functions for corresponding physical functions of the smart I/O peripheral. A hypervisor-based host supervisory service may, for example, responsive to a VM instance activation, place a hold on the VM instance&#39;s use of an I/O service that is to be provided by the smart I/O peripheral. As described further herein, the hold on the I/O service may be in place until the smart I/O peripheral sets one or multiple ready state indicators associated with virtual and/or physical functions associated with the I/O service to represent that the function(s) are ready to be used. 
     Therefore, in accordance with example implementations, the ready state indicators allow host supervisory services to be extended into the cloud operator domain to manage smart I/O peripherals without modifying legacy server architectures and without imposing rigid time constraints that do not take into account the complex natures of the smart I/O peripherals. In accordance with example implementations, extending host supervisory services into the cloud operator domain may involve the vertical integration of host software (e.g., host software that is executed to perform the host supervisory services, smart I/O peripheral hardware, and smart I/O peripheral software). For example, with this vertical integration, the hardware of the smart I/O peripheral may be constructed to manage a register space that stores bits that represent corresponding ready state indicators. A software services stack of the smart I/O peripheral may be constructed to manage the ready state indicators. Extending host supervisory services into the cloud operator domain may include modifying hardware of the smart I/O peripheral (e.g., modifying a register space to store bits representing corresponding ready state indicators). Moreover, extending host supervisory services into the cloud operator domain may include modifying software of the smart I/O peripheral (e.g., modifying a software services stack of the smart I/O peripheral) to manage the ready state indicators. 
     Referring to  FIG.  1   , as a more specific example, in accordance with some implementations, a cloud-based computer system  100  that is affiliated with a particular cloud operator may provide multi-tenant cloud services for multiple clients, or tenants. The cloud services may be any of a number of different cloud services, such as Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and so forth. Moreover, depending on the particular implementation, the cloud services may be affiliated with one of several different cloud infrastructures, such as a public cloud that is generally available to all potential users over a public network; a limited access private cloud that is provided over a private network (e.g., cloud services provided by an on-site data center); or a hybrid cloud that is a combination of public and private clouds. 
     The tenants may access domain nodes  110  (e.g., compute nodes) of the cloud computer system  100  via cloud clients (e.g., laptops, desktop computers, smartphones, tablet computers, wearable computers, and so forth). As depicted in  FIG.  1   , the domain nodes  110  may be interconnected by physical network fabric  184 . In general, the physical network fabric  184  may be associated with one or multiple types of communication networks, such as (as examples) Fibre Channel networks, Gen-Z fabrics, dedicated management networks, local area networks (LANs), wide area networks (WANs), global networks (e.g., the Internet), wireless networks, or any combination thereof. As depicted in  FIG.  1   , the domain nodes  110  may be interconnected by logical connections  194 , such as SDN, SDC and SDS connections. 
     As also depicted in  FIG.  1   , a domain node manager  190  (or “node manager  190 ”) may be coupled to the physical network fabric  184  and the logical connections  194 . Depending on the particular implementation, the node manager  190  may be located on the domain node  110 - 1 , located on another domain node  110 , located on a dedicated administrative node, distributed across multiple nodes of the cloud computer system  100  and so forth. In general, the node manager  190  contains physical hardware resources and logical software resources for managing the domain nodes  110 . In accordance with some implementations, the node manager  190  may be associated with a database that stores data pertaining to the domain nodes  110 , such as data representing domain node configurations, telemetry logs, audit logs, and so forth. The node manager  190  may include an orchestration engine that manages the operation of leaf managers of the cloud computer system  100  for purposes of controlling the lifecycles and configuring the leaf resources (e.g., smart input/output (I/O) peripherals  180 , described herein) of the system  100 . In accordance with some implementations, the node manager  190  may communicate intent-based commands (e.g., Redfish commands) with one or multiple vendor root managers (not shown). The vendor root manager contains physical hardware resources and logical software resources to translate the intent-based commands to vendor-specific commands for the leaf resources that are associated with a particular vendor, and these vendor-specific commands may be processed by leaf resource managers (not shown) for the leaf resources. Depending on the particular implementation, the vendor root managers and leaf resource managers may be disposed on one or multiple administrative nodes, may be distributed on one or multiple domain nodes  110 - 1 , and so forth. 
     The domain node  110 - 1  may be a computer platform, in accordance with example implementations. In this context, a “computer platform” refers to a unit that includes a chassis and hardware that is mounted to the chassis, where the hardware is capable of executing machine-executable instructions (or “software”). A blade server is an example of a computer platform, in accordance with an example implementation. The computer platform may, however, be a platform other than a blade server, in accordance with further implementations, such as a rack-mounted server, a client, a desktop, a smartphone, a storage array, a laptop computer, a tablet computer, and so forth. 
     For example implementations in which the computer platform is a blade server, the server may have a frame, or chassis; one or multiple motherboards may be mounted to the chassis; and each motherboard may contain one or multiple multicore central processing unit (CPU) semiconductor packages (or “sockets” or “chips”). In accordance with example implementations, the blade server may have a form factor, mechanical latch(es) and corresponding electrical connectors for purposes of allowing the server blade to be installed in and removed from a corresponding server blade opening, or slot, in rack-mounted blade enclosure. 
     In accordance with example implementations, the domain node  110 - 1  may have a cloud-segmented architecture in which smart I/O peripherals  180  of the domain node  110 - 1  may control different logical connection segments independently from the host of the domain node  110 - 1 . As depicted in  FIG.  1   , the domain node  110 - 1  may provide one or multiple application operating environments  120  that are within a cloud tenant domain  111 . In general, tenant application instances (e.g., VM instances, container instances, non-virtualized application instances, and so forth) may execute in the application operating environments  120 . In general, the application operating environments  120  may represent virtualized environments as well as non-virtualized environments. 
     As an example, an application operating environment  120  may be an operating system (OS) bare metal environment (an “OS-bare metal application operating environment”) that includes application instances that have access to the unabstracted physical resources of the domain node  110 - 1 . As another example, an application operating environment  120  may be an OS-container bare metal application operating environment in which application instances may execute inside container instances. As another example, an application operating environment  120  may be an OS-bare metal/VM environment in which application instances may be execute inside VM instances or outside VM instances. As another example, an application operating environment  120  may be an OS-container bare metal/VM environment in which application instances may execute inside container instances, inside VMs, or outside of a VM or container instance. 
     In this context, a “virtual machine,” or “VM” (also called a “guest virtual machine,” a “virtual machine instance,” or “a guest virtual machine instance”) refers to a virtual environment that functions as a machine level abstraction, or virtual computer system, which has its own physical resources (e.g., CPU(s), system memory, network interface(s) and storage). Moreover, the VM may have its own abstraction of an operating system; and in general, the VM is a virtual abstraction of hardware and software resources of the domain node  110 - 1 . The lifecycle (e.g., the deployment and termination) of the VM may be managed by a virtual machine monitor (VMM), or hypervisor  167 , of the domain node  110 - 1 . 
     A “container” (also called an “instantiated container,” “container instance,” or “software container”), as used herein, generally refers to a virtual run-time environment for one or multiple applications and/or application modules, and this virtual run-time environment is constructed to interface to an operating system kernel. A container for a given application may, for example, contain the executable code for the application and its dependencies, such as system tools, libraries, configuration files, executables and binaries for the application. In accordance with example implementations, the container contains an operating system kernel mount interface but does not include the operating system kernel. As such, a given computer platform may, for example, contain multiple containers that share an operating system kernel through respective operating system kernel mount interfaces. Docker containers and rkt containers are examples of software containers. 
     In accordance with example implementations, each application operating environment  120  may have an OS or hypervisor interface  121  (called an “OS-hypervisor interface  121 ”), and collectively, the OS-hypervisor interfaces  121  may form a tenant workload isolation barrier  197  between the cloud tenant domain  111  and a cloud operator domain  113  of the domain node  110 - 1 . In accordance with example implementations, the cloud tenant domain  111  is considered to be an untrusted domain of the cloud computing system  100 , as the domain  111  is associated with cloud tenant software. The cloud operator domain  113  may be considered to be a trusted domain relative to the cloud tenant domain  111 . 
     In accordance with example implementations, the cloud operator domain  113  includes a blended physical and logical infrastructure, including physical hardware and trusted supervisory software components. The components associated with the supervisory services described herein are isolated from tenant workloads by the tenant workload isolation barrier  197  and a tenant workload isolation barrier  198  that is formed by the host interfaces of the smart I/O peripherals  180 . For example, the physical hardware components associated with the supervisory services may include one or multiple CPUs  116  and memory components that form a system memory  118 . In accordance with example implementations, the system memory  118  and other memories that are discussed herein are non-transitory storage media that may be formed from semiconductor storage devices, memristor-based storage devices, magnetic storage devices, phase change memory devices, a combination of devices of one or more of these storage technologies, and so forth. The system memory may represent a collection of both volatile memory devices and non-volatile memory devices. 
     The trusted supervisory software components of the cloud operator domain  113  may include, as examples, a hypervisor  167 , a basic input/output operating system (BIOS)  165 , a Unified Extensible Firmware Interface (UEFI) and an OS  163 . As also depicted in  FIG.  1   , the smart I/O peripherals  180  are within the cloud operator domain  113 . The cloud operator domain  113  may, in accordance with example implementations, include one or multiple host supervisory services that are provided by trusted supervisory software component(s). For example, the cloud operator domain  113  may include a pre-boot supervisory service  156  and one or multiple host supervisory services that operate during the post-boot, or runtime, of the domain node  110 - 1 , in accordance with example implementations. 
     More specifically, in accordance with some implementations, the runtime host supervisory services may include an OS-container runtime supervisory service  150 , a hypervisor-VM runtime supervisory service  152  and an OS-bare metal runtime supervisory service  154 . In general, as described herein, the host supervisory services may serve a wide variety of purposes for the domain node  110 - 1 , which bridge the architecture gap between the host-centric control model and the cloud-native control model. As examples, the host supervisory services may, in accordance with example implementations, configure the smart I/O peripherals  180 ; control when holds on I/O services provided by the smart I/O peripherals  180  are released based on ready state indicators that are provided by the smart I/O peripherals  180 ; provision resources of the smart I/O peripherals  180 ; provide proof of operating states of the smart I/O peripherals  180 ; align audit logs for the smart I/O peripherals  180 ; and so forth. 
     In accordance with an example implementation, the smart I/O peripheral  180  provides one or multiple ready state indicators (RSIs). An RSI represents a corresponding ready state of the smart I/O peripheral  180  and may be accessed by a host supervisory service for purposes of the service determining a ready state of the smart I/O peripheral  180 . As examples, the “readiness” of the smart I/O peripheral  180  may be a readiness of the smart I/O peripheral  180  to be configured by a host supervisory service; a readiness of the smart I/O peripheral  180  to proceed after an I/O service provided by the peripheral or a logical connection created by the peripheral  180  has been changed or created; a readiness of a physical or virtual function of the smart I/O peripheral  180 ; and so forth. 
     The RSIs may be RSIs for physical functions and/or virtual functions of the smart I/O peripheral  180 . More specifically, in accordance with example implementations, the smart I/O peripheral  180  provides one or multiple physical function RSIs  182 , where each physical function RSI  182  represents a ready state of a corresponding physical function (e.g., a PCIe physical function) of the smart I/O peripheral  180 . In accordance with example implementations, the smart I/O peripheral  180  may provide one or multiple virtual function RSIs  183 , where each virtual function RSI  183  represents a ready state of a corresponding virtual function (e.g., a PCIe virtual function) of the smart I/O peripheral  180 . 
     As described further herein, a host supervisory service may perform one or multiple actions that rely on RSIs that are provided by the smart I/O peripherals  180 . For example, a host supervisory service may place a hold on a startup of a VM instance or container instance until an RSI corresponding to a virtual function or physical function for that instance represents that the function is available, or ready. As another example, the host supervisory service may, responsive to an RSI state, place a hold on an instance&#39;s use of a particular I/O service that is provided by a smart I/O peripheral while the smart I/O peripheral is reconfiguring the I/O service and release the hold (thereby allowing the instance to use the I/O service) when the RSI state changes to indicate that the I/O service is ready. As another example, a host supervisory service may wait to set up a configuration space of a smart I/O peripheral  180  until an RSI represents that the smart I/O peripheral  180  is ready to proceed with the configuration. 
     In accordance with some implementations, a host supervisory service may directly read the RSIs from the smart I/O peripherals  180 . However, in accordance with further implementations, a service processor of the domain node  110 - 1 , such as a baseboard management controller (BMC)  123 , may monitor the RSIs for the smart I/O peripherals  180  and update a register space  128  of the BMC  123  with values representing the RSI states. For example, the register space  128  may include one or multiple registers, where the registered bit fields correspond to different RSIs that indicate respective RSI states. The host supervisory service may, for example, read the BMC register space  128  via a sideband communication channel to determine a given RSI state. 
     In accordance with some implementations, all error notifications for the domain node  110 - 1  are directed to the BMC  123 . Moreover, the BMC  123 , in accordance with example implementations, may receive “hot plug notifications” from the host (e.g., the operating system  163  or hypervisor  167 ) responsive to corresponding instance changes. A “hot plug notification,” as used herein, refers to a notification that a particular backend I/O service of the smart I/O peripheral  180  is to be placed on hold. This hold on the service occurs while the smart I/O peripheral  180  performs a reconfiguration to address the instance change. The BMC  123 , in accordance with example implementations, may store data in its register space  128  representing the hold on a particular service that is provided by the smart I/O peripheral  180 . As used herein, an “instance change” (or “instance change event”) refers to a change that is associated with an instance (e.g., a VM instance or a container instance), which corresponds to a change in one or multiple configurations of a smart I/O peripheral  180 . As examples, an instance change may be associated with a configuration change for a virtual function or a physical function of a smart I/O peripheral  180 , a startup of an instance, a termination of an instance, an allocation of processing resources (e.g., cores) to an instance, a deallocation of resources for an instance, and so forth. 
     In accordance with example implementations, in response to an instance change, the host supervisory service may quiesce the instance&#39;s use of one or multiple affected I/O services until the smart I/O peripheral makes the configuration changes to address the instance change. As further described herein, in accordance with example implementations, the host supervisory service restores the instance to normal operation (e.g., the hold on the instance&#39;s use of the I/O service(s) is released) in response to one or multiple corresponding RSIs representing, or indicating, that the corresponding configuration change(s) of the smart I/O peripheral  180  are in place to allow the execution of the instance to proceed. 
     In general, as used herein, a “BMC” is a specialized service processor that monitors the physical state of a server, node, or other hardware entity using sensors and communicates with a management system through a management network. The BMC may communicate with applications executing at the operating system level through an input/output controller (IOCTL) interface driver, a representational state transfer (REST) application program interface (API), or some other system software proxy that facilitates communication between the BMC and applications. The BMC may have hardware level access to hardware devices of the hardware entity, including a system memory, local memories, and so forth. The BMC may be able to directly modify the hardware devices. The BMC may operate independently of any operating system instances of the hardware entity. The BMC may be located on a motherboard or main circuit board of the hardware entity. The fact that the BMC is mounted on the motherboard or otherwise connected or attached to the motherboard does not prevent the BMC from being considered “separate” from the processors, which are being monitored/managed by the BMC. As used herein, a BMC has management capabilities for sub-systems of the hardware entity, and is separate from a processing resource that executes an operating system of the computing device. 
     Regardless of its particular form or implementation, the BMC  123  may include one or multiple embedded processors  124  that execute machine executable instructions  125  (stored in a memory  126  of the BMC  123 ), for purposes of performing actions by the BMC  123 , as described herein. In accordance with example implementations, these actions may include communicating with the smart I/O peripherals  180  for purposes of reading the RSIs  182  and  183 . In accordance with example implementations, the BMC  123  may provide one or multiple application programming interfaces (APIs) for reading the RSIs  182  and  183 . For example, a host supervisory service may submit an API request to the BMC  123  for a particular RSI value, and the processor  124  of the BMC  123  may, responsive to the API request, read the RSI value and provide the read RSI value in a corresponding API response. In accordance with further implementations, the BMC  123  may regularly read the RSI values and store the RSI values in the register space  128  of the BMC  123 . A host supervisory service may, for example, read a particular register bit in the register space  128  to determine a state of a particular RSI. One or multiple processors  124  of the BMC  123  may, in accordance with example implementations, execute machine executable instructions  125  for purposes of receiving and managing hot plug notifications from the host. 
     The BMC  123  may perform the above-described roles as part of its management plane. The BMC  123  may provide a wide variety of other management services, other than those described above, such as monitoring sensors (e.g., temperature sensors, cooling fan speed sensors); monitoring an operating system status; monitoring power statuses; controlling power on and power off of the computer system; logging computer system events; allowing remote management of the computer system; performing recovery operations after an operating system or computer system failure; and so forth. In accordance with some implementations, the BMC  123  may also have a security plane, in which the BMC  123  performs various security-related functions for the domain node  110 - 1 , such as validating firmware before the firmware is allowed to be loaded and executed by the processor  124 ; storing, generating and providing cryptographic keys; and so forth. 
     In accordance with example implementations, the smart I/O peripheral  180  may include one or multiple hardware processing cores  187 , which execute instructions  189  stored in a memory  181  of the peripheral  180  for purposes of performing its functions that are described herein. In this manner, in accordance with some implementations, the instructions  189  may be associated with a software stack of the smart I/O peripheral  180 . In general, one or multiple processors  187  may execute the software stack for purposes of managing states of the RSIs  182  and  183  and providing one or multiple backend I/O services for the smart I/O peripheral  180 . In accordance with example implementations, one or multiple processors  187  may execute the instructions  189  for purposes of performing an example process  600  ( FIG.  6   ) to manage the RSI states. The smart I/O peripheral  180  may further include one or multiple hardware accelerators  175  (e.g., P4 hardware engines) for purposes of performing a wide variety of functions, such as encryption, decryption, packet inspection (e.g., packet inspection for a distributed firewall service), and so forth. It is noted that the smart I/O peripheral  180  may have a variety of different architectures, including components not illustrated in  FIG.  1   . Among its other components, the smart I/O peripheral  180  may include one or multiple network interface controllers (NICs)  186  that contain corresponding ports (e.g., Ethernet ports) for purposes of communicating with the physical network fabric  184 . In accordance with example implementations, the smart I/O peripherals form a network isolation barrier  199  of the domain node  110 - 1 . 
     As depicted in  FIG.  1   , in accordance with example implementations, the smart I/O peripheral  180  may include a configuration space  179 , such as a PCIe configuration space, that is set up, or configured by, a host supervisory service, as further described herein. 
     In accordance with some implementations, the smart I/O peripheral  180  includes a register space  177  that contains bit fields that represent respective RSIs (e.g., physical functions RSIs  182  and virtual function RSIs  183 ). In accordance with example implementations, the register space  177  may be accessible by the host supervisory services (e.g., the register space  177  may be accessible by an in-band communication bus, such as a PCIe bus). In accordance with further implementations, the register space  177  may be accessible by the BMC  123  via a sideband communication bus (e.g., the register space  177  may be accessible by a Serial Peripheral Interface (SPI) bus or an extended SPI (eSPI) bus). 
     In accordance with some implementations, the register space  177  may store data representing whether the smart I/O peripheral  180  is offline. For example, in accordance with some implementations, one or multiple processors  187  of the smart I/O peripheral  180  may execute instructions  189  to manage the offline indication. More specifically, in accordance with some implementations, an entity (e.g., a CPU  116  or BMC  123 ) reading the state of an RSI  182  or  183  may acknowledge the reading of the state by writing a corresponding acknowledgment bit (e.g., a bit of the register space  177 ). This allows the smart I/O peripheral  180  to determine whether or not the peripheral  180  is installed in a platform that uses the RSIs. For example, after the smart I/O peripheral  180  transitions through its power up sequence, the smart I/O peripheral  180  may read an acknowledgement bit corresponding to an RSI (e.g., an RSI corresponding to Primary Function Zero, as discussed further herein) to determine if a state of the RSI state has been read. If, for example, the smart I/O peripheral  180  determines that the state has not been acknowledged and the configuration space  179  of the smart I/O peripheral  180  has not been set up, then the smart I/O peripheral  180  may set an offline bit to indicate that the smart I/O peripheral  180  has taken itself offline. 
     Referring to  FIG.  3 A  in conjunction with  FIG.  1   , the pre-boot supervisory service  156  operates during the pre-boot or pre-operating system environment of the domain node  110 - 1  (i.e., the environment of the domain node  110 - 1 ). A CPU  116  may execute BIOS or UEFI instructions to provide the pre-boot supervisory service  156 , and providing the pre-boot supervisory service  156  may involve the CPU  116  communicating with the smart I/O peripheral  180  and the BMC  123 . The pre-boot supervisory service  156 , in accordance with example implementations, may set up the tenant visibility of resources of the domain node  110 - 1 . Resources may be hidden from a tenant for a number of different reasons, such as, for example, the tenant not wanting to pay for the resources, controlling the quality of service (as exposing the resources may affect performance of the domain node  110 - 1 ), and so forth. In accordance with some implementations, the pre-boot supervisory  156  may control the visibility of certain physical function(s) and/or virtual function(s) of the smart I/O peripheral  180  accordingly so that the function(s) are not exposed to the node manager  190 . The resources that are hidden may be any of a number of different resources, such as devices (e.g., graphics processing units (GPUs), CPUs, NVMe drives, and so forth), memory above a certain capacity, CPU core frequency above a certain limit, and so forth. In general, the pre-boot supervisory service  156  takes and inventory of the logical and physical infrastructure resources; controls the visibility of resources of the domain node  110 - 1 ; and sets up the application operating environments  120 . The pre-boot supervisory service  156 , in accordance with example implementations, searches for the node manager  190  (e.g., the pre-boot supervisory service  156  searches over an encrypted overlay network) and binds the domain node  110 - 1  to the node manager  190  for control. 
     As part of performing the pre-boot supervisory service  156 , the CPU  116  may, for example, set up a PCIe configuration space of the smart I/O peripheral  116  and determine an operational state of the smart I/O peripheral  116 . For example, the CPU  116  may determine whether the smart I/O peripheral  180  is active or offline. As part of the pre-boot supervisory service  156 , the CPU  116  may perform other actions, such as setting up the application operating environments  120 , aligning audit logs, verifying proof of an operating state, and so forth. 
     As part of the pre-boot supervisory service  156 , the CPU  116  may perform a bus enumeration, (e.g., a PCIe bus enumeration), which includes detecting the presence of smart I/O peripherals  180  and setting up bus configuration space (e.g., a PCIe configuration space) of each detected smart I/O peripheral. As a more specific example, in accordance with some implementations, the smart I/O peripheral  180  may be a PCIe smart I/O peripheral  180 , which has a Device 0, Function 0 (called a “Primary Function Zero” herein). The smart I/O peripheral  180  controls a bit value (i.e., a physical function RSI  182 ) of a control configuration register that is associated with the Primary Function Zero to indicate whether or not the overall smart I/O peripheral  180  is ready to be configured. As further described herein, such a bit value may be useful for PCIe bus enumeration for purposes of a bus enumeration service determining whether to proceed with configuring the smart I/O peripheral  180  after detecting the presence of the smart I/O peripheral  180 . In accordance with further implementations, the smart I/O peripheral  180  may provide a physical RSI  182 , other than an RSI associated with Primary Function Zero, for purposes of representing or indicating whether the smart I/O peripheral  180  is ready to be configured. As an example, in accordance with some implementations, responsive to the Physical Function Zero RSI representing that the smart I/O peripheral  180  is ready to be configured, the host supervisory service  156  may write to the configuration space of the smart I/O peripheral  180  for purposes of setting up the memory space and/or I/O space for the smart I/O peripheral  180 . 
     Referring to  FIG.  2    in conjunction with  FIG.  1   , due to the relative complexity of the smart I/O peripheral  180 , the smart I/O peripheral  180  may not be able to be ready for configuration within a set predetermined time (e.g., the one set PCIe time limit).  FIG.  2    depicts a power-up sequence  200  for a smart I/O peripheral  180  in accordance with an example implementation. As shown in  FIG.  2   , the power-up sequence occurs over a time  204  and involves interaction between the node manager  190  (e.g., a node manager of the cloud&#39;s management system) and the smart I/O peripheral. After the power on or reset of the smart I/O peripheral  180 , the peripheral  180  undergoes a first phase  204  in which the smart I/O peripheral configures its physical infrastructure. For example, this may involve setting up audit logs, setting up telemetry logs, determining a security health, setting up a working baseline measurement, and so forth. Next, in an SDN configuration phase  208 , the smart I/O peripheral  180  sets up its SDN connections of one or multiple overlay networks. This may involve setting up resiliency logs, setting up performance logs, and evaluating a security health. Subsequently, in an SDS configuration phase  212 , the smart I/O peripheral  180  may set up SDS connections. The phase  212  may include determining storage capacities, setting up isolations and access, and evaluating a security health. In an SDC phase  216 , the smart I/O peripheral  180  may determine a number of instances of virtual cores, determine parameters of virtual memory instances, determining numbers and types of network instances, and evaluating a security health. In accordance with example implementations, at the conclusion of the SDC phase  216 , the smart I/O peripheral  180  is ready to be configured by the pre-boot host supervisory service  156 . In a subsequent services configuration phase  220  that is depicted in  FIG.  2   , the smart I/O peripheral  180  may set up one or multiple backend I/O services to be provided by the smart I/O peripheral  180 . This set up may include instance deployment and sequencing; setting up instance audit and telemetry logs; setting up instance clusters; setting up failover clusters; setting up instance data replication; evaluating instance certifications and compliance; and so forth. 
       FIG.  4    depicts a bus enumeration process  400  that may be used by the host supervisory service  156  in response to a power on or reset of the computer platform  110 - 1 . Referring to  FIG.  4    in conjunction with  FIGS.  1  and  3 A , the process  400  may be initiated upon a power on or reset of the domain node  110 - 1  for purposes of detecting and configuring bus devices. More specifically, in accordance with example implementations, the host supervisory service  156  detects (block  402 ) the smart I/O peripheral  180 . 
     As an example, in accordance with some implementations, the smart I/O peripheral  180  may be a PCIe bus card. In general, for PCIe bus enumeration, the host supervisory service  156  may attempt to read a vendor identification (ID) and device ID for different bus device, bus number and physical function combinations for purposes of detecting corresponding PCIe bus devices. Here, a “PCIe bus device” refers to a particular PCIe physical function. Because all PCIe cards implement Primary Function Zero (i.e., physical function 0 for device 0), in accordance with example implementations, the host supervisory service  156  may attempt to read a bus device ID and a vendor ID for Primary Function Zero for all bus numbers and bus device numbers for purposes of detecting PCIe bus devices. 
     In accordance with example implementations, responsive to detecting the smart I/O peripheral  180 , the host supervisory service  156  determines (block  404 ) the state of the physical RSI  182 , which corresponds to Primary Function Zero. Based on the state of the RSI  182 , the host supervisory service  156  may then determine (decision block  408 ) whether the smart I/O peripheral  180  is ready to be configured. 
     More specifically, in accordance with example implementations, the host supervisory service  156  may program, or configure, the configuration space  179  ( FIG.  1   ) of the smart I/O peripheral  180  based on configuration criteria, which the smart I/O peripheral  180  provides via data in the configuration space  179 . For example, for implementations in which the smart I/O peripheral  180  is a PCIe card, the smart I/O peripheral  180  may communicate, through Base Address Registers (BARs), of the configuration space  280  corresponding memory criteria for the different corresponding physical functions of the smart I/O peripheral  180 . In this manner, the smart I/O peripheral  180  may, for example, for a particular physical function, request the allocation of memory space or I/O space, as well as the particular amount of memory and/or I/O space. When the smart I/O peripheral  180  initially boots up, however, after a power on or reset, the smart I/O peripheral  180  may not be ready for a duration of time to present the requested memory criteria, and the smart I/O peripheral  180  may not be ready for its configuration space  280  to be programmed by the host supervisory service  156 . By monitoring the corresponding physical function RSI  182 , the host supervisory service  156  may determine when to proceed with the configuration of configuration space  280 . 
     Therefore, upon determining (decision block  408 ) that the smart I/O peripheral  180  is ready to proceed with the configuration, pursuant to block  412 , the host supervisory service  156  communicates with the smart I/O peripheral  180  via the in-band communication channel  211  to determine configuration criteria requested by the smart I/O peripheral  180 . Pursuant to block  416 , the host supervisory service  156  sets up the configuration space  280  of the smart I/O peripheral  180  based on the requested configuration criteria. 
     Referring to  FIG.  3 B , in accordance with example implementations, the host supervisory service  150  may be associated with a bare metal operating system with direct I/O application operating environment  120 . For this example, an  0 /S container runtime host supervisory service  150  may respond to an indication from the BMC  123  that a particular configuration change event has occurred with a container instance  324 . In response to the container change event, the host supervisory service  150  may perform a process  500  that is depicted in  FIG.  5   . 
     Referring to  FIG.  5    in conjunction with  FIGS.  1  and  3 B , the process  500  may be performed in response to a runtime container instance change event, such as, for example, a container instance being created, a container instance ending, more cores being assigned to a container instance, and so forth. These changes may correspond to configuration changes for one or multiple physical or virtual functions of the smart I/O peripheral  180 . For example, a particular virtual function may be assigned to a new container instance, a new SDN connection may be set up, an SDN connection may be removed, a virtual function may be reassigned to another container instance, and so forth. 
     Pursuant to the process  500 , the host supervisory service  150  places (block  501 ) the container instance in a quiescent mode of operation (i.e., quiesces the container instance) to accommodate the change event. In this context, placing the container instance in the quiescent mode of operation refers to quiescing, or silencing, operations of the container instance that are affected by the change event. Stated differently, the quiescing may place the container instance&#39;s use of one or multiple I/O services that are affected by the change event on hold. For example, the change event may correspond to a reconfiguration of a particular virtual function of the smart I/O peripheral, and placing the container instance in a quiescent mode of operation may include the halting of the sending of workload transactions to a virtual function and draining, or removing, any outstanding transactions that are waiting for a response from the virtual function. Pursuant to block  502 , the host supervisory service  150  communicates with the smart I/O peripheral  180  to initiate a configuration change in one or multiple services that are provided by the smart I/O peripheral  180 . To make these changes, the smart I/O peripheral  180  may take an indeterminant amount of time. Accordingly, in accordance with example implementations, the host supervisory service  150  determines (block  504 ) the state(s) of the corresponding RSI(s)  182  and/or  183  and based on the state(s), determines (decision block  508 ) whether the changed function(s) are ready. When ready, then, pursuant to block  512 , the host supervisory service  150  reverts the container instance back to the normal mode of operation in which the container instance resumes using the function(s) that were affected by the configuration change. 
     In a similar manner, referring to  FIG.  3 C , a particular hypervisor-VM runtime host supervisory service  152  may operate in connection with a hypervisor with direct I/O application operating environment  120 . In this manner, the host supervisory service  152  may, in response to a configuration change event with a particular VM instance  344 , perform a process to quiesce operations of the VM instance  344  that are affected by the configuration change event until the corresponding RSI(s) indicate, or represent, that the corresponding backend I/O service(s) are ready; and when this occurs, the host supervisory service  152  returns the VM instance  344  back to a normal mode. In accordance with further implementations, the hypervisor with direct I/O application operating environment  120  of  FIG.  3 C  may be replaced with kernel-based virtualization environment, such as a kernel-based virtual machine (KVM) that is a module of the Linux kernel. 
       FIG.  3 D  depicts an OS-bare metal runtime host supervisory service  154  that works in conjunction with a bare metal O/S instance-based operating environment  120 . As an example, the host supervisory service  154  may perform BIOS-related and/or UEFI-related configuration services, which use the RSIs that are provided by the smart I/O peripherals  180 . For example, in accordance with some implementations, the BMC  123  may receive a hot plug notification in response to a particular card-based smart I/O peripheral  180  being installed in or removed from the domain node  110 - 1  while the domain node  110 - 1  is powered on and is in the runtime environment. For this example, the host supervisory service  154  may rely on an RSI of the smart I/O peripheral  180  to setup a configuration space of the peripheral  180 , similar to process  400  ( FIG.  4   ). In accordance with example implementations, the host supervisory service  154  may perform other BIOS-related or UEFI-related supervisory services for the smart I/O peripherals  180 . 
     Referring back to  FIG.  1   , in accordance with example implementations, one or multiple processing cores  187  of the smart I/O peripheral  180  may execute the instructions  189  for purposes of managing the RSIs  182  and  183  pursuant to a process  600  that is illustrated in  FIG.  6   . Referring to  FIG.  6    in conjunction with  FIG.  1   , the process  600  may be performed in response to a configuration or re-configuration of logical connections and/or backend I/O services that are provided by the smart I/O peripheral  180 . Pursuant to the process  600 , the processing core(s)  187  write (block  602 ) data to the register space  177  to change the state(s) of one or multiple RSIs  182  and/or  183  that are affected by the configuration to indicate that the corresponding physical and/or virtual functions are not ready. The processing core(s)  187  may then perform the configuration, pursuant to block  604 . If the processing core(s)  187  determine (decision block  606 ) that the function(s) are ready, then the processing core(s)  187  writes (block  612 ) data to the register space  177  to change the RSI state(s) affected by the configuration to indicate that the corresponding function(s) are ready. 
     Referring to  FIG.  7   , in accordance with example implementations, a process  700  includes, at least one hardware processor of a host executing (block  704 ) first instructions associated with an application operating environment and associated with an untrusted cloud tenant domain. The application operating environment uses a service of a smart input/output (I/O) peripheral; and the service of the smart I/O peripheral is associated with a trusted cloud operator domain. The process  700  includes extending (block  708 ) a supervisory service of the host into the trusted cloud operator domain. Extending the supervisory service includes the hardware processor(s) executing second instructions to supervise the smart I/O peripheral. Supervising the smart I/O peripheral includes the hardware processor(s) configuring the smart I/O peripheral and determining a state of a ready state indicator that is provided by the smart I/O peripheral. Supervising the smart I/O peripheral further includes determining whether the smart I/O peripheral is ready to be configured based on the state. 
     Referring to  FIG.  8   , in accordance with example implementations, a system  800  includes a connector  804  and a node  812 . The connector  804  receives a smart input/output (I/O) peripheral. The smart I/O peripheral is to provide an I/O service that is associated with a trusted cloud operator domain. The node  812  includes at least one central processing unit (CPU)  816  and a memory  820 . The CPU(s)  816  execute instructions  822  that are associated with an application operating environment and associated with an untrusted cloud tenant domain. The CPU(s)  816  execute instructions  824  that are associated with the trusted cloud operator domain to perform a supervisory service for the smart I/O peripheral in response to a change in an instance that is associated with the application operating environment. The change corresponds to a configuration change of the smart I/O peripheral. As part of the supervisory service, the CPU(s) execute the instructions  824  determine a state of a ready state indicator that is provided by the smart I/O peripheral and based on the state, regulate an availability of the instance. 
     Referring to  FIG.  9   , in accordance with example implementations, a non-transitory storage medium  910  stores machine-readable first instructions  914  that, when executed by a machine, cause the machine to provide an input/output (I/O) service that is associated with a trusted cloud operator domain and is associated with a smart I/O peripheral. The instructions  914 , when executed by the machine, cause the machine to manage a ready state indicator of the smart I/O peripheral that is accessible by a supervisory service of a host and represents a ready state of the smart I/O peripheral. The supervisory service is associated with the trusted cloud operator domain. The host provides an instance that is associated with an application operating environment and is associated with untrusted cloud tenant domain. The instance uses the I/O service. The instructions  914 , when executed by the machine, cause the machine to, in response to a configuration event, cause the ready state indicator to represent that a function that is associated with the I/O service and is provided by the smart I/O peripheral is unavailable. Moreover, the instructions  914 , when executed by the machine, cause the machine to change a configuration associated with the function and in response to the change of the configuration being complete, cause the ready state indicator to represent that the function is available. 
     In accordance with example implementations, the smart I/O peripheral includes a bus device. Configuring the smart I/O peripheral includes setting up a configuration space of the smart I/O peripheral as part of a bus enumeration process. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the smart I/O device includes a bus device that is associated with a primary physical function. Supervising the smart I/O peripheral further includes the supervisory service detecting a presence of the bus device. Determining the state of the ready state indicator includes determining a state of a ready state indicator associated with the primary physical function. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, executing the second instructions includes the hardware processor(s) executing at least one of basic input/output system (BIOS) instructions or Unified Extensible Firmware Interface (UEFI) instructions. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the smart I/O peripheral, responsive to powering up or being reset, undergoes a power up sequence, and the ready state indicator corresponds to a delay associated with the power up sequence. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the smart I/O peripheral, in the power up sequence, configures logical connections and a service associated with the logical connections. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the ready state indicator represents whether a configuration change is complete. Responsive to a change in the instance, a hold is placed on use of the I/O service by the instance. Responsive to the state representing that the configuration change is complete, the hold is released. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the change in the instance is a startup of the instance, a termination of the instance or a change in a resource allocation for the instance. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the configuration change is a change affecting at least one of a virtual function of the smart I/O peripheral associated with the instance or a physical function of the smart I/O peripheral associated with the instance. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, a baseboard management controller reads the state of the ready state indicator and provides a representation of the state to the supervisory service. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, a baseboard management controller stores a representation of the state in a register space of the baseboard management controller. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, a baseboard management controller receives a notification of the change in the configuration and notifies the supervisory service responsive to receiving the notification. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the instance is a VM instance. An operating system places a hold on use of the I/O service by the VM instance. Responsive to the state representing that the configuration change is complete, the hold is released. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the instance is a container instance. A hypervisor places a hold on use of the I/O service by the container instance. Responsive to the state representing that the configuration change is complete, the hold is released. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the smart I/O peripheral provides a single root-input output virtualization (SR-IOV), and the change in the configuration corresponds to a virtual function of the SR-IOV. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, a failure by the supervisory service to configure the smart I/O peripheral is detected, and in response to the detected failure, data is stored in a register space of the smart I/O peripheral representing that the smart I/O peripheral is offline. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the configuration event is an event that is associated with a power on or reset of the smart I/O peripheral. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     In accordance with example implementations, the configuration event is associated with an instance of an application operating environment of the host. A particular advantage, in accordance with some implementations, is that for a segmented domain node having a smart I/O peripheral, an architecture gap between a host-centric control model and a cloud-native architecture may be bridged without modifying a legacy server hardware. 
     While the present disclosure has been described with respect to a limited number of implementations, those skilled in the art, having the benefit of this disclosure, will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations.