## 4 New Virtualization Solutions for Network Functions
In the ETSI architecture, the interaction between NFVO and VNFM implements the lifecycle operation of VNFs. In particular, VNFM requests grants from NFVO before starting operations on controlled VNFs. When Kubernetes is  
Figure 3: New entities in NFV control plane (CISM, CCM, CIR) introduced to manage CIS cluster in NFVI as envisioned in Ref. [30]. Solid lines indicate NFV interfaces, dashed lines indicate other additional interfaces. Reference points names are omitted to improve readability. used, it does not interact with NFVO to scale its controlled Pods in and out, or to decide which worker node to instantiate them on. NFVO has to define only the desired final state in a declarative style (e.g., intent-driven constructs, such as deployment and YAML files), and the Kubernetes scheduler acts to reach that state, while VNFM keeps a detailed view of the status of its controlled VNFs, including virtualization information, and exposes this information to NFVO; Kubernetes does not expose anything on its internal status. The only way to control operations is to define appropriate intents to achieve. Kubernetes manages the lifecycle of CNFs implemented in Pods and performs scaling operations through changes in deployments by specifying constraints in manifest files. To sum up, the usage of Kubernetes breaks the tight control loop designed by ETSI MANO, replacing it with multiple small, declarative control loops, characterized by greater flexibility. In addition, the new structure of the NFV infrastructure (NFVI), capable of supporting containerized VNFs [35] is depicted in Figure 4. It shows that the containerized VNFs have two main options: to deploy the CIS either on the bare metal or inside VMs. The first option offers more flexibility and improved performance, but lacks some security and isolation features, as discussed above. On the other side, VMs can be effectively used to provide isolation of CIS (e.g., devoted to a single slice), but they suffer from promptness, resource efficiency and performance issues. Our solution addresses the limitations of the solutions envisioned in Ref. [35], which are shown in Figure 4. It is compliant with the latest ETSI NFV white paper [29], which provides a future outlook. Figure 2 shows the NFVI stack proposed in this paper. In particular, it relies on lightweight virtualization technologies in order to both overcome the limitations of traditional VMs and fulfil the CNFs' security needs. The virtualization layer consists of two sub-layers. In the top sub-layer, we use Kata containers [36], an OpenStack Foundation project. Unlike traditional container runtime technologies, such as Containerd [19], which rely on kernel features such as Cgroups and namespaces to provide isolation with the shared kernel, Kata offers better isolation by enclosing containers in lightweight VMs (_microVMs_). In particular, each Kata container runs in a dedicated and isolated kernel within a dedicated microVM. The Firecracker hypervisor [37] accelerates the deployment of Kata microVMs, provides additional isolation, and represents the lower virtualization sub-layer of our solution. New VNFCs are built and executed in NetBricks [38], an NFV framework which includes both a programming language and an execution environment. It makes use of the rust and data plane development kit (DPDK) [38]. In addition, it is also  
Figure 4: NFV infrastructure as envisioned in Ref. [35], providing containers on bare metal (right side of NFVI) and VMs (left side of NFVI). possible to reuse standard packages, such as Kafka or MQTT brokers, which safely run in Kata containers. OpenStack or Kubernetes can be used for resource management for such infrastructure, occupying the virtual infrastructure manager (VIM) block in the NFV reference architectural framework, as shown in Figures 3 and 4.