Project: Kuasar (Runtime / Container Runtime)
File: how-to-run-kuasar-with-isulad-and-stratovirt.md

Prepare a new aarch64 architecture physical server Optional - Recommend to install openEuler 23.03 version OS in the server openEuler 23.03 version OS image download link: https://repo.openeuler.org/openEuler-23.03/ISO/ Note: All of the following commands need to run with root privilege. If you use openEuler 23.03 OS in your physical server, you can directly install the newest StratoVirt package by yum. ```bash $ yum install stratovirt ``` If you use another Linux distribution OS, you can build the stratovirt from the source and install it: After you build or install the stratovirt package, you can find the following important binary file in your sever: ```bash $ ls /usr/bin/stratovirt /usr/bin/stratovirt $ ls /usr/bin/vhostuserfs /usr/bin/vhostuserfs ``` Kuasar use `docker` or `containerd` container engine to build guest os initrd image, so you need to make sure `docker` or `containerd` is correctly installed and can pull the image from the dockerhub registries. Tips: `make vmm` build command will download the Rust and Golang packages from the internet, so you need to provide the `httpproxy` and `httpsproxy` environments for the `make all` command. If a self-signed certificate is used in the `make all` build command execution environment, you may encounter SSL issues with downloading resources from https URL failed. Therefore, you need to provide a CA-signed certificate and copy it into the root directory of the Kuasar project, then rename it as "proxy.crt". In this way, our build script will use the "proxy.crt" certificate to access the https URLs of Rust and Golang installation packages. ```bash $ HYPERVISOR=stratovirt make vmm $ HYPERVISOR=stratovirt make install-vmm ``` After installation, you will find the required files in the specified path ```bash /usr/local/bin/vmm-sandboxer /var/lib/kuasar/vmlinux.bin /var/lib/kuasar/kuasar.initrd /var/lib/kuasar/config_stratovirt.toml ``` supports Kuasar with its master branch at the moment. For building iSulad from scratch, please refer to . Here we only emphasize the difference of the building steps. ```bash $ git clone https://gitee.com/openeuler/lcr.git $ cd lcr $ mkdir build $ cd build $ sudo -E cmake .. $ sudo -E make -j $(nproc) $ sudo -E make install ``` ```bash $ git clone https://gitee.com/openeuler/iSulad.git $ cd iSulad $ mkdir build $ cd build $ sudo -E cmake .. -DENABLECRIAPIV1=ON -DENABLESHIMV2=ON -DENABLESANDBOXER=ON $ sudo make -j $ sudo -E make install ``` Add the following configuration in the iSulad configuration file `/etc/isulad/daemon.json` ```json ... "default-sandboxer": "vmm", "cri-sandboxers": { "vmm": { "name": "vmm", "address": "/run/vmm-sandboxer.sock" } }, "cri-runtimes": { "vmm": "io.containerd.vmm.v1" }, ... ``` Sine some code have not been merged into the upstream containerd community, so you need to manually compile the containerd source code in the . git clone the codes of containerd fork version from kuasar repository. ```bash $ git clone https://github.com/kuasar-io/containerd.git $ cd containerd $ make bin/containerd $ install bin/containerd /usr/bin/containerd ``` Add the following sandboxer config in the containerd config file `/etc/containerd/config.toml` ```toml [proxy_plugins]

type = "sandbox" address = "/run/vmm-sandboxer.sock" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.vmm] runtime_type = "io.containerd.kuasar.v1" sandboxer = "vmm" io_type = "hvsock" ``` Install which is required by to configure pod network. ```bash $ wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-arm64-v1.2.0.tgz mkdir -p /opt/cni/bin/ mkdir -p /etc/cni/net.d tar -zxvf cni-plugins-linux-arm64-v1.2.0.tgz -C /opt/cni/bin/ ``` ```bash VERSION="v1.15.0" # check latest version in /releases page wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-arm64.tar.gz sudo tar zxvf crictl-$VERSION-linux-arm64.tar.gz -C /usr/local/bin rm -f crictl-$VERSION-linux-arm64.tar.gz ``` create the crictl config file in the `/etc/crictl.yaml` ```bash cat /etc/crictl.yaml runtime-endpoint: unix:///var/run/isulad.sock image-endpoint: unix:///var/run/isulad.sock timeout: 10 ``` The default config file `/var/lib/kuasar/config_stratovirt.toml` for stratovirt vmm-sandboxer: ```toml [sandbox] log_level = "info" [hypervisor] path = "/usr/bin/stratovirt" machine_type = "virt,mem-share=on" kernel_path = "/var/lib/kuasar/vmlinux.bin" image_path = "" initrd_path = "/var/lib/kuasar/kuasar.initrd" kernelparams = "task.loglevel=debug task.sharefs_type=virtiofs" vcpus = 1 memoryinmb = 1024 blockdevicedriver = "virtio-blk" debug = true enablememprealloc = false [hypervisor.virtiofsd_conf] path = "/usr/bin/vhostuserfs" ``` ```bash $ ENABLECRISANDBOXES=1 ./bin/containerd ``` ```bash $ systemctl start kuasar-vmm ``` ```bash $ cat podsandbox.yaml metadata: attempt: 1 name: busybox-sandbox2 namespace: default uid: hdishd83djaidwnduwk28bcsc log_directory: /tmp linux: namespaces: options: {} $ crictl runp --runtime=vmm podsandbox.yaml 5cbcf744949d8500e7159d6bd1e3894211f475549c0be15d9c60d3c502c7ede3 ``` Tips: `--runtime=vmm` indicates that containerd needs to use vmm-sandboxer runtime to run a pod sandbox List pod sandboxes and check the sandbox is in Ready state: ```bash $ crictl pods POD ID CREATED STATE NAME NAMESPACE ATTEMPT 5cbcf744949d8 About a minute ago Ready busybox-sandbox2 default 1 ``` Create a container in the podsandbox ```bash $ cat pod-container.yaml metadata: name: busybox1 image: image: docker.io/library/busybox:latest command: top log_path: busybox.0.log no pivot: true $ crictl create 5cbcf744949d8500e7159d6bd1e3894211f475549c0be15d9c60d3c502c7ede3 pod-container.yaml podsandbox.yaml c11df540f913e57d1e28372334c028fd6550a2ba73208a3991fbcdb421804a50 ``` List containers and check the container is in Created state: ```bash $ crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID c11df540f913e docker.io/library/busybox:latest 15 seconds ago Created busybox1 0 5cbcf744949d8 ``` Start container in the podsandbox ```bash $ crictl start c11df540f913e57d1e28372334c028fd6550a2ba73208a3991fbcdb421804a50 $ crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID c11df540f913e docker.io/library/busybox:latest 2 minutes ago Running busybox1 0 5cbcf744949d8 ``` Get the `vsock guest-cid` from stratovirt vm process ```bash $ ps -ef | grep stratovirt | grep 5cbcf744949d8 /usr/bin/stratovirt -name sandbox-5cbcf744949d8500e7159d6bd1e3894211f475549c0be15d9c60d3c502c7ede3 ... -device vhost-vsock-pci,id=vsock-395568061,guest-cid=395568061,bus=pcie.0,addr=0x3,vhostfd=3 ... ``` Enter the guest os debug console shell environment: ```bash $ ncat --vsock 395568061 1025 bash-5.1# busybox ip addr show busybox ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo validlft forever preferredlft forever inet6 ::1/128 scope host validlft forever preferredlft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWERUP> mtu 1500 qdisc pfifofast qlen 1000 link/ether 42:e2:92:d4:39:9f brd ff:ff:ff:ff:ff:ff inet 172.19.0.240/24 brd 172.19.0.255 scope global eth0 validlft forever preferredlft forever inet6 fe80::40e2:92ff:fed4:399f/64 scope link validlft forever preferredlft forever bash-5.1# busybox ping 172.19.0.1 busybox ping 172.19.0.1 PING 172.19.0.1 (172.19.0.1): 56 data bytes 64 bytes from 172.19.0.1: seq=0 ttl=64 time=0.618 ms 64 bytes from 172.19.0.1: seq=1 ttl=64 time=0.116 ms 64 bytes from 172.19.0.1: seq=2 ttl=64 time=0.152 ms ```