Project: KubeArmor (Provisioning / Security & Compliance)
File: development_guide.md

> Note Skip the steps for the vagrant setup if you're directly compiling KubeArmor on the Linux host. Proceed to setup K8s on the same host by resolving any dependencies. Requirements Here is the list of requirements for a Vagrant environment ```text Vagrant - v2.2.9 VirtualBox - v6.1 ``` Clone the KubeArmor github repository in your system ```text $ git clone https://github.com/kubearmor/KubeArmor.git ``` Install Vagrant and VirtualBox in your environment, go to the vagrant path and run the setup.sh file ```text $ cd KubeArmor/contribution/vagrant ~/KubeArmor/contribution/vagrant$ ./setup.sh ~/KubeArmor/contribution/vagrant$ sudo reboot ``` VM Setup using Vagrant Now, it is time to prepare a VM for development. To create a vagrant VM ```text ~/KubeArmor/KubeArmor$ make vagrant-up ``` Output will show up as ... <details> <summary>Click to expand!</summary> ```text cd /home/gourav/KubeArmor/contribution/vagrant; NETNEXT=0 DLV_RPORT=2345 vagrant up; true Bringing machine 'kubearmor-dev' up with 'virtualbox' provider... ==> kubearmor-dev: Importing base box 'ubuntu/bionic64'... ==> kubearmor-dev: Matching MAC address for NAT networking... ==> kubearmor-dev: Checking if box 'ubuntu/bionic64' version '20220131.0.0' is up to date... ==> kubearmor-dev: Setting the name of the VM: kubearmor-dev ==> kubearmor-dev: Clearing any previously set network interfaces... ==> kubearmor-dev: Preparing network interfaces based on configuration... kubearmor-dev: Adapter 1: nat ==> kubearmor-dev: Forwarding ports... kubearmor-dev: 2345 (guest) => 2345 (host) (adapter 1) kubearmor-dev: 22 (guest) => 2222 (host) (adapter 1) ==> kubearmor-dev: Running 'pre-boot' VM customizations... ==> kubearmor-dev: Booting VM... ==> kubearmor-dev: Waiting for machine to boot. This may take a few minutes... kubearmor-dev: SSH address: 127.0.0.1:2222 kubearmor-dev: SSH username: vagrant kubearmor-dev: SSH auth method: private key kubearmor-dev: Warning: Connection reset. Retrying... kubearmor-dev: Warning: Remote connection disconnect. Retrying... kubearmor-dev: kubearmor-dev: Vagrant insecure key detected. Vagrant will automatically replace kubearmor-dev: this with a newly generated keypair for better security. kubearmor-dev: kubearmor-dev: Inserting generated public key within guest... kubearmor-dev: Removing insecure key from the guest if it's present... kubearmor-dev: Key inserted! Disconnecting and reconnecting using new SSH key... ==> kubearmor-dev: Machine booted and ready! ==> kubearmor-dev: Checking for guest additions in VM... kubearmor-dev: The guest additions on this VM do not match the installed version of kubearmor-dev: VirtualBox! In most cases this is fine, but in rare cases it can kubearmor-dev: prevent things such as shared folders from working properly. If you see kubearmor-dev: shared folder errors, please make sure the guest additions within the kubearmor-dev: virtual machine match the version of VirtualBox you have installed on kubearmor-dev: your host and reload your VM. kubearmor-dev: kubearmor-dev: Guest Additions Version: 5.2.42 kubearmor-dev: VirtualBox Version: 6.1 ==> kubearmor-dev: Setting hostname... ==> kubearmor-dev: Mounting shared folders... kubearmor-dev: /vagrant => /home/gourav/KubeArmor/contribution/vagrant kubearmor-dev: /home/vagrant/KubeArmor => /home/gourav/KubeArmor ==> kubearmor-dev: Running provisioner: file... kubearmor-dev: ~/.ssh/idrsa.pub => /home/vagrant/.ssh/idrsa.pub ==> kubearmor-dev: Running provisioner: shell... kubearmor-dev: Running: inline script ==> kubearmor-dev: Running provisioner: file... kubearmor-dev: ~/.gitconfig =>

==> kubearmor-dev: Running provisioner: shell... kubearmor-dev: Running: /tmp/vagrant-shell20220202-55671-bn8u0f.sh ... ``` </details> To get into the vagrant VM ```text ~/KubeArmor/KubeArmor$ make vagrant-ssh ``` Output will show up as ... <details> <summary>Click to expand!</summary> ```text d /home/gourav/KubeArmor/contribution/vagrant; NETNEXT=0 DLV_RPORT=2345 vagrant ssh; true Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-167-generic x86_64) Documentation: https://help.ubuntu.com Management: https://landscape.canonical.com Support: https://ubuntu.com/advantage System information as of Wed Feb 2 10:35:55 UTC 2022 System load: 0.06 Processes: 128 Usage of /: 11.1% of 38.71GB Users logged in: 0 Memory usage: 10% IP address for enp0s3: 10.0.2.15 Swap usage: 0% IP address for docker0: 172.17.0.1 5 updates can be applied immediately. 1 of these updates is a standard security update. To see these additional updates run: apt list --upgradable New release '20.04.3 LTS' available. Run 'do-release-upgrade' to upgrade to it. vagrant@kubearmor-dev:~$ ``` </details> To destroy the vagrant VM ```text ~/KubeArmor/KubeArmor$ make vagrant-destroy ``` VM Setup using Vagrant with Ubuntu 21.10 (v5.13) To use the recent Linux kernel v5.13 for dev env, you can run `make` with the `NETNEXT` flag set to `1` for the respective make option. ```text ~/KubeArmor/KubeArmor$ make vagrant-up NETNEXT=1 ``` You can also make the setting static by changing `NETNEXT=0` to `NETNEXT=1` in the Makefile. ```text ~/KubeArmor/KubeArmor$ vi Makefile ``` Requirements Here is the list of minimum requirements for self-managed Kubernetes. ```text OS - Ubuntu 18.04 Kubernetes - v1.19 Docker - 18.09 or Containerd - 1.3.7 Linux Kernel - v4.15 LSM - AppArmor ``` KubeArmor is designed for Kubernetes environment. If Kubernetes is not setup yet, please refer to . KubeArmor leverages CRI (Container Runtime Interfaces) APIs and works with Docker or Containerd or CRIO based container runtimes. KubeArmor uses LSMs for policy enforcement; thus, please make sure that your environment supports LSMs \(either AppArmor or bpf-lsm\). Otherwise, KubeArmor will operate in Audit-Mode with no policy "enforcement" support. You can try the following alternative if you face any difficulty in the above Kubernetes (kubeadm) setup. > Note Please make sure to set up the alternative k8s environment on the same host where the KubeArmor development environment is running. K3s You can also develop and test KubeArmor on K3s instead of the self-managed Kubernetes. Please follow the instructions in . MicroK8s You can also develop and test KubeArmor on MicroK8s instead of the self-managed Kubernetes. Please follow the instructions in . No Support - Docker Desktops KubeArmor does not work with Docker Desktops on Windows and macOS because KubeArmor integrates with Linux-kernel native primitives (including LSMs). Development Setup In order to install all dependencies, please run the following command. ```text $ cd KubeArmor/contribution/self-managed-k8s ~/KubeArmor/contribution/self-managed-k8s$ ./setup.sh ``` will automatically install , , , and some other dependencies. Now, you are ready to develop any code for

Enjoy your journey with KubeArmor. Compilation Check if KubeArmor can be compiled on your environment without any problems. ```text $ cd KubeArmor/KubeArmor ~/KubeArmor/KubeArmor$ make ``` If you see any error messages, please let us know the issue with the full error messages through KubeArmor's slack. Execution In order to directly run KubeArmor in a host (not as a container), you need to run a local proxy in advance. ```text $ kubectl proxy & ``` Then, run KubeArmor on your environment. ```text $ cd KubeArmor/KubeArmor ~/KubeArmor/KubeArmor$ make run ``` > Note If you have followed all the above steps and still getting the warning `The node information is not available`, then this could be due to the case-sensitivity discrepancy in the actual hostname (obtained by running `hostname`) and the hostname used by Kubernetes (under `kubectl get nodes -o wide`).<br> K8s converts the hostname to lowercase, which results in a mismatch with the actual hostname.<br> To resolve this, change the hostname to lowercase using the command `hostnamectl set-hostname <lowercase-hostname>`. KubeArmor Controller Starting from KubeArmor v0.11 - annotations, container policies, and host policies are handled via kubearmor controller, the controller code can be found under `pkg/KubeArmorController`. To install the controller from KubeArmor docker repository run ```text $ cd KubeArmor/pkg/KubeArmorController ~/KubeArmor/pkg/KubeArmorController$ make deploy ``` To install the controller (local version) to your cluster run ```text $ cd KubeArmor/pkg/KubeArmorController ~/KubeArmor/pkg/KubeArmorController$ make docker-build deploy ``` if you need to setup a local registry to push you image, use `docker-registry.sh` script under `~/KubeArmor/contribution/local-registry` directory Here, we briefly give you an overview of KubeArmor's directories. Source code for KubeArmor \(/KubeArmor\) ```text KubeArmor/ BPF - eBPF code for system monitor common - Libraries internally used config - Configuration loader core - The main body (start point) of KubeArmor enforcer - Runtime policy enforcer (enforcing security policies into LSMs) feeder - gRPC-based feeder (sending audit/system logs to a log server) kvmAgent - KubeArmor VM agent log - Message logger (stdout) monitor - eBPF-based system monitor (mapping process IDs to container IDs) policy - gRPC service to manage Host Policies for VM environments types - Type definitions protobuf/ - Protocol buffer ``` Source code for KubeArmor Controller \(CRD\) ```text pkg/KubeArmorController/ - KubeArmorController generated by Kube-Builder for KubeArmor Annotations, KubeArmorPolicy and KubeArmorHostPolicy ``` Deployment tools and files for KubeArmor ```text deployments/ <cloud-platform-name> - Deployments specific to respective cloud platform (deprecated - use karmor install or helm) controller - Deployments for installing KubeArmorController alongwith cert-manager CRD - KubeArmorPollicy and KubeArmorHostPolicy CRDs get - Stores source code for deploygen, a tool used for specifying kubearmor deployments helm/ KubeArmor - KubeArmor's Helm chart KubeArmorOperator - KubeArmorOperator's Helm chart ``` Files for testing ```text examples/ - Example microservices for testing tests/ - Automated test framework for KubeArmor ```