Project: Capsule (Orchestration & Management / Scheduling & Orchestration)
File: DEVELOPMENT.md

Make sure you have these tools installed: , or or , with `kubectl` (if you want to run locally with remote Kubernetes) OpenSSL A lightweight Kubernetes within your laptop can be very handy for Kubernetes-native development like Capsule. ```shell $ brew install k3d $ export LAPTOPHOSTIP=192.168.10.101 $ k3d cluster create k3s-capsule --servers 1 --agents 1 --no-lb --k3s-server-arg --tls-san=${LAPTOPHOSTIP} $ k3d kubeconfig get k3s-capsule > /tmp/k3s-capsule && export KUBECONFIG="/tmp/k3s-capsule" $ kubectl get nodes NAME STATUS ROLES AGE VERSION k3d-k3s-capsule-server-0 Ready control-plane,master 2m13s v1.21.2+k3s1 k3d-k3s-capsule-agent-0 Ready <none> 2m3s v1.21.2+k3s1 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c26ad840c62 rancher/k3s:v1.21.2-k3s1 "/bin/k3s agent" 53 seconds ago Up 45 seconds k3d-k3s-capsule-agent-0 753998879b28 rancher/k3s:v1.21.2-k3s1 "/bin/k3s server --t" 53 seconds ago Up 51 seconds 0.0.0.0:49708->6443/tcp k3d-k3s-capsule-server-0 ``` ```shell $ brew install kind $ cat > kind.yaml <<EOF kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: apiServerAddress: "0.0.0.0" nodes: role: control-plane kubeadmConfigPatches: | kind: ClusterConfiguration metadata: name: config apiServer: certSANs: localhost 127.0.0.1 kubernetes kubernetes.default.svc kubernetes.default.svc.cluster.local kind 0.0.0.0 ${LAPTOPHOSTIP} role: worker EOF $ kind create cluster --name kind-capsule --config kind.yaml $ kubectl get nodes NAME STATUS ROLES AGE VERSION kind-capsule-control-plane Ready control-plane,master 84s v1.21.1 kind-capsule-worker Ready <none> 56s v1.21.1 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b329fd3a838 kindest/node:v1.21.1 "/usr/local/bin/entr" About a minute ago Up About a minute 0.0.0.0:54894->6443/tcp kind-capsule-control-plane 7d50f1633555 kindest/node:v1.21.1 "/usr/local/bin/entr" About a minute ago Up About a minute kind-capsule-worker ``` The `fork-clone-contribute-pr` flow is common for contributing to OSS projects like Kubernetes and Capsule. Let's assume you've forked it into your GitHub namespace, say `myuser`, and then you can clone it with Git protocol. Do remember to change the `myuser` to yours. ```shell $ git clone git@github.com:myuser/capsule.git && cd capsule ``` It's a good practice to add the upstream as the remote too so we can easily fetch and merge the upstream to our fork: ```shell $ git remote add upstream https://github.com/projectcapsule/capsule.git $ git remote -vv origin git@github.com:myuser/capsule.git (fetch) origin git@github.com:myuser/capsule.git (push) upstream https://github.com/projectcapsule/capsule.git (fetch) upstream https://github.com/projectcapsule/capsule.git (push) ``` Pull all tags ``` $ git fetch --all && git pull upstream ``` Build and deploy: ```shell $ go mod download $ make docker-build $ export CAPSULEIMAGEVESION=`docker images --format '{{.Tag}}' clastix/capsule` $ k3d image import --cluster k3s-capsule capsule clastix/capsule:${CAPSULEIMAGEVESION} $ kind load docker-image --name kind-capsule clastix/capsule:${CAPSULEIMAGEVESION} $ make deploy $ kubectl get pod -n capsule-system NAME READY STATUS RESTARTS AGE capsule-controller-manager-5c6b8445cf-566dc 1/1 Running 0 23s $ kubectl -n capsule-system logs --all-containers -l control-plane=controller-manager $ kubectl apply -f - <<EOF apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: oil spec: owners: name: alice kind: User name: system:serviceaccount:capsule-system:default kind: ServiceAccount EOF $ kubectl get tenants NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE oil Active 0 14s ``` If you want to test namespace creation or such stuff, make sure to use impersonation: ```sh $ kubectl ... --as system:serviceaccount:capsule-system:default --as-group

``` As of now, a complete Capsule environment has been set up in `kind`- or `k3d`-powered cluster, and the `capsule-controller-manager` is running as a deployment serving as: The reconcilers for CRDs and; A series of webhooks During development, we prefer that the code is running within our IDE locally, instead of running as the normal Pod(s) within the Kubernetes cluster. Such a setup can be illustrated as below diagram: To achieve that, there are some necessary steps we need to walk through, which have been made as a `make` target within our `Makefile`. So the TL;DR answer is: ```shell $ make deploy $ LAPTOPHOSTIP="<YOURLAPTOPIP>" make dev-setup ``` This is a very common setup for typical Kubernetes Operator development so we'd better walk them through with more details here. Scaling down the deployed Pod(s) to 0 We need to scale the existing replicas of `capsule-controller-manager` to 0 to avoid reconciliation competition between the Pod(s) and the code running outside of the cluster, in our preferred IDE for example. ```shell $ kubectl -n capsule-system scale deployment capsule-controller-manager --replicas=0 deployment.apps/capsule-controller-manager scaled ``` Preparing TLS certificate for the webhooks Running webhooks requires TLS, we can prepare the TLS key pair in our development env to handle HTTPS requests. ```shell $ cat > _tls.cnf <<EOF [ req ] default_bits = 4096 distinguishedname = reqdistinguished_name reqextensions = reqext [ reqdistinguishedname ] countryName = SG stateOrProvinceName = SG localityName = SG organizationName = CAPSULE commonName = CAPSULE [ req_ext ] subjectAltName = @alt_names [alt_names] IP.1 = ${LAPTOPHOSTIP} EOF $ mkdir -p /tmp/k8s-webhook-server/serving-certs $ openssl req -newkey rsa:4096 -days 3650 -nodes -x509 \ -subj "/C=SG/ST=SG/L=SG/O=CAPSULE/CN=CAPSULE" \ -extensions req_ext \ -config _tls.cnf \ -keyout /tmp/k8s-webhook-server/serving-certs/tls.key \ -out /tmp/k8s-webhook-server/serving-certs/tls.crt $ rm -f _tls.cnf ``` Patching the Webhooks By default, the webhooks will be registered with the services, which will route to the Pods, inside the cluster. We need to delegate the controllers' and webhooks' services to the code running in our IDE by patching the `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`. ```shell $ export WEBHOOKURL="https://${LAPTOPHOST_IP}:9443" $ export CA_BUNDLE=`openssl base64 -in /tmp/k8s-webhook-server/serving-certs/tls.crt | tr -d '\n'` $ kubectl patch MutatingWebhookConfiguration capsule-mutating-webhook-configuration \ --type='json' -p="[\ {'op': 'replace', 'path': '/webhooks/0/clientConfig', 'value':{'url':\"${WEBHOOKURL}/mutate-v1-namespace-owner-reference\",'caBundle':\"${CABUNDLE}\"}}\ ]" $ kubectl get MutatingWebhookConfiguration capsule-mutating-webhook-configuration -o yaml $ kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \ --type='json' -p="[\ {'op': 'replace', 'path': '/webhooks/0/clientConfig', 'value':{'url':\"${WEBHOOKURL}/cordoning\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/1/clientConfig', 'value':{'url':\"${WEBHOOKURL}/ingresses\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/2/clientConfig', 'value':{'url':\"${WEBHOOKURL}/namespaces\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/3/clientConfig', 'value':{'url':\"${WEBHOOKURL}/networkpolicies\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/4/clientConfig', 'value':{'url':\"${WEBHOOKURL}/pods\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/5/clientConfig', 'value':{'url':\"${WEBHOOKURL}/persistentvolumeclaims\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/6/clientConfig', 'value':{'url':\"${WEBHOOKURL}/services\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/7/clientConfig', 'value':{'url':\"${WEBHOOKURL}/tenants\",'caBundle':\"${CABUNDLE}\"}},\ {'op': 'replace', 'path': '/webhooks/8/clientConfig', 'value':{'url':\"${WEBHOOKURL}/nodes\",'caBundle':\"${CABUNDLE}\"}}\ ]" $ kubectl get ValidatingWebhookConfiguration capsule-validating-webhook-configuration -o yaml ``` Now we can run Capsule controllers with webhooks outside of the Kubernetes cluster: ```shell $ export NAMESPACE=capsule-system && export TMPDIR=/tmp/ $ go run . ``` To verify that, we can open a new console and create a new Tenant: ```shell $ kubectl apply -f - <<EOF apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: gas spec: owners: name: alice kind: User EOF ``` We should see output and logs in the `make run` console. Now it's time to work through our familiar inner loop for development in our preferred IDE. For example, if you're using , this `launch.json` file can be a good start. ```json { "version": "0.2.0", "configurations": [ { "name": "Launch", "type": "go", "request": "launch", "mode": "auto", "program": "${workspaceFolder}", "args": [ "--zap-encoder=console", "--zap-log-level=debug", "--configuration-name=capsule-default" ], "env": { "NAMESPACE": "capsule-system", "TMPDIR": "/tmp/" } } ] } ```