Repository: kubernetes-sigs/security-profiles-operator
Pull request #283 (2021-02-09 10:29:36)

Add basic Vagrant test job
#### What type of PR is this?

/kind design

#### What this PR does / why we need it:

This basic test stub will be later on used for VM based e2e testing.
#### Which issue(s) this PR fixes:

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.

Fixes #

or

None
-->
None
#### Does this PR have test?

<!--
If tests aren't applicable just write N/A.
-->
None
#### Special notes for your reviewer:
/hold for testing
#### Does this PR introduce a user-facing change?

<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
None
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/283#" title="Author self-approved">saschagrunert</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)~~ [saschagrunert]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
saschagrunert: > Seems to work fine, I just let two minor questions in the PR.

I think I can include a basic CRI-O and kubernetes setup within this PR.
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/283#pullrequestreview-586531784" title="Approved">JAORMX</a>*, *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/283#" title="Author self-approved">saschagrunert</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)~~ [saschagrunert]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | c2d078d0d6cd3cdf516f9894ecc754f3db618896 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/283/pull-security-profiles-operator-verify/1359132312528228352) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=283). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @saschagrunert: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 9a1709a3f9aa65a5c18ccf6ab58a090bb7940467 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/283/pull-security-profiles-operator-verify/1359141277441986560) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=283). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Asaschagrunert). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
saschagrunert: Bootstrapping the cluster via `hack/local-up-cluster.sh` already takes 18min, we may need to use some pre-built binaries, maybe them directly from [yum.k8s.io](https://yum.k8s.io).
saschagrunert: Cluster bootstrap takes now ~5minutes with downloading the binaries. The output of the e2e GitHub action job seems also fine:

```
2021-02-10T09:27:01.4993610Z NAME        STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION           CONTAINER-RUNTIME
2021-02-10T09:27:01.5051080Z 127.0.0.1   Ready    <none>   44s   v1.20.2   127.0.0.1     <none>        Fedora 33 (Cloud Edition)   5.8.15-301.fc33.x86_64   cri-o://1.20.0
```

I think we now can fill the job with test logic.

/hold cancel

PTAL @pjbgf @JAORMX @jhrozek 
JAORMX: /lgtm

Patch:
diff --git a/.gitignore b/.gitignore
index 243a51aa9..d8073b22c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,6 @@
 *.orig
 *.swp
-build
 .idea
+.vagrant
+Vagrantfile
+build
diff --git a/hack/ci/Vagrantfile b/hack/ci/Vagrantfile
new file mode 100644
index 000000000..04f731d12
--- /dev/null
+++ b/hack/ci/Vagrantfile
@@ -0,0 +1,47 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Vagrant box for testing
+Vagrant.configure("2") do |config|
+  config.vm.box = "fedora/33-cloud-base"
+  memory = 4096
+  cpus = 4
+
+  config.vm.provider :virtualbox do |v|
+    v.memory = memory
+    v.cpus = cpus
+  end
+
+  config.vm.provider :libvirt do |v|
+    v.memory = memory
+    v.cpus = cpus
+  end
+
+  config.vm.provision "install-dependencies", type: "shell", run: "once" do |sh|
+    sh.inline = <<~SHELL
+    whoami
+      set -euxo pipefail
+      dnf install -y \
+        conntrack \
+        container-selinux \
+        golang-go \
+        iptables \
+        make \
+        openssl
+    SHELL
+  end
+
+  config.vm.provision "install-cri-o", type: "shell", run: "once" do |sh|
+    sh.inline = <<~SHELL
+      set -euxo pipefail
+      sudo /vagrant/hack/ci/install-cri-o.sh
+    SHELL
+  end
+
+  config.vm.provision "install-kubernetes", type: "shell", run: "once" do |sh|
+    sh.inline = <<~SHELL
+      set -euxo pipefail
+      sudo /vagrant/hack/ci/install-kubernetes.sh
+    SHELL
+  end
+end
diff --git a/hack/ci/env.sh b/hack/ci/env.sh
new file mode 100755
index 000000000..e2250690f
--- /dev/null
+++ b/hack/ci/env.sh
@@ -0,0 +1,29 @@
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+local_ip() {
+    ip route get 1.2.3.4 | cut -d ' ' -f7 | tr -d '[:space:]'
+}
+
+GOPATH=$(go env GOPATH)
+
+export KUBE_CONTAINER_RUNTIME=remote
+export KUBERUN=/var/run/kubernetes
+export KUBECONFIG=$KUBERUN/admin.kubeconfig
+export PATH="$GOPATH/src/k8s.io/kubernetes/_output/bin:$PATH"
+
+IP=$(local_ip)
+export KUBE_MASTER_URL=$IP
+export KUBE_MASTER_IP=$IP
+export KUBE_MASTER=$IP
diff --git a/hack/ci/install-cri-o.sh b/hack/ci/install-cri-o.sh
new file mode 100755
index 000000000..0bff6328c
--- /dev/null
+++ b/hack/ci/install-cri-o.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+
+cd "$(mktemp -d)"
+
+VERSION=v1.20.0
+URL=https://storage.googleapis.com/k8s-conform-cri-o/artifacts/crio-$VERSION.tar.gz
+
+curl -sfL --retry 5 --retry-delay 3 --show-error \
+    -o crio.tar.gz \
+    $URL
+
+tar xfvz crio.tar.gz
+mv crio-* crio
+
+make -C crio
+
+chcon -u system_u -r object_r -t container_runtime_exec_t \
+    /usr/local/bin/crio \
+    /usr/local/bin/crio-status \
+    /usr/local/bin/runc \
+    /usr/local/bin/crun
+
+chcon -u system_u -r object_r -t bin_t \
+    /usr/local/bin/conmon \
+    /usr/local/bin/crictl \
+    /usr/local/bin/pinns
+
+chcon -R -u system_u -r object_r -t bin_t /opt/cni/bin
+
+mkdir -p /var/lib/kubelet
+chcon -R -u system_u -r object_r -t var_lib_t /var/lib/kubelet
+
+mkdir -p /etc/crio/crio.conf.d
+
+cat <<EOT >>/etc/crio/crio.conf.d/20-crun.conf
+[crio.runtime]
+default_runtime = "crun"
+
+[crio.runtime.runtimes.crun]
+runtime_path = "/usr/local/bin/crun"
+EOT
+
+cat <<EOT >>/etc/crio/crio.conf.d/30-cgroup-manager.conf
+[crio.runtime]
+conmon_cgroup = "pod"
+cgroup_manager = "cgroupfs"
+EOT
+
+chcon -R -u system_u -r object_r -t container_config_t \
+    /etc/crio \
+    /etc/crio/crio.conf \
+    /usr/local/share/oci-umount/oci-umount.d/crio-umount.conf
+
+systemctl enable crio.service
+restorecon /usr/local/lib/systemd/system/crio.service
+systemctl start crio.service
diff --git a/hack/ci/install-kubernetes.sh b/hack/ci/install-kubernetes.sh
new file mode 100755
index 000000000..0ea44966b
--- /dev/null
+++ b/hack/ci/install-kubernetes.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+
+ENVFILE=$(dirname "${BASH_SOURCE[0]}")/env.sh
+. "$ENVFILE"
+
+K8SPATH="$GOPATH/src/k8s.io"
+VERSION=v1.20.2
+
+download-kubernetes() {
+    export KUBERNETES_RELEASE=$VERSION
+    export KUBERNETES_SKIP_CONFIRM=1
+    export KUBERNETES_SKIP_CREATE_CLUSTER=1
+    cluster/get-kube.sh
+
+    mkdir -p _output/bin
+    tar xfz kubernetes/server/kubernetes-server-linux-amd64.tar.gz
+    cp kubernetes/server/bin/{kubectl,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubelet} \
+        _output/bin
+}
+
+local-up() {
+    export PATH="$GOPATH/src/k8s.io/kubernetes/third_party/etcd:$PATH"
+    export CONTAINER_RUNTIME=remote
+    export CGROUP_DRIVER=cgroupfs
+    export CONTAINER_RUNTIME_ENDPOINT=/var/run/crio/crio.sock
+    export ALLOW_PRIVILEGED=1
+
+    echo "Using IP: $IP"
+    export DNS_SERVER_IP=$IP
+    export API_HOST_IP=$IP
+
+    iptables -F
+    download-kubernetes
+    hack/local-up-cluster.sh -O
+}
+
+mkdir -p "$K8SPATH"
+cd "$K8SPATH"
+
+TARBALL=k8s.tar.gz
+curl -sfL --retry 5 --retry-delay 3 --show-error \
+    -o $TARBALL https://github.com/kubernetes/kubernetes/tarball/$VERSION
+tar xfz $TARBALL
+rm $TARBALL
+mv kubernetes-kubernetes-* kubernetes
+cd kubernetes
+
+hack/install-etcd.sh
+
+OUTPUT=$(mktemp)
+local-up 2>&1 | tee "$OUTPUT" &
+PID=$!
+
+echo Waiting for hack/local-up-cluster.sh
+until grep -q "Local Kubernetes cluster is running" "$OUTPUT"; do
+    if ! ps $PID >/dev/null; then
+        exit 1
+    fi
+    sleep 1
+done
+
+echo Cluster is up and running
diff --git a/hack/ci/run.sh b/hack/ci/run.sh
new file mode 100755
index 000000000..2b246a7ae
--- /dev/null
+++ b/hack/ci/run.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+
+exec vagrant ssh -- sudo "bash -c '. /vagrant/hack/ci/env.sh && $*'"


Test patch:
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
new file mode 100644
index 000000000..332166ae1
--- /dev/null
+++ b/.github/workflows/e2e.yml
@@ -0,0 +1,21 @@
+name: e2e
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+jobs:
+  test:
+    runs-on: macos-10.15
+    timeout-minutes: 30
+    env:
+      RUN: ./hack/ci/run.sh
+    steps:
+      - uses: actions/checkout@v2
+      - name: Boot Virtual Machine
+        run: |
+          ln -sf hack/ci/Vagrantfile .
+          vagrant up
+      - name: Show environment information
+        run: |
+          $RUN kubectl get nodes -o wide