Repository: kubernetes-sigs/security-profiles-operator
Pull request #170 (2020-12-14 14:53:23)

Enable e2e tests to run on OpenShift
Enable e2e tests to run on OpenShift

This changes the e2e tests to not only run on a kind cluster, but to
also be able to run on a running OpenShift cluster.

To enable this, there is now the `E2E_CLUTER_TYPE` environment variable
which the Makefile can take into use and that will be taken into account
by the test itself.

Note that in order for the security-profiles-operator to work on
OpenShift, I temporarily added spc_t as the SELinux type. This ensures
that the operator is allowed to place files on the folder it's trying to
access. This is a placeholder while we work on a better mechanism to
consume both workload and bootstrap SELinux policies [1]. This mechanism
will eventually replace the tiny workloads that we currently spawn to
install policies and will be included as part of the main daemonset.

[1] https://github.com/JAORMX/selinuxd

#### What this PR does / why we need it:

This allows us to run the e2e tests on an OpenShift cluster. We can
subsequently enable the OpenShift infrastructure to run tests here and thus get
coverage there as well.

```release-note
NONE
```

k8s-ci-robot: Hi @JAORMX. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/170#" title="Author self-approved">JAORMX</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saschagrunert** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saschagrunert` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saschagrunert"]} -->
pjbgf: /ok-to-test
k8s-ci-robot: @JAORMX: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-verify/1338581902323355648) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=170). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AJAORMX). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @JAORMX: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-verify/1338581902323355648) | `/test pull-security-profiles-operator-verify`
pull-security-profiles-operator-test-e2e | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-test-e2e/1338581902382075904) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=170). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AJAORMX). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
pjbgf: /retest
k8s-ci-robot: @JAORMX: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-test-e2e/1338581902382075904) | `/test pull-security-profiles-operator-test-e2e`
pull-security-profiles-operator-verify | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-verify/1338583002770313216) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=170). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AJAORMX). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @JAORMX: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-verify/1338583002770313216) | `/test pull-security-profiles-operator-verify`
pull-security-profiles-operator-test-e2e | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-test-e2e/1338583002795479040) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=170). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AJAORMX). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
JAORMX: Pull-request updated, HEAD is now b2c6d2078496917118efae33c5c8e8484a34b627
k8s-ci-robot: @JAORMX: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 5d7c0b6c5716692468e5660d2b890a3d2cea3368 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-test-e2e/1338583002795479040) | `/test pull-security-profiles-operator-test-e2e`
pull-security-profiles-operator-verify | b2c6d2078496917118efae33c5c8e8484a34b627 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/170/pull-security-profiles-operator-verify/1338729745499557888) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=170). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AJAORMX). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
JAORMX: Pull-request updated, HEAD is now 20624798d834a1177f66c9538d802a18cbdb2019
JAORMX: Pull-request updated, HEAD is now ee54383746fe5429a01bd7a9a083ea82f1dc2ee2
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/170#" title="Author self-approved">JAORMX</a>*, *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/170#pullrequestreview-552374341" title="LGTM">pjbgf</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)~~ [pjbgf]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/Makefile b/Makefile
index 9ef260129..9ce3d3cc6 100644
--- a/Makefile
+++ b/Makefile
@@ -43,7 +43,7 @@ else
   LDFLAGS := -s -w -extldflags "-static" $(LDVARS)
 endif
 
-CONTAINER_RUNTIME ?= docker
+export CONTAINER_RUNTIME ?= docker
 IMAGE ?= $(PROJECT):latest
 
 CRD_OPTIONS ?= "crd:crdVersions=v1"
@@ -51,6 +51,8 @@ CRD_OPTIONS ?= "crd:crdVersions=v1"
 GOLANGCI_LINT_VERSION = v1.32.0
 REPO_INFRA_VERSION = v0.1.2
 
+export E2E_CLUSTER_TYPE ?= kind
+
 # Utility targets
 
 all: $(BUILD_DIR)/$(PROJECT) ## Build the security-profiles-operator binary
diff --git a/deploy/base/operator.yaml b/deploy/base/operator.yaml
index a72f16366..4641cf87c 100644
--- a/deploy/base/operator.yaml
+++ b/deploy/base/operator.yaml
@@ -88,6 +88,9 @@ spec:
             runAsGroup: 2000
             capabilities:
               drop: ["ALL"]
+            seLinuxOptions:
+              # FIXME(jaosorior): Use a more restricted selinux type
+              type: spc_t
           resources:
             requests:
               memory: "64Mi"
diff --git a/deploy/namespace-operator.yaml b/deploy/namespace-operator.yaml
index c4db2c841..9753afdc1 100644
--- a/deploy/namespace-operator.yaml
+++ b/deploy/namespace-operator.yaml
@@ -481,6 +481,8 @@ spec:
           readOnlyRootFilesystem: true
           runAsGroup: 2000
           runAsUser: 2000
+          seLinuxOptions:
+            type: spc_t
         volumeMounts:
         - mountPath: /var/lib/kubelet/seccomp/operator
           name: host-operator-volume
diff --git a/deploy/operator.yaml b/deploy/operator.yaml
index 264a55f9b..2a4f57ed1 100644
--- a/deploy/operator.yaml
+++ b/deploy/operator.yaml
@@ -479,6 +479,8 @@ spec:
           readOnlyRootFilesystem: true
           runAsGroup: 2000
           runAsUser: 2000
+          seLinuxOptions:
+            type: spc_t
         volumeMounts:
         - mountPath: /var/lib/kubelet/seccomp/operator
           name: host-operator-volume


Test patch:
diff --git a/test/e2e_test.go b/test/e2e_test.go
index 62142a9fc..d1ea173b2 100644
--- a/test/e2e_test.go
+++ b/test/e2e_test.go
@@ -109,13 +109,14 @@ func (e *e2e) deployOperator(manifest, profiles string) {
 	// ones from the nodes
 	e.logf("Setting imagePullPolicy to 'Never' in manifest: %s", manifest)
 	e.run(
-		"sed", "-i", "s;imagePullPolicy: Always;imagePullPolicy: Never;g",
+		"sed", "-i",
+		fmt.Sprintf("s;imagePullPolicy: Always;imagePullPolicy: %s;g", e.pullPolicy),
 		manifest,
 	)
 
 	// Update the image name to match the test image
 	e.run(
-		"sed", "-i", fmt.Sprintf("s;image: .*gcr.io/.*;image: %s;g", testImage),
+		"sed", "-i", fmt.Sprintf("s;image: .*gcr.io/.*;image: %s;g", e.testImage),
 		manifest,
 	)
 
diff --git a/test/suite_test.go b/test/suite_test.go
index d97608d30..af2a8a6bb 100644
--- a/test/suite_test.go
+++ b/test/suite_test.go
@@ -21,6 +21,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 
@@ -37,25 +38,64 @@ const (
 	kindVersion = "v0.9.0"
 	kindSHA512  = "e7152acf5fd7a4a56af825bda64b1b8343a1f91588f9b3ddd5420ae5c5a95577d87431f2e417a7e03dd23914e1da9bed855ec19d0c4602729b311baccb30bd7f" // nolint: lll
 	kindImage   = "kindest/node:v1.19.1"
-	testImage   = config.OperatorName + ":latest"
+)
+
+var (
+	clusterType      = os.Getenv("E2E_CLUSTER_TYPE")
+	containerRuntime = os.Getenv("CONTAINER_RUNTIME")
 )
 
 type e2e struct {
 	suite.Suite
-	kindPath    string
 	kubectlPath string
-	clusterName string
+	testImage   string
+	pullPolicy  string
 	logger      logr.Logger
+	execNode    func(node string, args ...string) string
+}
+
+type kinde2e struct {
+	e2e
+	kindPath    string
+	clusterName string
+}
+
+type openShifte2e struct {
+	e2e
 }
 
 func TestSuite(t *testing.T) {
-	suite.Run(t, &e2e{logger: klogr.New()})
+	fmt.Printf("cluster-type: %s\n", clusterType)
+	switch {
+	case clusterType == "" || strings.EqualFold(clusterType, "kind"):
+		suite.Run(t, &kinde2e{
+			e2e{
+				logger:     klogr.New(),
+				testImage:  config.OperatorName + ":latest",
+				pullPolicy: "Never",
+			},
+			"", "",
+		})
+	case strings.EqualFold(clusterType, "openshift"):
+		suite.Run(t, &openShifte2e{
+			e2e{
+				logger: klogr.New(),
+				// Need to pull the image as it'll be uploaded to the cluster OCP
+				// image registry and not on the nodes.
+				pullPolicy: "Always",
+			},
+		})
+	default:
+		t.Fatalf("Unknown cluster type.")
+	}
 }
 
 // SetupSuite downloads kind and searches for kubectl in $PATH.
-func (e *e2e) SetupSuite() {
+func (e *kinde2e) SetupSuite() {
 	e.logf("Setting up suite")
 	command.SetGlobalVerbose(true)
+	// Override execNode function
+	e.e2e.execNode = e.execNodeKind
 	cwd, err := os.Getwd()
 	e.Nil(err)
 
@@ -76,7 +116,7 @@ func (e *e2e) SetupSuite() {
 }
 
 // SetupTest starts a fresh kind cluster for each test.
-func (e *e2e) SetupTest() {
+func (e *kinde2e) SetupTest() {
 	// Deploy the cluster
 	e.logf("Deploying the cluster")
 	e.clusterName = fmt.Sprintf("spo-e2e-%d", time.Now().Unix())
@@ -98,15 +138,15 @@ func (e *e2e) SetupTest() {
 
 	// Build and load the test image
 	e.logf("Building operator container image")
-	e.run("make", "image", "IMAGE="+testImage)
+	e.run("make", "image", "IMAGE="+e.testImage)
 	e.logf("Loading container image into nodes")
 	e.run(
-		e.kindPath, "load", "docker-image", "--name="+e.clusterName, testImage,
+		e.kindPath, "load", "docker-image", "--name="+e.clusterName, e.testImage,
 	)
 }
 
 // TearDownTest stops the kind cluster.
-func (e *e2e) TearDownTest() {
+func (e *kinde2e) TearDownTest() {
 	e.logf("Destroying cluster")
 	e.run(
 		e.kindPath, "delete", "cluster",
@@ -115,6 +155,93 @@ func (e *e2e) TearDownTest() {
 	)
 }
 
+func (e *kinde2e) execNodeKind(node string, args ...string) string {
+	return e.run(containerRuntime, append([]string{"exec", node}, args...)...)
+}
+
+func (e *openShifte2e) SetupSuite() {
+	e.logf("Setting up suite")
+	command.SetGlobalVerbose(true)
+	// Override execNode function
+	e.e2e.execNode = e.execNodeOCP
+	cwd, err := os.Getwd()
+	e.Nil(err)
+
+	parentCwd := filepath.Dir(cwd)
+	e.Nil(os.Chdir(parentCwd))
+	buildDir := filepath.Join(parentCwd, "build")
+	e.Nil(os.MkdirAll(buildDir, 0o755))
+
+	e.kubectlPath, err = exec.LookPath("oc")
+	e.Nil(err)
+
+	e.logf("Using deployed OpenShift cluster")
+
+	e.logf("Waiting for cluster to be ready")
+	e.kubectl("wait", "--for", "condition=ready", "nodes", "--all")
+
+	e.logf("Exposing registry")
+	e.kubectl("patch", "configs.imageregistry.operator.openshift.io/cluster",
+		"--patch", "{\"spec\":{\"defaultRoute\":true}}", "--type=merge")
+	defer e.kubectl("patch", "configs.imageregistry.operator.openshift.io/cluster",
+		"--patch", "{\"spec\":{\"defaultRoute\":false}}", "--type=merge")
+
+	testImageRef := config.OperatorName + ":latest"
+
+	// Build and load the test image
+	e.logf("Building operator container image")
+	e.run("make", "image", "IMAGE="+testImageRef)
+	e.logf("Loading container image into nodes")
+
+	// Get credentials
+	user := e.kubectl("whoami")
+	token := e.kubectl("whoami", "-t")
+	registry := e.kubectl(
+		"get", "route", "default-route", "-n", "openshift-image-registry",
+		"--template={{ .spec.host }}",
+	)
+
+	e.run(
+		containerRuntime, "login", "--tls-verify=false", "-u", user, "-p", token, registry,
+	)
+
+	registryTarget := fmt.Sprintf("%s/openshift/%s", registry, testImageRef)
+	e.run(
+		containerRuntime, "push", "--tls-verify=false", testImageRef, registryTarget,
+	)
+	// Enable "local" lookup without full path
+	e.kubectl("patch", "imagestream", "-n", "openshift", config.OperatorName,
+		"--patch", "{\"spec\":{\"lookupPolicy\":{\"local\":true}}}", "--type=merge")
+
+	e.testImage = e.kubectl(
+		"get", "imagestreamtag", "-n", "openshift", testImageRef, "-o", "jsonpath={.image.dockerImageReference}",
+	)
+}
+
+func (e *openShifte2e) TearDownSuite() {
+	testImageRef := config.OperatorName + ":latest"
+	e.kubectl(
+		"delete", "imagestream", "-n", "openshift", testImageRef,
+	)
+}
+
+func (e *openShifte2e) SetupTest() {
+	e.logf("Setting up test")
+}
+
+// TearDownTest stops the kind cluster.
+func (e *openShifte2e) TearDownTest() {
+	e.logf("Tearing down test")
+}
+
+func (e *openShifte2e) execNodeOCP(node string, args ...string) string {
+	return e.kubectl(
+		"debug", "-q", fmt.Sprintf("node/%s", node), "--",
+		"chroot", "/host", "/bin/bash", "-c",
+		strings.Join(args, " "),
+	)
+}
+
 func (e *e2e) run(cmd string, args ...string) string {
 	output, err := command.New(cmd, args...).RunSuccessOutput()
 	e.Nil(err)
@@ -161,7 +288,3 @@ func (e *e2e) kubectlOperatorNS(args ...string) string {
 func (e *e2e) logf(format string, a ...interface{}) {
 	e.logger.Info(fmt.Sprintf(format, a...))
 }
-
-func (e *e2e) execNode(node string, args ...string) string {
-	return e.run("docker", append([]string{"exec", node}, args...)...)
-}