Repository: kubernetes-sigs/security-profiles-operator
Pull request #1070 (2022-08-04 08:08:13)

Two minor test deflakes

<!--  Thanks for sending a pull request!  Here are some tips for you:

- If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
- Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
- If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
- If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind flake

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
- tests: Clean up selinuxprofiles when tests end, too - avoids the test hanging in case a selinux profile is not cleaned up correctly for one reason or another
- tests: Wait for profile name to be reconciled before creating pod - avoids a race in case a profile takes a little while to reconcile and the pod would have been created too fast

#### Which issue(s) this PR fixes:
None

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.

Fixes #

or

None
-->

#### Does this PR have test?
N/A

<!--
If tests aren't applicable just write N/A.
-->

#### Special notes for your reviewer:
N/A

#### Does this PR introduce a user-facing change?

<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
NONE
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/1070#" title="Author self-approved">jhrozek</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/OWNERS)~~ [jhrozek]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @jhrozek: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 9b73b021ef0caffc94d4b74ab556c38ffae3916c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/1070/pull-security-profiles-operator-test-e2e/1555103301349937152) | true | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=1070). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/.github/workflows/helm-chart-package.yaml b/.github/workflows/helm-chart-package.yaml
new file mode 100644
index 000000000..ca4b95a0d
--- /dev/null
+++ b/.github/workflows/helm-chart-package.yaml
@@ -0,0 +1,53 @@
+name: helm-chart-package
+on:
+  pull_request:
+  release:
+    types: [published]
+permissions:
+  actions: none
+  checks: none
+  contents: write
+  deployments: none
+  id-token: none
+  issues: none
+  discussions: none
+  packages: none
+  pull-requests: none
+  repository-projects: none
+  security-events: none
+  statuses: none
+jobs:
+  helm-package:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - name: Set up Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v3.8.1
+      - run: mkdir -p build
+      - id: package
+        run: |
+          helm_output="$(helm package -d build deploy/helm)"
+          tgz_path="${helm_output##*saved it to: }"
+          echo "helm chart tgz path: '${tgz_path}'"
+          [ ! -f "${tgz_path}" ] && echo "failed to find helm chart from 'helm package' stdout" && exit 1
+          echo "::set-output name=helm_tgz_path::${tgz_path}"
+          echo "::set-output name=helm_tgz_name::${tgz_path##*/}"
+      - uses: actions/upload-artifact@v3
+        with:
+          name: ${{ steps.package.outputs.helm_tgz_name}}
+          path: ${{ steps.package.outputs.helm_tgz_path}}
+          if-no-files-found: error
+      - uses: actions/upload-release-asset@v1
+        if: github.event_name == 'release'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          upload_url: ${{ github.event.release.upload_url }}
+          asset_path: ${{ steps.package.outputs.helm_tgz_path}}
+          asset_name: ${{ steps.package.outputs.helm_tgz_name}}
+          asset_content_type: 'application/x-tar'


Test patch:
diff --git a/test/e2e_test.go b/test/e2e_test.go
index 31a89977f..180eb1966 100644
--- a/test/e2e_test.go
+++ b/test/e2e_test.go
@@ -335,6 +335,7 @@ func (e *e2e) cleanupOperator(manifest string) {
 	// Clean up the operator
 	e.logf("Cleaning up operator")
 	e.kubectl("delete", "seccompprofiles", "--all", "--all-namespaces")
+	e.kubectl("delete", "selinuxprofiles", "--all", "--all-namespaces")
 	e.kubectl("delete", "--ignore-not-found", "-f", manifest)
 }
 
diff --git a/test/suite_test.go b/test/suite_test.go
index 7db0aa411..695d982e2 100644
--- a/test/suite_test.go
+++ b/test/suite_test.go
@@ -95,6 +95,7 @@ type e2e struct {
 	bpfRecorderEnabled     bool
 	skipNamespacedTests    bool
 	testWebhookConfig      bool
+	singleNodeEnvironment  bool
 	logger                 logr.Logger
 	execNode               func(node string, args ...string) string
 	waitForReadyPods       func()
@@ -254,6 +255,9 @@ func TestSuite(t *testing.T) {
 				skipNamespacedTests:    skipNamespacedTests,
 				operatorManifest:       operatorManifest,
 				testWebhookConfig:      testWebhookConfig,
+				// NOTE(jaosorior): Our current vanilla jobs are
+				// single-node only.
+				singleNodeEnvironment: true,
 			},
 		})
 	default:
@@ -691,6 +695,12 @@ func (e *e2e) bpfRecorderOnlyTestCase() {
 	e.enableBpfRecorderInSpod()
 }
 
+func (e *e2e) singleNodeTestCase() {
+	if !e.singleNodeEnvironment {
+		e.T().Skip("Skipping test because we're in a multi-node environment.")
+	}
+}
+
 func (e *e2e) enableBpfRecorderInSpod() {
 	e.logf("Enable bpf recorder in SPOD")
 	e.kubectlOperatorNS("patch", "spod", "spod", "-p", `{"spec":{"enableBpfRecorder": true}}`, "--type=merge")
diff --git a/test/tc_log_enricher_test.go b/test/tc_log_enricher_test.go
index df193d997..2ab564312 100644
--- a/test/tc_log_enricher_test.go
+++ b/test/tc_log_enricher_test.go
@@ -49,6 +49,9 @@ spec:
 	defer profileCleanup()
 	defer e.kubectl("delete", "sp", profileName)
 
+	e.logf("Waiting for profile to be reconciled")
+	e.waitFor("condition=ready", "sp", profileName)
+
 	e.logf("Creating test pod")
 	namespace := e.getCurrentContextNamespace(defaultNamespace)
 
@@ -106,14 +109,19 @@ spec:
 	e.Contains(output, `"syscallName"="listen"`)
 	e.Contains(output, `"syscallID"=50`)
 
-	metrics := e.getSpodMetrics()
-	e.Regexp(fmt.Sprintf(`(?m)security_profiles_operator_seccomp_profile_audit_total{`+
-		`container="%s",`+
-		`executable="/usr/sbin/nginx",`+
-		`namespace="%s",`+
-		`node=".*",`+
-		`pod="%s",`+
-		`syscall="listen"} \d+`,
-		containerName, namespace, podName,
-	), metrics)
+	if e.singleNodeEnvironment {
+		// we only run the metrics checks in a single node environment because otherwise it's a lottery
+		// which spod instance do we hit and the test is not stable
+
+		metrics := e.getSpodMetrics()
+		e.Regexp(fmt.Sprintf(`(?m)security_profiles_operator_seccomp_profile_audit_total{`+
+			`container="%s",`+
+			`executable="/usr/sbin/nginx",`+
+			`namespace="%s",`+
+			`node=".*",`+
+			`pod="%s",`+
+			`syscall="listen"} \d+`,
+			containerName, namespace, podName,
+		), metrics)
+	}
 }
diff --git a/test/tc_metrics_test.go b/test/tc_metrics_test.go
index bc9529fe6..aca380b18 100644
--- a/test/tc_metrics_test.go
+++ b/test/tc_metrics_test.go
@@ -27,6 +27,7 @@ const profileName = "metrics-profile"
 
 func (e *e2e) testCaseSeccompMetrics(nodes []string) {
 	e.seccompOnlyTestCase()
+	e.singleNodeTestCase()
 
 	const (
 		operationDelete = `security_profiles_operator_seccomp_profile_total{operation="delete"}`
@@ -72,6 +73,7 @@ spec:
 
 func (e *e2e) testCaseSelinuxMetrics(nodes []string) {
 	e.selinuxOnlyTestCase()
+	e.singleNodeTestCase()
 
 	const (
 		operationDelete = `security_profiles_operator_selinux_profile_total{operation="delete"}`