Repository: kubernetes-sigs/security-profiles-operator
Pull request #1153 (2022-09-06 19:39:29)

tests: Run the flaky tests, too
<!--  Thanks for sending a pull request!  Here are some tips for you:

- If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
- Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
- If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
- If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind bug

<!--
Add one of the following kinds:
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
Turns out that my previous patch to run the flaky tests was incomplete.
We need some more changes:
    - the `testify.m` parameter must be, for some reason, specified last
      or else the `go test` call fails with "no Go files in XYZ".
    - after the namespaced tests (last in the regular non-flaky tests),
      we need to reset the namespace or else we test everything in the
      test-ns namespace
    - we need to tear down the operator after the namespaced tests to
      be able to roll it out again in the flaky tests

The tests are still flaky, but at least they run now.


#### Which issue(s) this PR fixes:
None

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.

Fixes #

or

-->

#### Does this PR have test?
implicit

<!--
If tests aren't applicable just write N/A.
-->

#### Special notes for your reviewer:
N/A

#### Does this PR introduce a user-facing change?

<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
NONE
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/1153#" title="Author self-approved">jhrozek</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/OWNERS)~~ [jhrozek]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov-commenter: # [Codecov](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/1153?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#1153](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/1153?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (0f255a1) into [main](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/commit/7d5b719bcc224b00c387ff5505137ceb2113b5f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (7d5b719) will **not change** coverage.
> The diff coverage is `n/a`.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #1153   +/-   ##
=======================================
  Coverage   50.70%   50.70%           
=======================================
  Files          42       42           
  Lines        4753     4753           
=======================================
  Hits         2410     2410           
  Misses       2266     2266           
  Partials       77       77           
```

</details>

Patch:
diff --git a/Makefile b/Makefile
index 1d27ca856..c65a2ebce 100644
--- a/Makefile
+++ b/Makefile
@@ -396,7 +396,7 @@ test-e2e: ## Run the end-to-end tests
 test-flaky-e2e: ## Only run the flaky end-to-end tests
 	CGO_LDFLAGS= \
 	E2E_SKIP_FLAKY_TESTS=false \
-	$(GO) test -parallel 1 -testify.m '^(TestSecurityProfilesOperator_Flaky)$$' -timeout 20m -count=1 ./test/... -v
+	$(GO) test -parallel 1 -timeout 20m -count=1 ./test/... -v -testify.m '^(TestSecurityProfilesOperator_Flaky)$$'
 
 # Generate CRD manifests
 manifests: $(BUILD_DIR)/kubernetes-split-yaml $(BUILD_DIR)/kustomize


Test patch:
diff --git a/test/e2e_flaky_test.go b/test/e2e_flaky_test.go
index 9e2c479f5..2ab1b764a 100644
--- a/test/e2e_flaky_test.go
+++ b/test/e2e_flaky_test.go
@@ -16,12 +16,18 @@ limitations under the License.
 
 package e2e_test
 
+import "sigs.k8s.io/security-profiles-operator/internal/pkg/config"
+
 func (e *e2e) TestSecurityProfilesOperator_Flaky() {
 	if e.skipFlakyTests {
 		e.T().Skip("Skipping flaky tests")
 		return
 	}
 
+	// If we ran the non-flaky tests before, we would have ran them with the
+	// context set to the test-ns namespace. Reset the context.
+	e.kubectl("config", "set-context", "--current", "--namespace", config.OperatorName)
+
 	e.waitForReadyPods()
 
 	// Deploy prerequisites
diff --git a/test/e2e_test.go b/test/e2e_test.go
index f11220d13..d27dcd679 100644
--- a/test/e2e_test.go
+++ b/test/e2e_test.go
@@ -172,8 +172,15 @@ func (e *e2e) testNamespacedOperator(manifest, namespace string, testCases []tes
 	// Deploy the namespace operator
 	e.kubectl("create", "namespace", namespace)
 	e.updateManifest(manifest, "NS_REPLACE", namespace)
+
 	// All following operations such as create pod will be in the test namespace
+	// at the same time, let's re-set the context to allow subsequent test runs
+	curNs := e.getCurrentContextNamespace(config.OperatorName)
 	e.kubectl("config", "set-context", "--current", "--namespace", namespace)
+	defer func() {
+		e.kubectl("config", "set-context", "--current", "--namespace", curNs)
+	}()
+
 	e.deployOperator(manifest)
 
 	for _, testCase := range testCases {
@@ -182,7 +189,9 @@ func (e *e2e) testNamespacedOperator(manifest, namespace string, testCases []tes
 			tc.fn(nodes)
 		})
 	}
+	e.cleanupOperator(e.operatorManifest)
 	e.run("git", "checkout", manifest)
+	e.kubectl("delete", "namespace", namespace)
 }
 
 func doDeployCertManager(e *e2e) {