Repository: kubernetes-sigs/security-profiles-operator
Pull request #383 (2021-03-19 18:48:28)

seccomp: Don't delete a profile in use
<!--  Thanks for sending a pull request!  Here are some tips for you:

- If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
- Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
- If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
- If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind bug

<!--
Add one of the following kinds:
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
the finalizer we set using the pod reconciler was never taken into use
and thus the profiles could have been removed while still in use.

#### Which issue(s) this PR fixes:
None

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.

Fixes #

or

None
-->

#### Does this PR have test?
yes

<!--
If tests aren't applicable just write N/A.
-->

#### Special notes for your reviewer:
I haven't tested this at all because it's friday evening.

#### Does this PR introduce a user-facing change?

<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
A bug where a profile could have been deleted while still in use by pods was fixed
```

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/383#" title="Author self-approved">jhrozek</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saschagrunert** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saschagrunert` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saschagrunert"]} -->
k8s-ci-robot: @jhrozek: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-verify/1372983330856767488) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=383). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @jhrozek: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-verify/1372983330856767488) | `/test pull-security-profiles-operator-verify`
pull-security-profiles-operator-test-e2e | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-test-e2e/1372983330932264960) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=383). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
JAORMX: /retest
k8s-ci-robot: @jhrozek: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-test-e2e/1372983330932264960) | `/test pull-security-profiles-operator-test-e2e`
pull-security-profiles-operator-verify | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-verify/1373206941823668224) | `/test pull-security-profiles-operator-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=383). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @jhrozek: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-verify | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-verify/1373206941823668224) | `/test pull-security-profiles-operator-verify`
pull-security-profiles-operator-test-e2e | 0ac4e1cd3d681d9be93b77f992b1f8fb50084567 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-test-e2e/1373206941865611264) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=383). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jhrozek: On Mon, Mar 22, 2021 at 01:22:52AM -0700, Sascha Grunert wrote:
> @saschagrunert requested changes on this pull request.
> 
> Code LGTM, can you please fix the lint issues that the CI is happy?

I dream of the day when I'll be able to submit a PR without the verify
target kicking my ass.

this version splits the delete handler into its own function, because
the linter was complaining (rightfully so) that there is too mant if-s
in the delete branch.

k8s-ci-robot: @jhrozek: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-security-profiles-operator-test-e2e | 93da77a7d91ecd7d2940af4269bd20df6f389bbd | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_security-profiles-operator/383/pull-security-profiles-operator-test-e2e/1373943785154678784) | `/test pull-security-profiles-operator-test-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=security-profiles-operator&pr=383). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ajhrozek). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/security-profiles-operator/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @jhrozek: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
jhrozek: > I think this is worth a release note since it fixes a user-facing bug

OK, I amended the PR description, if there's more I need to do, please let me know.
codecov-commenter: # [Codecov](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/383?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#383](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/pull/383?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (2c94989) into [master](https://codecov.io/gh/kubernetes-sigs/security-profiles-operator/commit/7e8641ff02411fa8ed9f050f981d143ccbcb1da3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (7e8641f) will **decrease** coverage by `0.08%`.
> The diff coverage is `0.00%`.


```diff
@@            Coverage Diff             @@
##           master     #383      +/-   ##
==========================================
- Coverage   38.08%   38.00%   -0.09%     
==========================================
  Files          20       20              
  Lines         948      950       +2     
==========================================
  Hits          361      361              
- Misses        572      574       +2     
  Partials       15       15              
```

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/383#" title="Author self-approved">jhrozek</a>*, *<a href="https://github.com/kubernetes-sigs/security-profiles-operator/pull/383#pullrequestreview-638553542" title="LGTM">saschagrunert</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fsecurity-profiles-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/security-profiles-operator/blob/master/OWNERS)~~ [saschagrunert]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/internal/pkg/daemon/seccompprofile/seccompprofile.go b/internal/pkg/daemon/seccompprofile/seccompprofile.go
index b035e3720..d0d9e7852 100644
--- a/internal/pkg/daemon/seccompprofile/seccompprofile.go
+++ b/internal/pkg/daemon/seccompprofile/seccompprofile.go
@@ -32,6 +32,7 @@ import (
 	"github.com/pkg/errors"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 
 	"sigs.k8s.io/security-profiles-operator/api/seccompprofile/v1alpha1"
@@ -307,6 +308,11 @@ func (r *Reconciler) reconcileDeletion(
 		}
 	}
 
+	if controllerutil.ContainsFinalizer(sp, util.HasActivePodsFinalizerString) {
+		r.log.Info("cannot delete profile in use by pod, requeuing")
+		return reconcile.Result{RequeueAfter: wait}, nil
+	}
+
 	if err := handleDeletion(sp, r.log); err != nil {
 		r.log.Error(err, "cannot delete profile")
 		r.record.Event(sp, event.Warning(reasonCannotRemoveProfile, err))
diff --git a/internal/pkg/manager/workloadannotator/workloadannotator.go b/internal/pkg/manager/workloadannotator/workloadannotator.go
index 4116b078f..9132be480 100644
--- a/internal/pkg/manager/workloadannotator/workloadannotator.go
+++ b/internal/pkg/manager/workloadannotator/workloadannotator.go
@@ -123,16 +123,15 @@ func (r *PodReconciler) updatePodReferences(ctx context.Context, sp *v1alpha1.Se
 	}, util.IsNotFoundOrConflict); err != nil {
 		return errors.Wrap(err, "updating SeccompProfile status")
 	}
-	hasActivePodsFinalizerString := "in-use-by-active-pods"
 	if len(linkedPods.Items) > 0 {
 		if err := util.Retry(func() error {
-			return util.AddFinalizer(ctx, r.client, sp, hasActivePodsFinalizerString)
+			return util.AddFinalizer(ctx, r.client, sp, util.HasActivePodsFinalizerString)
 		}, util.IsNotFoundOrConflict); err != nil {
 			return errors.Wrap(err, "adding finalizer")
 		}
 	} else {
 		if err := util.Retry(func() error {
-			return util.RemoveFinalizer(ctx, r.client, sp, hasActivePodsFinalizerString)
+			return util.RemoveFinalizer(ctx, r.client, sp, util.HasActivePodsFinalizerString)
 		}, util.IsNotFoundOrConflict); err != nil {
 			return errors.Wrap(err, "removing finalizer")
 		}
diff --git a/internal/pkg/util/constants.go b/internal/pkg/util/constants.go
index 1af901ca6..ceb613e19 100644
--- a/internal/pkg/util/constants.go
+++ b/internal/pkg/util/constants.go
@@ -17,5 +17,6 @@ limitations under the License.
 package util
 
 const (
-	ErrGetProfile = "cannot get security profile"
+	ErrGetProfile                = "cannot get security profile"
+	HasActivePodsFinalizerString = "in-use-by-active-pods"
 )


Test patch:
diff --git a/test/e2e_test.go b/test/e2e_test.go
index 6e894c4b3..1c8e1a141 100644
--- a/test/e2e_test.go
+++ b/test/e2e_test.go
@@ -270,6 +270,18 @@ func (e *e2e) getSeccompProfileNodeStatus(
 	return &secpolNodeStatusList.Items[0]
 }
 
+func (e *e2e) getAllSeccompProfileNodeStatuses(
+	name, namespace string,
+) *secprofnodestatusv1alpha1.SecurityProfileNodeStatusList {
+	selector := fmt.Sprintf("spo.x-k8s.io/profile-name=%s", name)
+	seccompProfileNodeStatusJSON := e.kubectl(
+		"-n", namespace, "get", "securityprofilenodestatus", "-l", selector, "-o", "json",
+	)
+	secpolNodeStatusList := &secprofnodestatusv1alpha1.SecurityProfileNodeStatusList{}
+	e.Nil(json.Unmarshal([]byte(seccompProfileNodeStatusJSON), secpolNodeStatusList))
+	return secpolNodeStatusList
+}
+
 func (e *e2e) getCurrentContextNamespace(alt string) string {
 	ctxns := e.kubectl("config", "view", "--minify", "-o", "jsonpath={..namespace}")
 	if ctxns == "" {
diff --git a/test/tc_delete_profiles_test.go b/test/tc_delete_profiles_test.go
index 8f9f4fd8e..ec77a609c 100644
--- a/test/tc_delete_profiles_test.go
+++ b/test/tc_delete_profiles_test.go
@@ -166,6 +166,16 @@ spec:
 		sp := e.getSeccompProfile(deleteProfileName, namespace)
 		e.Equal(sp.Status.Status, secprofnodestatusv1alpha1.ProfileStateTerminating)
 
+		// The node statuses should still be there, just terminating
+		nodeStatuses := e.getAllSeccompProfileNodeStatuses(deleteProfileName, namespace)
+		for i := range nodeStatuses.Items {
+			e.Equal(nodeStatuses.Items[i].Status, secprofnodestatusv1alpha1.ProfileStateTerminating)
+			// On each node, there should still be the profile on the disk
+			nodeWithPodName := nodeStatuses.Items[i].NodeName
+			path := sp.GetProfilePath()
+			e.execNode(nodeWithPodName, "test", "-f", path)
+		}
+
 		isDeleted := make(chan bool)
 		go func() {
 			e.waitFor("delete", "seccompprofile", deleteProfileName)