Repository: vmware-tanzu/velero
Pull request #8777: Issue #8772 ensure pv removed
Author: login: ix-rzi
Labels: name: has-unit-tests
name: has-changelog
Created: 2025-03-11T14:58:15Z | Merged: 2025-03-13T14:39:25Z

Thank you for contributing to Velero!

# Please add a summary of your change

The node-agent does not wait for the PV to be removed on the cluster before deleting the VolumeSnapshot. The change
lets the node-agent wait for the PV to be removed from the cluster before removing the VS.

# Does your change fix a particular issue?

Fixes #8772 

# Please indicate you've done the following:

- [x] [Accepted the DCO](https://velero.io/docs/v1.5/code-standards/#dco-sign-off). Commits without the DCO will delay acceptance.
- [x] [Created a changelog file (`make new-changelog`)](https://velero.io/docs/main/code-standards/#adding-a-changelog) or comment `/kind changelog-not-required` on this PR.
- [ ] Updated the corresponding documentation in `site/content/docs/main`.


Linked issue #8772: node-agent not waiting for PV removal
**What steps did you take and what happened:**

The node-agent does not wait for the PV to be removed on the cluster before deleting the VolumeSnapshot. As we are using NetApp Trident and NetApp ONTAP based storage this affects us in the following way:

Most of the time the NetApp FlexClone that is made available as PVC for the data upload is still in a deleting state on the NetApp filer when the VolumeSnapshot is deleted as part of the cleanup. As a consequence, NetApp Trident identifies the snapshot that is the source of the FlexClone as 'busy' and will perform a split clone operation, turning the FlexClone into a real clone. 

This means that all the data will be copied into an independent volume and will be deleted afterwards. While the waste of space is temporary, it can take hours to perform this operation and may have the potential to fill up the storage. Trident is not able to remove the volume that is the base for the PVC as long as the clone operation is in progress.

I do not think this is a problem with NetApp Trident!

***Commands***
velero install --provider aws \
 --bucket velero \
 --plugins velero/velero-plugin-for-aws:v1.11.0 \
 --secret-file ./secret-file \
 --use-volume-snapshots=true \
 --backup-location-config region="minio",s3ForcePathStyle="true",s3Url=http://xxx.yyy.zzz.aaa:9000 \
 --snapshot-location-config region="minio" \
 --use-node-agent \
 --features=EnableCSI \
 --node-agent-configmap=node-agent-config

velero backup create test01 --include-namespaces nginx --snapshot-move-data

**What did you expect to happen:**

node-agent waits for the PV to disappear before deleting the VolumeSnapshot.

**The following information will help us better understand what's going on**:

PoC code to wait for the PV delete. When running the forked node-agent the split clones no longer happen:

https://github.com/vmware-tanzu/velero/compare/main...ix-rzi:velero:fix-ensure-pv-removed

I'm happy to discuss and open a PR.


**Environment:**

Note, this is a local dev/test environment!

- Velero version :  v1.15.2
- Velero features : <NOT SET>
- Kubernetes version: v1.31.5+k3s1
- Kubernetes installer & version: K3s 
- Cloud provider or hardware configuration: K3s on Fedora 41
- OS (e.g. from `/etc/os-release`):
NAME="Fedora Linux"
VERSION="41 (Server Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Server Edition"
VARIANT_ID=server

**Vote on this issue!**

This is an invitation to the Velero community to vote on issues, you can see the project's [top voted issues listed here](https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).  
Use the "reaction smiley face" up to the right of this comment to vote.

- :+1: for "I would like to see this bug fixed as soon as possible"
- :-1: for "There are more important bugs to focus on right now"

Linked issue comments: user: login: blackpiglet
created_at: 2025-03-11T03:31:31Z
body: Let me try to sort out what happens here.
* The problem happens when Velero deletes the backup or restore Pod, PVC, PV, VS, and VSC.
* The unwanted is that the backup or restore PVC's data source, which is the backup or restore VS, are deleted before the PVC is deleted. That caused the full clone of the PVC volume from the snapshot. I think that is reasonable because the volume source is gone, and the solution should be making the volume independent from the snapshot source.

If my understanding is correct, then the enhancement should modify the backup restore resource cleaning order.
Your modification should work, but I think we should also consider whether there are some error cases that may also have a similar issue.
author_association: CONTRIBUTOR
user: login: Lyndon-Li
created_at: 2025-03-11T05:49:59Z
body: @ix-rzi 

> PVC for the data upload is still in a deleting state on the NetApp filer when the VolumeSnapshot is deleted as part of the cleanup

Thanks for the good catch and analysis. Please submit the PR and let's discuss in the PR for minor changes.

The changes are based on the fact that the FlexClone has been fully deleted from the storage when the backupPV disappears from the cluster, otherwise, it still doesn't help. But I think Kubernetes and CSI driver should be working in this way and guarantee this.
author_association: CONTRIBUTOR
user: login: Lyndon-Li
created_at: 2025-03-11T06:10:58Z
body: @blackpiglet Let's track the cleanup logics for error handling separately. Probably, for various cases in error handling, make `DeletePVAndPVCIfAny` do the same wait is a good idea.
author_association: CONTRIBUTOR
user: login: ix-rzi
created_at: 2025-03-11T12:51:33Z
body: @Lyndon-Li Indeed, K8s and the CSI driver should take care of this, assuming a correct CSI driver implementation. I will submit the PR ASAP.
author_association: CONTRIBUTOR

Diff:
diff --git a/changelogs/unreleased/8777-ix-rzi b/changelogs/unreleased/8777-ix-rzi
new file mode 100644
index 0000000000..397d4aa361
--- /dev/null
+++ b/changelogs/unreleased/8777-ix-rzi
@@ -0,0 +1 @@
+ensure that PV is removed before VS is deleted
diff --git a/pkg/util/kube/pvc_pv.go b/pkg/util/kube/pvc_pv.go
index 49eb91dbe4..3b4ddc2750 100644
--- a/pkg/util/kube/pvc_pv.go
+++ b/pkg/util/kube/pvc_pv.go
@@ -74,6 +74,10 @@ func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interfac
 	if err := EnsureDeletePVC(ctx, client, pvcName, pvcNamespace, ensureTimeout); err != nil {
 		log.Warnf("failed to delete pvc %s/%s with err %v", pvcNamespace, pvcName, err)
 	}
+
+	if err := EnsurePVDeleted(ctx, client, pvcObj.Spec.VolumeName, ensureTimeout); err != nil {
+		log.Warnf("pv %s was not removed with err %v", pvcObj.Spec.VolumeName, err)
+	}
 }
 
 // WaitPVCBound wait for binding of a PVC specified by name and returns the bound PV object
@@ -157,6 +161,37 @@ func EnsureDeletePVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface
 	return nil
 }
 
+// EnsurePVDeleted ensures a PV has been deleted. This function is supposed to be called after EnsureDeletePVC
+// If timeout is 0, it doesn't wait and return nil
+func EnsurePVDeleted(ctx context.Context, pvGetter corev1client.CoreV1Interface, pvName string, timeout time.Duration) error {
+	if timeout == 0 {
+		return nil
+	}
+
+	err := wait.PollUntilContextTimeout(ctx, waitInternal, timeout, true, func(ctx context.Context) (bool, error) {
+		_, err := pvGetter.PersistentVolumes().Get(ctx, pvName, metav1.GetOptions{})
+		if err != nil {
+			if apierrors.IsNotFound(err) {
+				return true, nil
+			}
+
+			return false, errors.Wrapf(err, "error to get pv %s", pvName)
+		}
+
+		return false, nil
+	})
+
+	if err != nil {
+		if errors.Is(err, context.DeadlineExceeded) {
+			return errors.Errorf("timeout to assure pv %s is deleted", pvName)
+		} else {
+			return errors.Wrapf(err, "error to ensure pv is deleted for %s", pvName)
+		}
+	}
+
+	return nil
+}
+
 // RebindPVC rebinds a PVC by modifying its VolumeName to the specific PV
 func RebindPVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface,
 	pvc *corev1api.PersistentVolumeClaim, pv string) (*corev1api.PersistentVolumeClaim, error) {
diff --git a/pkg/util/kube/pvc_pv_test.go b/pkg/util/kube/pvc_pv_test.go
index 06e8058f29..5f0abcf858 100644
--- a/pkg/util/kube/pvc_pv_test.go
+++ b/pkg/util/kube/pvc_pv_test.go
@@ -707,6 +707,94 @@ func TestEnsureDeletePVC(t *testing.T) {
 	}
 }
 
+func TestEnsureDeletePV(t *testing.T) {
+	pvObject := &corev1api.PersistentVolume{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "fake-pv",
+		},
+	}
+
+	tests := []struct {
+		name      string
+		clientObj []runtime.Object
+		pvName    string
+		reactors  []reactor
+		timeout   time.Duration
+		err       string
+	}{
+		{
+			name:   "get fail",
+			pvName: "fake-pv",
+			err:    "error to get pv fake-pv: persistentvolumes \"fake-pv\" not found",
+		},
+		{
+			name:      "0 timeout",
+			pvName:    "fake-pv",
+			clientObj: []runtime.Object{pvObject},
+			reactors: []reactor{
+				{
+					verb:     "get",
+					resource: "persistentvolumes",
+					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
+						return true, pvObject, nil
+					},
+				},
+			},
+		},
+		{
+			name:      "wait fail",
+			pvName:    "fake-pv",
+			clientObj: []runtime.Object{pvObject},
+			timeout:   time.Millisecond,
+			reactors: []reactor{
+				{
+					verb:     "get",
+					resource: "persistentvolumes",
+					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("fake-get-error")
+					},
+				},
+			},
+			err: "error to ensure pv is deleted for fake-pv: error to get pv fake-pv: fake-get-error",
+		},
+		{
+			name:      "wait timeout",
+			pvName:    "fake-pv",
+			clientObj: []runtime.Object{pvObject},
+			timeout:   time.Millisecond,
+			reactors: []reactor{
+				{
+					verb:     "get",
+					resource: "persistentvolumes",
+					reactorFunc: func(action clientTesting.Action) (handled bool, ret runtime.Object, err error) {
+						return true, pvObject, nil
+					},
+				},
+			},
+			err: "timeout to assure pv fake-pv is deleted",
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			fakeKubeClient := fake.NewSimpleClientset(test.clientObj...)
+
+			for _, reactor := range test.reactors {
+				fakeKubeClient.Fake.PrependReactor(reactor.verb, reactor.resource, reactor.reactorFunc)
+			}
+
+			var kubeClient kubernetes.Interface = fakeKubeClient
+
+			err := EnsurePVDeleted(context.Background(), kubeClient.CoreV1(), test.pvName, test.timeout)
+			if err != nil {
+				assert.EqualError(t, err, test.err)
+			} else {
+				assert.NoError(t, err)
+			}
+		})
+	}
+}
+
 func TestRebindPVC(t *testing.T) {
 	pvcObject := &corev1api.PersistentVolumeClaim{
 		ObjectMeta: metav1.ObjectMeta{


Reviews:
user: login: kaovilai
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-03-11T18:43:14Z
user: login: Lyndon-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T00:24:17Z
user: login: kaovilai
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T14:21:12Z
user: login: ix-rzi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T08:48:38Z
user: login: Lyndon-Li
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T08:51:50Z
user: login: kaovilai
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-13T14:38:10Z

Review comments:
diff_hunk: @@ -74,6 +74,10 @@ func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interfac
 	if err := EnsureDeletePVC(ctx, client, pvcName, pvcNamespace, ensureTimeout); err != nil {
 		log.Warnf("failed to delete pvc %s/%s with err %v", pvcNamespace, pvcName, err)
 	}
+
+	if err := EnsureDeletePV(ctx, client, pvcObj.Spec.VolumeName, ensureTimeout); err != nil {
+		log.Warnf("pv %s was not removed with err %v", &pvcObj.Spec.VolumeName, err)
path: pkg/util/kube/pvc_pv.go
user: login: Lyndon-Li
body: Why do we pass a pointer (of `pvcObj.Spec.VolumeName`) here for `%s`?
created_at: 2025-03-12T00:22:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -157,6 +161,37 @@ func EnsureDeletePVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface
 	return nil
 }
 
+// EnsureDeletePV ensures a PV has been deleted. This function is supposed to be called after EnsureDeletePVC
+// If timeout is 0, it doesn't wait and return nil
+func EnsureDeletePV(ctx context.Context, pvGetter corev1client.CoreV1Interface, pvName string, timeout time.Duration) error {
path: pkg/util/kube/pvc_pv.go
user: login: Lyndon-Li
body: It's better to have the name as `EnsurePVDeleted` since we don't make any deletion as same as other EnsureDelete* functions.
created_at: 2025-03-12T00:23:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,6 +74,10 @@ func DeletePVAndPVCIfAny(ctx context.Context, client corev1client.CoreV1Interfac
 	if err := EnsureDeletePVC(ctx, client, pvcName, pvcNamespace, ensureTimeout); err != nil {
 		log.Warnf("failed to delete pvc %s/%s with err %v", pvcNamespace, pvcName, err)
 	}
+
+	if err := EnsureDeletePV(ctx, client, pvcObj.Spec.VolumeName, ensureTimeout); err != nil {
+		log.Warnf("pv %s was not removed with err %v", &pvcObj.Spec.VolumeName, err)
path: pkg/util/kube/pvc_pv.go
user: login: kaovilai
body: good catch..
```suggestion
		log.Warnf("pv %s was not removed with err %v", pvcObj.Spec.VolumeName, err)
```
created_at: 2025-03-12T14:21:11Z
author_association: COLLABORATOR
diff_hunk: @@ -157,6 +161,37 @@ func EnsureDeletePVC(ctx context.Context, pvcGetter corev1client.CoreV1Interface
 	return nil
 }
 
+// EnsureDeletePV ensures a PV has been deleted. This function is supposed to be called after EnsureDeletePVC
+// If timeout is 0, it doesn't wait and return nil
+func EnsureDeletePV(ctx context.Context, pvGetter corev1client.CoreV1Interface, pvName string, timeout time.Duration) error {
path: pkg/util/kube/pvc_pv.go
user: login: ix-rzi
body: Renamed as suggested
created_at: 2025-03-13T08:48:38Z
author_association: CONTRIBUTOR

PR comments:
user: login: ix-rzi
created_at: 2025-03-11T15:27:44Z
body: Don't think there is something to update in the docs, but I might be mistaken.
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-11T18:52:12Z
body: ## [Codecov](https://app.codecov.io/gh/vmware-tanzu/velero/pull/8777?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu) Report
Attention: Patch coverage is `81.81818%` with `4 lines` in your changes missing coverage. Please review.
> Project coverage is 59.58%. Comparing base [(`1516e72`)](https://app.codecov.io/gh/vmware-tanzu/velero/commit/1516e72ccb22f5e3e020363e3de469167dc5c469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu) to head [(`15a964b`)](https://app.codecov.io/gh/vmware-tanzu/velero/commit/15a964b29b4c2d16a746aaff5cc8ca6796dde787?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu).
> Report is 13 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/vmware-tanzu/velero/pull/8777?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu) | Patch % | Lines |
|---|---|---|
| [pkg/util/kube/pvc\_pv.go](https://app.codecov.io/gh/vmware-tanzu/velero/pull/8777?src=pr&el=tree&filepath=pkg%2Futil%2Fkube%2Fpvc_pv.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu#diff-cGtnL3V0aWwva3ViZS9wdmNfcHYuZ28=) | 81.81% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/vmware-tanzu/velero/pull/8777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #8777      +/-   ##
==========================================
+ Coverage   59.53%   59.58%   +0.04%     
==========================================
  Files         371      370       -1     
  Lines       40196    40233      +37     
==========================================
+ Hits        23932    23971      +39     
+ Misses      14767    14764       -3     
- Partials     1497     1498       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/vmware-tanzu/velero/pull/8777?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware-tanzu).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE