Repository: vmware-tanzu/velero (Go)

Issue:
v1.14.0 Scheduled backup fails if namespace doesn't exist
**What steps did you take and what happened:**
Create a schedule and include a non-existing namespace:

```
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-application-backup
  namespace: velero
  annotations:
  labels:
    app.kubernetes.io/name: velero
    app.kubernetes.io/instance: velero
spec:
  useOwnerReferencesInBackup: false
  schedule: "0 0 * * *"
  template:
    csiSnapshotTimeout: 14400s
    hooks: {}
    includeClusterResources: false
    includedNamespaces:
    - my-namespace
    - a-non-existing-namespace
    itemOperationTimeout: 0s
    metadata: {}
    ttl: 720h
```

**What did you expect to happen:**

Velero before v1.14.0 executed the backup without complaint. Velero v1.14.0 stops the scheduled backup instantly and sets the status `FailedValidation` instead of executing the backup and e.g. setting the `PartiallyFailed` status. This new behavior is dangerous (or even critical?) as scheduled backups become fragile and stop working as soon as a defined namespace doesn't exist anymore.


Discussion:
It seems the behavior was changed in https://github.com/vmware-tanzu/velero/issues/7431 by @ywk253100. While I agree that there should be some failure reporting if "--include-namespaces" references namespaces which do not exist, I don't agree that scheduled backups should be skipped entirely.

This behavior is dangerous as the backup could be skipped without being recognized. IMHO the backup should be executed and the status should become `PartiallyFailed` instead. This will backup existing items and report that some items (like the namespaces which have been declared to be backed up) were missing.
Ignoring this being in a schedule, has velero ever claimed that includedNamespaces can include namespaces that do not exists?

If we ever implement wildcard/regex for namespaces to include https://github.com/vmware-tanzu/velero/issues/1874 then perhaps you could define that regex which I would expect would not fail a backup if no namespace exists.
@kaovilai
Velero doesn't support the wildcard or regular expression in the namespace filter parameters.

@phoenix-bjoern 
There are some issues similar to this topic, and some requests fail the backup on nonexisting namespaces, and some requests are identical to yours.
> @kaovilai
> Velero doesn't support the wildcard or regular expression in the namespace filter parameters.

Right, referring to potential future implementation. 🤞
I prefer not to change this logic.

The reason is marking backup as PartiallyFail on a non-existing namespace has been the existing behavior for quite some time.
Modifying it is a breaking change, which is not an obvious improvement.

This [PR](https://github.com/vmware-tanzu/velero/pull/1386) introduces the behavior of marking backup as `PartiallyFail` on a non-existing namespace.
Before this PR, the backup was marked as `Fail` immediately.

Until PR #6320 accidentally broke the check, then #7431 restored that behavior.

@blackpiglet Not sure if I can follow. `PartiallyFail` would be actually fine for me. But in v1.14.0 backups fail immediately with status `FailedValidation`, so no namespace is backed up. That means that users can easily break the configured backup procedure, which is not good. `PartiallyFail` gives operators time to fix either the backup config or re-add the missing namespace without compromising safety. Not taking a backup is the worst option.
> I prefer not to change this logic.

Same same.

My suggestion is that if we ever support wildcard/regex, that it be a completely new field in the backup spec just for regex/wildcard support. Then the behavior to support this issue won't be breaking change.

You'd have existing field for exact name, and new field for regex/wildcard.
@kaovilai this issue is not about namespaces and regex/wildcards. It is about plain namespaces in schedules, which might be missing for whatever reason, and Velero v1.14.0 now doesn’t perform any backup. IMHO this is a critical issue and should be changed to a status „PartiallyFailed“ instead of „FailedValidation“ to ensure all other namespaces get backed up as intended by the schedule definition. 
I get the point about this not being regex.. I am simply suggesting that

"if namespace don't exists, don't error" be part of the regex implementation in a new `includedRegexNamespaces`.. meaning if your regex is "namespaceExactName", and it doesn't match a current list of namespaces, it won't error.
current available fields behavior is unlikely to change per https://github.com/vmware-tanzu/velero/issues/7928#issuecomment-2199548047
@kaovilai hmm. I think PartiallyFailed is the longstanding Velero behavior.  1.12 introduced a regression that resulted in Completed rather than PartiallyFailed, but unfortunately the 1.14 fix resulted in FailedValidation backups rather than PartiallyFailed. Based on what @blackpiglet said above ("The reason is marking backup as PartiallyFail on a non-existing namespace has been the existing behavior for quite some time.
Modifying it is a breaking change, which is not an obvious improvement.") it would appear that we should restore the PartialFailure change and not fail validation
@phoenix-bjoern 
Ah, I see. Sorry for the misunderstood.

@kaovilai @sseago 
My understanding is the agreement is we should mark the Backup with a non-existing namespace name in the namespace filters as `PartiallyFailed` instead of `FailedValidation`.

If that's correct, I will create a PR to fix it.
@blackpiglet CORRECT! Looking forward to your PR and happy to test it ❤️
@blackpiglet I think we're all in agreement now. Go ahead and create the PR.
@blackpiglet Do you plan to release this change as a new Velero version or should this change be considered as a bugfix for the v1.14 release? If it is a bugfix for v1.14 could you please merge the changes also to the release-1.14 branch?
OK. I will create a cherry-pick PR to the release-1.14
@blackpiglet Confirming the solution works, thank you!
Lookup forward to the v1.14.1 bugfix release :-)
@blackpiglet Thank you for the fix. Closing the ticket as the issue is resolved. Looking forward for the v1.14.1 release.



Patch:
diff --git a/changelogs/unreleased/7965-blackpiglet b/changelogs/unreleased/7965-blackpiglet
new file mode 100644
index 0000000000..72c8a4fdd9
--- /dev/null
+++ b/changelogs/unreleased/7965-blackpiglet
@@ -0,0 +1,1 @@
+Check whether the namespaces specified in namespace filter exist.
\ No newline at end of file
diff --git a/pkg/backup/item_collector.go b/pkg/backup/item_collector.go
index c40f152a96..af7ffe9ae6 100644
--- a/pkg/backup/item_collector.go
+++ b/pkg/backup/item_collector.go
@@ -741,6 +741,23 @@ func (r *itemCollector) collectNamespaces(
 		return nil, errors.WithStack(err)
 	}
 
+	for _, includedNSName := range r.backupRequest.Backup.Spec.IncludedNamespaces {
+		nsExists := false
+		// Skip checking the namespace existing when it's "*".
+		if includedNSName == "*" {
+			continue
+		}
+		for _, unstructuredNS := range unstructuredList.Items {
+			if unstructuredNS.GetName() == includedNSName {
+				nsExists = true
+			}
+		}
+
+		if !nsExists {
+			log.Errorf("fail to get the namespace %s specified in backup.Spec.IncludedNamespaces", includedNSName)
+		}
+	}
+
 	var singleSelector labels.Selector
 	var orSelectors []labels.Selector
 
diff --git a/pkg/controller/backup_controller.go b/pkg/controller/backup_controller.go
index 8ccd8db188..34d3d43ee1 100644
--- a/pkg/controller/backup_controller.go
+++ b/pkg/controller/backup_controller.go
@@ -464,7 +464,7 @@ func (b *backupReconciler) prepareBackupRequest(backup *velerov1api.Backup, logg
 	}
 
 	// validate the included/excluded namespaces
-	for _, err := range b.validateNamespaceIncludesExcludes(request.Spec.IncludedNamespaces, request.Spec.ExcludedNamespaces) {
+	for _, err := range collections.ValidateNamespaceIncludesExcludes(request.Spec.IncludedNamespaces, request.Spec.ExcludedNamespaces) {
 		request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid included/excluded namespace lists: %v", err))
 	}
 
@@ -596,24 +596,6 @@ func (b *backupReconciler) validateAndGetSnapshotLocations(backup *velerov1api.B
 	return providerLocations, nil
 }
 
-func (b *backupReconciler) validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces []string) []error {
-	var errs []error
-	if errs = collections.ValidateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces); len(errs) > 0 {
-		return errs
-	}
-
-	namespace := &corev1api.Namespace{}
-	for _, name := range collections.NewIncludesExcludes().Includes(includedNamespaces...).GetIncludes() {
-		if name == "" || name == "*" {
-			continue
-		}
-		if err := b.kbClient.Get(context.Background(), kbclient.ObjectKey{Name: name}, namespace); err != nil {
-			errs = append(errs, err)
-		}
-	}
-	return errs
-}
-
 // runBackup runs and uploads a validated backup. Any error returned from this function
 // causes the backup to be Failed; if no error is returned, the backup's status's Errors
 // field is checked to see if the backup was a partial failure.


Test patch:
diff --git a/pkg/backup/item_collector_test.go b/pkg/backup/item_collector_test.go
index 01c8dd2163..562e6a2c33 100644
--- a/pkg/backup/item_collector_test.go
+++ b/pkg/backup/item_collector_test.go
@@ -224,6 +224,17 @@ func TestItemCollectorBackupNamespaces(t *testing.T) {
 			},
 			expectedTrackedNS: []string{"ns1", "ns2"},
 		},
+		{
+			name:   "ns specified by the IncludeNamespaces cannot be found",
+			backup: builder.ForBackup("velero", "backup").IncludedNamespaces("ns1", "invalid", "*").Result(),
+			ie:     collections.NewIncludesExcludes().Includes("ns1", "invalid", "*"),
+			namespaces: []*corev1.Namespace{
+				builder.ForNamespace("ns1").ObjectMeta(builder.WithLabels("name", "ns1")).Result(),
+				builder.ForNamespace("ns2").Result(),
+				builder.ForNamespace("ns3").Result(),
+			},
+			expectedTrackedNS: []string{"ns1"},
+		},
 	}
 
 	for _, tc := range tests {
diff --git a/pkg/controller/backup_controller_test.go b/pkg/controller/backup_controller_test.go
index 2f613af2cb..46f123f1ab 100644
--- a/pkg/controller/backup_controller_test.go
+++ b/pkg/controller/backup_controller_test.go
@@ -190,16 +190,10 @@ func TestProcessBackupValidationFailures(t *testing.T) {
 		},
 		{
 			name:           "use old filter parameters and new filter parameters together",
-			backup:         defaultBackup().IncludeClusterResources(true).IncludedNamespaceScopedResources("Deployment").IncludedNamespaces("foo").Result(),
+			backup:         defaultBackup().IncludeClusterResources(true).IncludedNamespaceScopedResources("Deployment").IncludedNamespaces("default").Result(),
 			backupLocation: defaultBackupLocation,
 			expectedErrs:   []string{"include-resources, exclude-resources and include-cluster-resources are old filter parameters.\ninclude-cluster-scoped-resources, exclude-cluster-scoped-resources, include-namespace-scoped-resources and exclude-namespace-scoped-resources are new filter parameters.\nThey cannot be used together"},
 		},
-		{
-			name:           "nonexisting namespace",
-			backup:         defaultBackup().IncludedNamespaces("non-existing").Result(),
-			backupLocation: defaultBackupLocation,
-			expectedErrs:   []string{"Invalid included/excluded namespace lists: namespaces \"non-existing\" not found"},
-		},
 	}
 
 	for _, test := range tests {
@@ -214,11 +208,10 @@ func TestProcessBackupValidationFailures(t *testing.T) {
 			require.NoError(t, err)
 
 			var fakeClient kbclient.Client
-			namespace := builder.ForNamespace("foo").Result()
 			if test.backupLocation != nil {
-				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, test.backupLocation, namespace)
+				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, test.backupLocation)
 			} else {
-				fakeClient = velerotest.NewFakeControllerRuntimeClient(t, namespace)
+				fakeClient = velerotest.NewFakeControllerRuntimeClient(t)
 			}
 
 			c := &backupReconciler{
@@ -1574,43 +1567,6 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) {
 	}
 }
 
-func TestValidateNamespaceIncludesExcludes(t *testing.T) {
-	namespace := builder.ForNamespace("default").Result()
-	reconciler := &backupReconciler{
-		kbClient: velerotest.NewFakeControllerRuntimeClient(t, namespace),
-	}
-
-	// empty string as includedNamespaces
-	includedNamespaces := []string{""}
-	excludedNamespaces := []string{"test"}
-	errs := reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-
-	// "*" as includedNamespaces
-	includedNamespaces = []string{"*"}
-	excludedNamespaces = []string{"test"}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-
-	// invalid namespaces
-	includedNamespaces = []string{"1@#"}
-	excludedNamespaces = []string{"2@#"}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Len(t, errs, 2)
-
-	// not exist namespaces
-	includedNamespaces = []string{"non-existing-namespace"}
-	excludedNamespaces = []string{}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Len(t, errs, 1)
-
-	// valid namespaces
-	includedNamespaces = []string{"default"}
-	excludedNamespaces = []string{}
-	errs = reconciler.validateNamespaceIncludesExcludes(includedNamespaces, excludedNamespaces)
-	assert.Empty(t, errs)
-}
-
 // Test_getLastSuccessBySchedule verifies that the getLastSuccessBySchedule helper function correctly returns
 // the completion timestamp of the most recent completed backup for each schedule, including an entry for ad-hoc
 // or non-scheduled backups.