Repository: tektoncd/operator
Pull request #620 (2022-02-09 12:21:35)

Improve TekktonInstallerSet Reconciler
Add explicit check to ensure that resources are present on cluster
after an manifestival.Apply() call

Add DeletePropagation policy 'Foreground' to TektonInstallerSet
finalizer logic

Signed-off-by: Nikhil Thomas <nikthoma@redhat.com>

# Changes

<!-- 🎉🎉🎉 Thank you for the PR!!! 🎉🎉🎉 -->

<!-- Describe your changes here- ideally you can get that description straight from
your descriptive commit message(s)! -->

# Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

- [X] Includes [tests](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if functionality changed/added)
- [ ] Includes [docs](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if user facing)
- [ ] Commit messages follow [commit message best practices](https://github.com/tektoncd/community/blob/master/standards.md#commit-messages)

_See [the contribution guide](https://github.com/tektoncd/operator/blob/master/CONTRIBUTING.md) for more details._

# Release Notes

<!--
Describe any user facing changes here, or delete this block.

Examples of user facing changes:
- API changes
- Bug fixes
- Any changes in behavior
- Changes requiring upgrade notices or deprecation warnings

For pull requests with a release note:

```release-note
Your release note here
```

For pull requests that require additional action from users switching to the new release, include the string "action required" (case insensitive) in the release note:

```release-note
action required: your release note here
```

For pull requests that don't need to be mentioned at release time, use the `/release-note-none` Prow command to add the `release-note-none` label to the PR. You can also write the string "NONE" as a release note in your PR description:

```release-note
NONE
```
-->
```release-note
NONE
```

tekton-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **nikhil-thomas**
You can assign the PR to them by writing `/assign @nikhil-thomas` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=tektoncd%2Foperator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/tektoncd/operator/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["nikhil-thomas"]} -->
tekton-robot: @nikhil-thomas: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-unit-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-unit-tests/1491386877427585026/) | `/test pull-tekton-operator-unit-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/620). [Your PR dashboard](https://prow.tekton.dev/pr/nikhil-thomas).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: @nikhil-thomas: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-unit-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-unit-tests/1491386877427585026/) | `/test pull-tekton-operator-unit-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/620). [Your PR dashboard](https://prow.tekton.dev/pr/nikhil-thomas).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: @nikhil-thomas: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-unit-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-unit-tests/1491386877427585026/) | `/test pull-tekton-operator-unit-tests`
pull-tekton-operator-build-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-build-tests/1491386877427585027/) | `/test pull-tekton-operator-build-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/620). [Your PR dashboard](https://prow.tekton.dev/pr/nikhil-thomas).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: @nikhil-thomas: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-unit-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-unit-tests/1491386877427585026/) | `/test pull-tekton-operator-unit-tests`
pull-tekton-operator-build-tests | 7ce712d68cbc210c9490b46be336ccbfc4e4cf58 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-build-tests/1491386877427585027/) | `/test pull-tekton-operator-build-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/620). [Your PR dashboard](https://prow.tekton.dev/pr/nikhil-thomas).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: The following is the coverage report on the affected files.
Say `/test pull-tekton-operator-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/reconciler/kubernetes/tektoninstallerset/install.go](https://storage.cloud.google.com/tekton-prow/pr-logs/pull/tektoncd_operator/620/pull-tekton-operator-go-coverage/1491415944575062017/artifacts/line-cov.html#file0) | 54.7% | 76.5% | 21.8

tekton-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/tektoncd/operator/pull/620#pullrequestreview-877735891" title="Approved">vdemeester</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=tektoncd%2Foperator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/tektoncd/operator/blob/main/OWNERS)~~ [vdemeester]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
concaf: /lgtm

Patch:
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/install.go b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
index 52ae315dc..a9098a554 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/install.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
@@ -71,15 +71,44 @@ type installer struct {
 	Manifest mf.Manifest
 }
 
-func (i *installer) EnsureCRDs() error {
-	if err := i.Manifest.Filter(mf.Any(mf.CRDs)).Apply(); err != nil {
+func ensureResources(mani *mf.Manifest) error {
+	freshCreate := true
+	// check if all resources in a given set of resources exists
+	ok, err := allResourcesExists(mani)
+	if err != nil {
+		if !apierrs.IsNotFound(err) {
+			return err
+		}
+	}
+	// if error == NotFound error or !ok
+	// then Apply all resources
+	// if ok, that means the reosource already exists
+	// but this reconcile could be an modification eg: concig-defaults change
+	// set freshCreate flag to false, and then Apply all
+	// so that we can skip (break) RECONCILE_AGAIN loop path
+	if ok {
+		freshCreate = false
+	}
+
+	if err := mani.Apply(); err != nil {
 		return err
 	}
+	// on err == nil after Apply() return RECONCILE_AGAIN
+	// if freshCreate == true return RECONCILE_AGAIN
+	// this ensures freshly created resources are in place before proceeding to next stages of reconciler
+	if freshCreate {
+		return v1alpha1.RECONCILE_AGAIN_ERR
+	}
 	return nil
 }
 
+func (i *installer) EnsureCRDs() error {
+	resourceList := i.Manifest.Filter(mf.Any(mf.CRDs))
+	return ensureResources(&resourceList)
+}
+
 func (i *installer) EnsureClusterScopedResources() error {
-	if err := i.Manifest.Filter(
+	resourceList := i.Manifest.Filter(
 		mf.Any(
 			namespacePred,
 			clusterRolePred,
@@ -92,14 +121,12 @@ func (i *installer) EnsureClusterScopedResources() error {
 			consoleQuickStartPred,
 			ConsoleYAMLSamplePred,
 			securityContextConstraints,
-		)).Apply(); err != nil {
-		return err
-	}
-	return nil
+		))
+	return ensureResources(&resourceList)
 }
 
 func (i *installer) EnsureNamespaceScopedResources() error {
-	if err := i.Manifest.Filter(
+	resourceList := i.Manifest.Filter(
 		mf.Any(
 			serviceAccountPred,
 			clusterRoleBindingPred,
@@ -114,10 +141,10 @@ func (i *installer) EnsureNamespaceScopedResources() error {
 			eventListenerPred,
 			triggerBindingPred,
 			triggerTemplatePred,
-		)).Apply(); err != nil {
-		return err
-	}
-	return nil
+			servicePred,
+			routePred,
+		))
+	return ensureResources(&resourceList)
 }
 
 func (i *installer) EnsureDeploymentResources() error {
@@ -128,13 +155,6 @@ func (i *installer) EnsureDeploymentResources() error {
 		}
 	}
 
-	if err := i.Manifest.Filter(
-		mf.Any(
-			servicePred,
-			routePred,
-		)).Apply(); err != nil {
-		return err
-	}
 	return nil
 }
 
@@ -202,7 +222,11 @@ func (i *installer) updateDeployment(existing *unstructured.Unstructured, existi
 	}
 	existing.SetUnstructuredContent(unstrObj)
 
-	return i.Manifest.Client.Update(existing)
+	err = i.Manifest.Client.Update(existing)
+	if err != nil {
+		return v1alpha1.RECONCILE_AGAIN_ERR
+	}
+	return err
 }
 
 func (i *installer) ensureDeployment(expected *unstructured.Unstructured) error {
@@ -213,7 +237,11 @@ func (i *installer) ensureDeployment(expected *unstructured.Unstructured) error
 
 		// If deployment doesn't exist, then create new
 		if apierrs.IsNotFound(err) {
-			return i.createDeployment(expected)
+			errInner := i.createDeployment(expected)
+			if errInner == nil {
+				return v1alpha1.RECONCILE_AGAIN_ERR
+			}
+			return errInner
 		}
 		return err
 	}
@@ -348,3 +376,25 @@ func isDeploymentAvailable(d *appsv1.Deployment) bool {
 	}
 	return false
 }
+
+func allResourcesExists(m *mf.Manifest) (bool, error) {
+	c := m.Client
+	for _, item := range m.Resources() {
+		ok, err := resourceExists(c, &item)
+		if err != nil {
+			return false, err
+		}
+		if !ok {
+			return false, nil
+		}
+	}
+	return true, nil
+}
+
+func resourceExists(c mf.Client, u *unstructured.Unstructured) (bool, error) {
+	_, err := c.Get(u)
+	if err != nil {
+		return false, err
+	}
+	return true, nil
+}
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go b/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
index 373de1be4..1c2b256fc 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
@@ -57,7 +57,8 @@ func (r *Reconciler) FinalizeKind(ctx context.Context, installerSet *v1alpha1.Te
 	// Delete all resources except CRDs and Namespace as they are own by owner of
 	// TektonInstallerSet
 	// They will be deleted when the component CR is deleted
-	err = deleteManifests.Filter(mf.Not(mf.Any(namespacePred, mf.CRDs))).Delete()
+	deleteManifests = deleteManifests.Filter(mf.Not(mf.Any(namespacePred, mf.CRDs)))
+	err = deleteManifests.Delete(mf.PropagationPolicy(v1.DeletePropagationForeground))
 	if err != nil {
 		logger.Error("failed to delete resources")
 		return err
@@ -107,7 +108,7 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 	err = installer.EnsureCRDs()
 	if err != nil {
 		installerSet.Status.MarkCRDsInstallationFailed(err.Error())
-		return err
+		return r.handleError(err, installerSet)
 	}
 
 	// Update Status for CRD condition
@@ -117,7 +118,7 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 	err = installer.EnsureClusterScopedResources()
 	if err != nil {
 		installerSet.Status.MarkClustersScopedInstallationFailed(err.Error())
-		return err
+		return r.handleError(err, installerSet)
 	}
 
 	// Update Status for ClustersScope Condition
@@ -127,7 +128,7 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 	err = installer.EnsureNamespaceScopedResources()
 	if err != nil {
 		installerSet.Status.MarkNamespaceScopedInstallationFailed(err.Error())
-		return err
+		return r.handleError(err, installerSet)
 	}
 
 	// Update Status for NamespaceScope Condition
@@ -137,7 +138,7 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 	err = installer.EnsureDeploymentResources()
 	if err != nil {
 		installerSet.Status.MarkDeploymentsAvailableFailed(err.Error())
-		return err
+		return r.handleError(err, installerSet)
 	}
 
 	// Update Status for Deployment Resources
@@ -179,3 +180,11 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 
 	return nil
 }
+
+func (r *Reconciler) handleError(err error, installerSet *v1alpha1.TektonInstallerSet) error {
+	if err == v1alpha1.RECONCILE_AGAIN_ERR {
+		r.enqueueAfter(installerSet, 10*time.Second)
+		return nil
+	}
+	return err
+}


Test patch:
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go b/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
index 5fd96fdf8..b29581b67 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
@@ -23,6 +23,8 @@ import (
 	"github.com/google/go-cmp/cmp"
 	mf "github.com/manifestival/manifestival"
 	"github.com/manifestival/manifestival/fake"
+	"github.com/tektoncd/operator/pkg/apis/operator/v1alpha1"
+	"gotest.tools/v3/assert"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -54,6 +56,7 @@ type fakeClient struct {
 	getErr         error
 	createErr      error
 	resourcesExist bool
+	gets           []unstructured.Unstructured
 	creates        []unstructured.Unstructured
 	deletes        []unstructured.Unstructured
 }
@@ -61,7 +64,12 @@ type fakeClient struct {
 func (f *fakeClient) Get(obj *unstructured.Unstructured) (*unstructured.Unstructured, error) {
 	var resource *unstructured.Unstructured
 	if f.resourcesExist {
-		resource = &unstructured.Unstructured{}
+		for _, item := range f.gets {
+			if obj.GetKind() == item.GetKind() && obj.GetName() == item.GetName() {
+				return &item, nil
+			}
+		}
+
 	}
 	return resource, f.getErr
 }
@@ -97,7 +105,7 @@ func clusterScopedResource(apiVersion, kind, name string) unstructured.Unstructu
 }
 
 func TestInstaller(t *testing.T) {
-
+	crd.SetDeletionTimestamp(&metav1.Time{})
 	in := []unstructured.Unstructured{namespace, deployment, clusterRole, role,
 		roleBinding, clusterRoleBinding, serviceAccount, crd, validatingWebhook, mutatingWebhook, configMap, service, hpa, secret}
 
@@ -140,7 +148,7 @@ func TestInstaller(t *testing.T) {
 	client.creates = []unstructured.Unstructured{}
 
 	want = []unstructured.Unstructured{serviceAccount, clusterRoleBinding, role,
-		roleBinding, configMap, secret, hpa}
+		roleBinding, configMap, secret, hpa, service}
 
 	err = i.EnsureNamespaceScopedResources()
 	if err != nil {
@@ -159,17 +167,20 @@ func TestInstaller(t *testing.T) {
 		Resource: "Deployment",
 	}, "test-deployment")
 
-	want = []unstructured.Unstructured{deployment, service}
-
 	err = i.EnsureDeploymentResources()
-	if err != nil {
-		t.Fatal("Unexpected Error while installing resources: ", err)
-	}
+	assert.Error(t, err, v1alpha1.RECONCILE_AGAIN_ERR.Error())
 
+	want = []unstructured.Unstructured{deployment}
 	if len(want) != len(client.creates) {
 		t.Fatalf("Unexpected creates: %s", fmt.Sprintf("(-got, +want): %s", cmp.Diff(client.creates, want)))
 	}
 
+	client.resourcesExist = true
+	client.gets = []unstructured.Unstructured{deployment}
+	err = i.EnsureDeploymentResources()
+	if err != nil {
+		t.Fatal("Unexpected Error while installing resources: ", err)
+	}
 }
 
 var (