Repository: tektoncd/operator
Pull request #614 (2022-02-04 07:40:55)

Update labels of tekton dashboard installersets
# Changes
1. Update labels and annotation while creating installersets
2. Fetch installersets by labelselector
3. Add unit test to verify installersets based on labelselector

<!-- 🎉🎉🎉 Thank you for the PR!!! 🎉🎉🎉 -->

<!-- Describe your changes here- ideally you can get that description straight from
your descriptive commit message(s)! -->

# Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

- [ ] Includes [tests](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if functionality changed/added)
- [ ] Includes [docs](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if user facing)
- [ ] Commit messages follow [commit message best practices](https://github.com/tektoncd/community/blob/master/standards.md#commit-messages)

_See [the contribution guide](https://github.com/tektoncd/operator/blob/master/CONTRIBUTING.md) for more details._

# Release Notes

<!--
Describe any user facing changes here, or delete this block.

Examples of user facing changes:
- API changes
- Bug fixes
- Any changes in behavior
- Changes requiring upgrade notices or deprecation warnings

For pull requests with a release note:

```release-note
Your release note here
```

For pull requests that require additional action from users switching to the new release, include the string "action required" (case insensitive) in the release note:

```release-note
action required: your release note here
```

For pull requests that don't need to be mentioned at release time, use the `/release-note-none` Prow command to add the `release-note-none` label to the PR. You can also write the string "NONE" as a release note in your PR description:

```release-note
NONE
```
-->
```release-note
NONE
```

tekton-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **savitaashture**
You can assign the PR to them by writing `/assign @savitaashture` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=tektoncd%2Foperator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/tektoncd/operator/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["savitaashture"]} -->
tekton-robot: @savitaashture: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-integration-tests | 684472a6bc2406469ef61d31bce6dc206897b136 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/614/pull-tekton-operator-integration-tests/1489504349016035328/) | `/test pull-tekton-operator-integration-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/614). [Your PR dashboard](https://prow.tekton.dev/pr/savitaashture).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: @savitaashture: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-tekton-operator-integration-tests | 684472a6bc2406469ef61d31bce6dc206897b136 | [link](https://tekton-releases.appspot.com/build/tekton-prow/pr-logs/pull/tektoncd_operator/614/pull-tekton-operator-integration-tests/1489504349016035328/) | `/test pull-tekton-operator-integration-tests`

[Full PR test history](https://tekton-releases.appspot.com/pr/tektoncd_operator/614). [Your PR dashboard](https://prow.tekton.dev/pr/savitaashture).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/Makefile b/Makefile
index 96f4e0e87..4b7c302f3 100644
--- a/Makefile
+++ b/Makefile
@@ -34,6 +34,7 @@ TEKTON_PIPELINE_VERSION ?= latest
 TEKTON_TRIGGERS_VERSION ?= latest
 TEKTON_DASHBOARD_VERSION ?= latest
 TEKTON_RESULTS_VERSION ?= v0.3.1 # latest returns an older version hence hard coding to v0.3.1 for now (tektoncd/results#138)
+PAC_VERSION ?= 0.5.2
 
 $(BIN)/ko: PACKAGE=github.com/google/ko/cmd/ko
 
@@ -87,7 +88,7 @@ bin/%: cmd/% FORCE
 
 .PHONY: get-releases
 get-releases: |
-	$Q ./hack/fetch-releases.sh $(TARGET) $(TEKTON_PIPELINE_VERSION) $(TEKTON_TRIGGERS_VERSION) $(TEKTON_DASHBOARD_VERSION) $(TEKTON_RESULTS_VERSION) || exit ;
+	$Q ./hack/fetch-releases.sh $(TARGET) $(TEKTON_PIPELINE_VERSION) $(TEKTON_TRIGGERS_VERSION) $(TEKTON_DASHBOARD_VERSION) $(TEKTON_RESULTS_VERSION) $(PAC_VERSION) || exit ;
 
 .PHONY: apply
 apply: | $(KO) $(KUSTOMIZE) get-releases ; $(info $(M) ko apply on $(TARGET)) @ ## Apply config to the current cluster
diff --git a/docs/TektonAddon.md b/docs/TektonAddon.md
index 139ec3fbc..d28641ec3 100644
--- a/docs/TektonAddon.md
+++ b/docs/TektonAddon.md
@@ -25,6 +25,7 @@ spec:
     value: "true"
   - name: pipelineTemplates
     value: "true"
+  enablePipelinesAsCode: true
 ```
 You can install this component using [TektonConfig](./TektonConfig.md) by choosing appropriate `profile`.
 
@@ -40,3 +41,8 @@ User can disable the installation of resources by changing the value to `false`.
 
 Pipelines templates uses clustertasks in them so to install pipelineTemplates, clusterTasks must be `true`.
 
+### PipelinesAsCode
+
+`enablePipelinesAsCode` field is provided in spec to enable/disable PipelinesAsCode installation on the cluster.
+
+User can disable the installation of PipelinesAsCode by changing the value to `false`. By default, it is true.
diff --git a/hack/fetch-releases.sh b/hack/fetch-releases.sh
index 55b21b5b0..9e9f35f7a 100755
--- a/hack/fetch-releases.sh
+++ b/hack/fetch-releases.sh
@@ -93,7 +93,34 @@ release_yaml() {
     echo ""
 }
 
-#Args: <target-platform> <pipelines version> <triggers version> <dashboard version> <results version>
+# release_yaml_pac <component> <release-yaml-name> <version>
+release_yaml_pac() {
+    echo fetching '|' component: ${1} '|' file: ${2} '|' version: ${3}
+    local comp=$1
+    local fileName=$2
+    local version=$3
+
+    ko_data=${SCRIPT_DIR}/cmd/${TARGET}/operator/kodata
+    dirPath=${ko_data}/tekton-addon/pipelines-as-code/${version}
+    rm -rf ${dirPath} || true
+    mkdir -p ${dirPath} || true
+
+    url="https://raw.githubusercontent.com/openshift-pipelines/pipelines-as-code/release-${version}/release-${version}.yaml"
+
+    dest=${dirPath}/${fileName}.yaml
+    http_response=$(curl -s -o ${dest} -w "%{http_code}" ${url})
+    echo url: ${url}
+
+    if [[ $http_response != "200" ]]; then
+        echo "Error: failed to get $comp yaml, status code: $http_response"
+        exit 1
+    fi
+
+    echo "Info: Added $comp/$fileName:$version release yaml !!"
+    echo ""
+}
+
+#Args: <target-platform> <pipelines version> <triggers version> <dashboard version> <results version> <pac version>
 main() {
   TARGET=$1
   p_version=${2}
@@ -112,6 +139,9 @@ main() {
 
     r_version=${5}
     release_yaml results release 00-results ${r_version}
+  else
+    pac_version=${6}
+    release_yaml_pac pipelinesascode release ${pac_version}
   fi
 }
 
diff --git a/pkg/apis/operator/v1alpha1/tektonaddon_defaults.go b/pkg/apis/operator/v1alpha1/tektonaddon_defaults.go
index 9e71f956d..e5820ee54 100644
--- a/pkg/apis/operator/v1alpha1/tektonaddon_defaults.go
+++ b/pkg/apis/operator/v1alpha1/tektonaddon_defaults.go
@@ -18,33 +18,35 @@ package v1alpha1
 
 import (
 	"context"
+
+	"knative.dev/pkg/ptr"
 )
 
 func (ta *TektonAddon) SetDefaults(ctx context.Context) {
-	setAddonDefaults(&ta.Spec.Params)
+	setAddonDefaults(&ta.Spec.Addon)
 }
 
-func setAddonDefaults(params *[]Param) {
+func setAddonDefaults(addon *Addon) {
 
-	paramsMap := ParseParams(*params)
+	paramsMap := ParseParams(addon.Params)
 	_, ptOk := paramsMap[PipelineTemplatesParam]
 	ct, ctOk := paramsMap[ClusterTasksParam]
 
 	// If clusterTasks is false and pipelineTemplate is not set, then set it as false
 	// as pipelines templates are created using clusterTasks
 	if ctOk && (ct == "false" && !ptOk) {
-		*params = append(*params, Param{
+		addon.Params = append(addon.Params, Param{
 			Name:  PipelineTemplatesParam,
 			Value: "false",
 		})
-		paramsMap = ParseParams(*params)
+		paramsMap = ParseParams(addon.Params)
 	}
 
 	// set the params with default values if not set in cr
 	for d := range AddonParams {
 		_, ok := paramsMap[d]
 		if !ok {
-			*params = append(*params,
+			addon.Params = append(addon.Params,
 				Param{
 					Name:  d,
 					Value: AddonParams[d].Default,
@@ -52,4 +54,8 @@ func setAddonDefaults(params *[]Param) {
 		}
 	}
 
+	// by default enable pac
+	if addon.EnablePAC == nil {
+		addon.EnablePAC = ptr.Bool(true)
+	}
 }
diff --git a/pkg/apis/operator/v1alpha1/tektonaddon_types.go b/pkg/apis/operator/v1alpha1/tektonaddon_types.go
index 8f9ba9edf..1157be7df 100644
--- a/pkg/apis/operator/v1alpha1/tektonaddon_types.go
+++ b/pkg/apis/operator/v1alpha1/tektonaddon_types.go
@@ -47,9 +47,7 @@ func (tp *TektonAddon) GetStatus() TektonComponentStatus {
 // TektonAddonSpec defines the desired state of TektonAddon
 type TektonAddonSpec struct {
 	CommonSpec `json:",inline"`
-	// The params to customize different components of Addon
-	// +optional
-	Params []Param `json:"params,omitempty"`
+	Addon      `json:",inline"`
 	// Config holds the configuration for resources created by Addon
 	// +optional
 	Config Config `json:"config,omitempty"`
@@ -73,6 +71,9 @@ type Addon struct {
 	// Params is the list of params passed for Addon customization
 	// +optional
 	Params []Param `json:"params,omitempty"`
+	// EnablePAC field defines whether to install PAC
+	// +optional
+	EnablePAC *bool `json:"enablePipelinesAsCode,omitempty"`
 }
 
 func (a Addon) IsEmpty() bool {
diff --git a/pkg/apis/operator/v1alpha1/tektonconfig_defaults.go b/pkg/apis/operator/v1alpha1/tektonconfig_defaults.go
index 9e6bb8350..95e94eb3d 100644
--- a/pkg/apis/operator/v1alpha1/tektonconfig_defaults.go
+++ b/pkg/apis/operator/v1alpha1/tektonconfig_defaults.go
@@ -28,7 +28,7 @@ func (tc *TektonConfig) SetDefaults(ctx context.Context) {
 	tc.Spec.Pipeline.PipelineProperties.setDefaults()
 	tc.Spec.Trigger.TriggersProperties.setDefaults()
 
-	setAddonDefaults(&tc.Spec.Addon.Params)
+	setAddonDefaults(&tc.Spec.Addon)
 
 	// before adding webhook we had default value for pruner's keep as 1
 	// but we expect user to define all values now otherwise webhook reject
diff --git a/pkg/apis/operator/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/operator/v1alpha1/zz_generated.deepcopy.go
index 7ecb83e85..47f589c71 100644
--- a/pkg/apis/operator/v1alpha1/zz_generated.deepcopy.go
+++ b/pkg/apis/operator/v1alpha1/zz_generated.deepcopy.go
@@ -35,6 +35,11 @@ func (in *Addon) DeepCopyInto(out *Addon) {
 		*out = make([]Param, len(*in))
 		copy(*out, *in)
 	}
+	if in.EnablePAC != nil {
+		in, out := &in.EnablePAC, &out.EnablePAC
+		*out = new(bool)
+		**out = **in
+	}
 	return
 }
 
@@ -419,11 +424,7 @@ func (in *TektonAddonList) DeepCopyObject() runtime.Object {
 func (in *TektonAddonSpec) DeepCopyInto(out *TektonAddonSpec) {
 	*out = *in
 	out.CommonSpec = in.CommonSpec
-	if in.Params != nil {
-		in, out := &in.Params, &out.Params
-		*out = make([]Param, len(*in))
-		copy(*out, *in)
-	}
+	in.Addon.DeepCopyInto(&out.Addon)
 	in.Config.DeepCopyInto(&out.Config)
 	return
 }
diff --git a/pkg/reconciler/kubernetes/tektondashboard/tektondashboard.go b/pkg/reconciler/kubernetes/tektondashboard/tektondashboard.go
index 8ddc8e023..2e2273602 100644
--- a/pkg/reconciler/kubernetes/tektondashboard/tektondashboard.go
+++ b/pkg/reconciler/kubernetes/tektondashboard/tektondashboard.go
@@ -27,6 +27,7 @@ import (
 	pipelineinformer "github.com/tektoncd/operator/pkg/client/informers/externalversions/operator/v1alpha1"
 	tektondashboardreconciler "github.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektondashboard"
 	"github.com/tektoncd/operator/pkg/reconciler/common"
+	"github.com/tektoncd/operator/pkg/reconciler/kubernetes/tektoninstallerset"
 	"github.com/tektoncd/operator/pkg/reconciler/shared/hash"
 	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -62,7 +63,14 @@ type Reconciler struct {
 var _ tektondashboardreconciler.Interface = (*Reconciler)(nil)
 var _ tektondashboardreconciler.Finalizer = (*Reconciler)(nil)
 
-var watchedResourceName = "dashboard"
+var (
+	ls = metav1.LabelSelector{
+		MatchLabels: map[string]string{
+			v1alpha1.CreatedByKey:     createdByValue,
+			v1alpha1.InstallerSetType: v1alpha1.DashboardResourceName,
+		},
+	}
+)
 
 const createdByValue = "TektonDashboard"
 
@@ -86,11 +94,6 @@ func (r *Reconciler) FinalizeKind(ctx context.Context, original *v1alpha1.Tekton
 		return err
 	}
 
-	ls := metav1.LabelSelector{
-		MatchLabels: map[string]string{
-			v1alpha1.CreatedByKey: createdByValue,
-		},
-	}
 	labelSelector, err := common.LabelSelector(ls)
 	if err != nil {
 		return err
@@ -119,9 +122,9 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, td *v1alpha1.TektonDashb
 
 	logger.Infow("Reconciling TektonDashboards", "status", td.Status)
 
-	if td.GetName() != watchedResourceName {
+	if td.GetName() != v1alpha1.DashboardResourceName {
 		msg := fmt.Sprintf("Resource ignored, Expected Name: %s, Got Name: %s",
-			watchedResourceName,
+			v1alpha1.DashboardResourceName,
 			td.GetName(),
 		)
 		logger.Error(msg)
@@ -152,11 +155,16 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, td *v1alpha1.TektonDashb
 	// Mark PreReconcile Complete
 	td.Status.MarkPreReconcilerComplete()
 
-	// Check if an tekton installer set already exists, if not then create
-	existingInstallerSet := td.Status.GetTektonInstallerSet()
+	// Check if an tektoninstallerset already exists, if not then create
+	labelSelector, err := common.LabelSelector(ls)
+	if err != nil {
+		return err
+	}
+	existingInstallerSet, err := tektoninstallerset.CurrentInstallerSetName(ctx, r.operatorClientSet, labelSelector)
+	if err != nil {
+		return err
+	}
 	if existingInstallerSet == "" {
-		td.Status.MarkInstallerSetNotAvailable("Dashboard Installer Set Not Available")
-
 		createdIs, err := r.createInstallerSet(ctx, td)
 		if err != nil {
 			return err
@@ -181,7 +189,7 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, td *v1alpha1.TektonDashb
 	}
 
 	installerSetTargetNamespace := installedTIS.Annotations[v1alpha1.TargetNamespaceKey]
-	installerSetReleaseVersion := installedTIS.Annotations[v1alpha1.ReleaseVersionKey]
+	installerSetReleaseVersion := installedTIS.Labels[v1alpha1.ReleaseVersionKey]
 
 	// Check if TargetNamespace of existing TektonInstallerSet is same as expected
 	// Check if Release Version in TektonInstallerSet is same as expected
@@ -344,7 +352,7 @@ func (r *Reconciler) createInstallerSet(ctx context.Context, td *v1alpha1.Tekton
 	}
 
 	// create installer set
-	tis := makeInstallerSet(td, manifest, specHash, r.operatorVersion)
+	tis := r.makeInstallerSet(td, manifest, specHash)
 	createdIs, err := r.operatorClientSet.OperatorV1alpha1().TektonInstallerSets().
 		Create(ctx, tis, metav1.CreateOptions{})
 	if err != nil {
@@ -353,16 +361,17 @@ func (r *Reconciler) createInstallerSet(ctx context.Context, td *v1alpha1.Tekton
 	return createdIs, nil
 }
 
-func makeInstallerSet(td *v1alpha1.TektonDashboard, manifest mf.Manifest, tdSpecHash, releaseVersion string) *v1alpha1.TektonInstallerSet {
+func (r *Reconciler) makeInstallerSet(td *v1alpha1.TektonDashboard, manifest mf.Manifest, tdSpecHash string) *v1alpha1.TektonInstallerSet {
 	ownerRef := *metav1.NewControllerRef(td, td.GetGroupVersionKind())
 	return &v1alpha1.TektonInstallerSet{
 		ObjectMeta: metav1.ObjectMeta{
 			GenerateName: fmt.Sprintf("%s-", v1alpha1.DashboardResourceName),
 			Labels: map[string]string{
-				v1alpha1.CreatedByKey: createdByValue,
+				v1alpha1.CreatedByKey:      createdByValue,
+				v1alpha1.InstallerSetType:  v1alpha1.DashboardResourceName,
+				v1alpha1.ReleaseVersionKey: r.operatorVersion,
 			},
 			Annotations: map[string]string{
-				v1alpha1.ReleaseVersionKey:  releaseVersion,
 				v1alpha1.TargetNamespaceKey: td.Spec.TargetNamespace,
 				v1alpha1.LastAppliedHashKey: tdSpecHash,
 			},
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/install.go b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
index fec796a29..52ae315dc 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/install.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
@@ -42,6 +42,10 @@ var (
 	deploymentPred                     = mf.ByKind("Deployment")
 	servicePred                        = mf.ByKind("Service")
 	serviceAccountPred                 = mf.ByKind("ServiceAccount")
+	cronJobPred                        = mf.ByKind("CronJob")
+	eventListenerPred                  = mf.ByKind("EventListener")
+	triggerBindingPred                 = mf.ByKind("TriggerBinding")
+	triggerTemplatePred                = mf.ByKind("TriggerTemplate")
 	rolePred                           = mf.ByKind("Role")
 	roleBindingPred                    = mf.ByKind("RoleBinding")
 	clusterRolePred                    = mf.ByKind("ClusterRole")
@@ -106,6 +110,10 @@ func (i *installer) EnsureNamespaceScopedResources() error {
 			horizontalPodAutoscalerPred,
 			pipelinePred,
 			serviceMonitorPred,
+			cronJobPred,
+			eventListenerPred,
+			triggerBindingPred,
+			triggerTemplatePred,
 		)).Apply(); err != nil {
 		return err
 	}
diff --git a/pkg/reconciler/openshift/tektonaddon/const.go b/pkg/reconciler/openshift/tektonaddon/const.go
index c82b9e19f..196cb7381 100644
--- a/pkg/reconciler/openshift/tektonaddon/const.go
+++ b/pkg/reconciler/openshift/tektonaddon/const.go
@@ -23,5 +23,6 @@ const (
 	TriggersResourcesInstallerSet      = "TriggersResources"
 	ConsoleCLIInstallerSet             = "ConsoleCLI"
 	MiscellaneousResourcesInstallerSet = "MiscellaneousResources"
+	PACInstallerSet                    = "PipelinesAsCode"
 	CreatedByValue                     = "TektonAddon"
 )
diff --git a/pkg/reconciler/openshift/tektonaddon/tektonaddon.go b/pkg/reconciler/openshift/tektonaddon/tektonaddon.go
index 38cff2dfb..0c12746e9 100644
--- a/pkg/reconciler/openshift/tektonaddon/tektonaddon.go
+++ b/pkg/reconciler/openshift/tektonaddon/tektonaddon.go
@@ -322,6 +322,27 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, ta *v1alpha1.TektonAddon
 		return nil
 	}
 
+	// Check if PAC is enabled
+
+	if *ta.Spec.EnablePAC {
+
+		// make sure pac is installed
+		exist, err := checkIfInstallerSetExist(ctx, r.operatorClientSet, r.operatorVersion,
+			fmt.Sprintf("%s=%s", v1alpha1.InstallerSetType, PACInstallerSet))
+		if err != nil {
+			return err
+		}
+		if !exist {
+			return r.ensurePAC(ctx, ta)
+		}
+
+	} else {
+		// if disabled then delete the installer Set if exist
+		if err := r.deleteInstallerSet(ctx, fmt.Sprintf("%s=%s", v1alpha1.InstallerSetType, PACInstallerSet)); err != nil {
+			return err
+		}
+	}
+
 	ta.Status.MarkInstallerSetReady()
 
 	if err := r.extension.PostReconcile(ctx, ta); err != nil {
@@ -367,8 +388,30 @@ func (r *Reconciler) checkComponentStatus(ctx context.Context, labelSelector str
 	return nil
 }
 
+func (r *Reconciler) ensurePAC(ctx context.Context, ta *v1alpha1.TektonAddon) error {
+	pacManifest := mf.Manifest{}
+
+	koDataDir := os.Getenv(common.KoEnvKey)
+	pacLocation := filepath.Join(koDataDir, "tekton-addon", "pipelines-as-code")
+	if err := common.AppendManifest(&pacManifest, pacLocation); err != nil {
+		return err
+	}
+
+	// Run transformers
+	if err := r.addonTransform(ctx, &pacManifest, ta); err != nil {
+		return err
+	}
+
+	if err := createInstallerSet(ctx, r.operatorClientSet, ta, pacManifest, r.operatorVersion,
+		PACInstallerSet, "addon-pac"); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 func (r *Reconciler) ensureTriggerResources(ctx context.Context, ta *v1alpha1.TektonAddon) error {
-	triggerResourcesManifest := r.manifest
+	triggerResourcesManifest := mf.Manifest{}
 
 	if err := applyAddons(&triggerResourcesManifest, "01-clustertriggerbindings"); err != nil {
 		return err
@@ -387,7 +430,7 @@ func (r *Reconciler) ensureTriggerResources(ctx context.Context, ta *v1alpha1.Te
 }
 
 func (r *Reconciler) ensurePipelineTemplates(ctx context.Context, ta *v1alpha1.TektonAddon) error {
-	pipelineTemplateManifest := r.manifest
+	pipelineTemplateManifest := mf.Manifest{}
 
 	// Read pipeline template manifest from kodata
 	if err := applyAddons(&pipelineTemplateManifest, "03-pipelines"); err != nil {
@@ -414,7 +457,7 @@ func (r *Reconciler) ensurePipelineTemplates(ctx context.Context, ta *v1alpha1.T
 
 // installerset for non versioned clustertask like buildah and community clustertask
 func (r *Reconciler) ensureClusterTasks(ctx context.Context, ta *v1alpha1.TektonAddon) error {
-	clusterTaskManifest := r.manifest
+	clusterTaskManifest := mf.Manifest{}
 	// Read clusterTasks from ko data
 	if err := applyAddons(&clusterTaskManifest, "02-clustertasks"); err != nil {
 		return err
@@ -451,7 +494,7 @@ func (r *Reconciler) ensureClusterTasks(ctx context.Context, ta *v1alpha1.Tekton
 
 // installerset for versioned clustertask like buildah-1-6-0
 func (r *Reconciler) ensureVersionedClusterTasks(ctx context.Context, ta *v1alpha1.TektonAddon) error {
-	clusterTaskManifest := r.manifest
+	clusterTaskManifest := mf.Manifest{}
 	// Read clusterTasks from ko data
 	if err := applyAddons(&clusterTaskManifest, "02-clustertasks"); err != nil {
 		return err
diff --git a/pkg/reconciler/openshift/tektonconfig/extension/addon.go b/pkg/reconciler/openshift/tektonconfig/extension/addon.go
index 17b6af229..4e5e45ee2 100644
--- a/pkg/reconciler/openshift/tektonconfig/extension/addon.go
+++ b/pkg/reconciler/openshift/tektonconfig/extension/addon.go
@@ -57,8 +57,8 @@ func ensureTektonAddonExists(ctx context.Context, clients op.TektonAddonInterfac
 			updated = true
 		}
 
-		if !reflect.DeepEqual(config.Spec.Addon.Params, taCR.Spec.Params) {
-			taCR.Spec.Params = config.Spec.Addon.Params
+		if !reflect.DeepEqual(config.Spec.Addon, taCR.Spec.Addon) {
+			taCR.Spec.Addon = config.Spec.Addon
 			updated = true
 		}
 
@@ -92,7 +92,9 @@ func ensureTektonAddonExists(ctx context.Context, clients op.TektonAddonInterfac
 				CommonSpec: v1alpha1.CommonSpec{
 					TargetNamespace: config.Spec.TargetNamespace,
 				},
-				Params: config.Spec.Addon.Params,
+				Addon: v1alpha1.Addon{
+					Params: config.Spec.Addon.Params,
+				},
 				Config: config.Spec.Config,
 			},
 		}
diff --git a/third_party/github.com/hashicorp/errwrap/LICENSE b/third_party/github.com/hashicorp/errwrap/LICENSE
new file mode 100644
index 000000000..c33dcc7c9
--- /dev/null
+++ b/third_party/github.com/hashicorp/errwrap/LICENSE
@@ -0,0 +1,354 @@
+Mozilla Public License, version 2.0
+
+1. Definitions
+
+1.1. “Contributor”
+
+     means each individual or legal entity that creates, contributes to the
+     creation of, or owns Covered Software.
+
+1.2. “Contributor Version”
+
+     means the combination of the Contributions of others (if any) used by a
+     Contributor and that particular Contributor’s Contribution.
+
+1.3. “Contribution”
+
+     means Covered Software of a particular Contributor.
+
+1.4. “Covered Software”
+
+     means Source Code Form to which the initial Contributor has attached the
+     notice in Exhibit A, the Executable Form of such Source Code Form, and
+     Modifications of such Source Code Form, in each case including portions
+     thereof.
+
+1.5. “Incompatible With Secondary Licenses”
+     means
+
+     a. that the initial Contributor has attached the notice described in
+        Exhibit B to the Covered Software; or
+
+     b. that the Covered Software was made available under the terms of version
+        1.1 or earlier of the License, but not also under the terms of a
+        Secondary License.
+
+1.6. “Executable Form”
+
+     means any form of the work other than Source Code Form.
+
+1.7. “Larger Work”
+
+     means a work that combines Covered Software with other material, in a separate
+     file or files, that is not Covered Software.
+
+1.8. “License”
+
+     means this document.
+
+1.9. “Licensable”
+
+     means having the right to grant, to the maximum extent possible, whether at the
+     time of the initial grant or subsequently, any and all of the rights conveyed by
+     this License.
+
+1.10. “Modifications”
+
+     means any of the following:
+
+     a. any file in Source Code Form that results from an addition to, deletion
+        from, or modification of the contents of Covered Software; or
+
+     b. any new file in Source Code Form that contains any Covered Software.
+
+1.11. “Patent Claims” of a Contributor
+
+      means any patent claim(s), including without limitation, method, process,
+      and apparatus claims, in any patent Licensable by such Contributor that
+      would be infringed, but for the grant of the License, by the making,
+      using, selling, offering for sale, having made, import, or transfer of
+      either its Contributions or its Contributor Version.
+
+1.12. “Secondary License”
+
+      means either the GNU General Public License, Version 2.0, the GNU Lesser
+      General Public License, Version 2.1, the GNU Affero General Public
+      License, Version 3.0, or any later versions of those licenses.
+
+1.13. “Source Code Form”
+
+      means the form of the work preferred for making modifications.
+
+1.14. “You” (or “Your”)
+
+      means an individual or a legal entity exercising rights under this
+      License. For legal entities, “You” includes any entity that controls, is
+      controlled by, or is under common control with You. For purposes of this
+      definition, “control” means (a) the power, direct or indirect, to cause
+      the direction or management of such entity, whether by contract or
+      otherwise, or (b) ownership of more than fifty percent (50%) of the
+      outstanding shares or beneficial ownership of such entity.
+
+
+2. License Grants and Conditions
+
+2.1. Grants
+
+     Each Contributor hereby grants You a world-wide, royalty-free,
+     non-exclusive license:
+
+     a. under intellectual property rights (other than patent or trademark)
+        Licensable by such Contributor to use, reproduce, make available,
+        modify, display, perform, distribute, and otherwise exploit its
+        Contributions, either on an unmodified basis, with Modifications, or as
+        part of a Larger Work; and
+
+     b. under Patent Claims of such Contributor to make, use, sell, offer for
+        sale, have made, import, and otherwise transfer either its Contributions
+        or its Contributor Version.
+
+2.2. Effective Date
+
+     The licenses granted in Section 2.1 with respect to any Contribution become
+     effective for each Contribution on the date the Contributor first distributes
+     such Contribution.
+
+2.3. Limitations on Grant Scope
+
+     The licenses granted in this Section 2 are the only rights granted under this
+     License. No additional rights or licenses will be implied from the distribution
+     or licensing of Covered Software under this License. Notwithstanding Section
+     2.1(b) above, no patent license is granted by a Contributor:
+
+     a. for any code that a Contributor has removed from Covered Software; or
+
+     b. for infringements caused by: (i) Your and any other third party’s
+        modifications of Covered Software, or (ii) the combination of its
+        Contributions with other software (except as part of its Contributor
+        Version); or
+
+     c. under Patent Claims infringed by Covered Software in the absence of its
+        Contributions.
+
+     This License does not grant any rights in the trademarks, service marks, or
+     logos of any Contributor (except as may be necessary to comply with the
+     notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+     No Contributor makes additional grants as a result of Your choice to
+     distribute the Covered Software under a subsequent version of this License
+     (see Section 10.2) or under the terms of a Secondary License (if permitted
+     under the terms of Section 3.3).
+
+2.5. Representation
+
+     Each Contributor represents that the Contributor believes its Contributions
+     are its original creation(s) or it has sufficient rights to grant the
+     rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+     This License is not intended to limit any rights You have under applicable
+     copyright doctrines of fair use, fair dealing, or other equivalents.
+
+2.7. Conditions
+
+     Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
+     Section 2.1.
+
+
+3. Responsibilities
+
+3.1. Distribution of Source Form
+
+     All distribution of Covered Software in Source Code Form, including any
+     Modifications that You create or to which You contribute, must be under the
+     terms of this License. You must inform recipients that the Source Code Form
+     of the Covered Software is governed by the terms of this License, and how
+     they can obtain a copy of this License. You may not attempt to alter or
+     restrict the recipients’ rights in the Source Code Form.
+
+3.2. Distribution of Executable Form
+
+     If You distribute Covered Software in Executable Form then:
+
+     a. such Covered Software must also be made available in Source Code Form,
+        as described in Section 3.1, and You must inform recipients of the
+        Executable Form how they can obtain a copy of such Source Code Form by
+        reasonable means in a timely manner, at a charge no more than the cost
+        of distribution to the recipient; and
+
+     b. You may distribute such Executable Form under the terms of this License,
+        or sublicense it under different terms, provided that the license for
+        the Executable Form does not attempt to limit or alter the recipients’
+        rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+     You may create and distribute a Larger Work under terms of Your choice,
+     provided that You also comply with the requirements of this License for the
+     Covered Software. If the Larger Work is a combination of Covered Software
+     with a work governed by one or more Secondary Licenses, and the Covered
+     Software is not Incompatible With Secondary Licenses, this License permits
+     You to additionally distribute such Covered Software under the terms of
+     such Secondary License(s), so that the recipient of the Larger Work may, at
+     their option, further distribute the Covered Software under the terms of
+     either this License or such Secondary License(s).
+
+3.4. Notices
+
+     You may not remove or alter the substance of any license notices (including
+     copyright notices, patent notices, disclaimers of warranty, or limitations
+     of liability) contained within the Source Code Form of the Covered
+     Software, except that You may alter any license notices to the extent
+     required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+     You may choose to offer, and to charge a fee for, warranty, support,
+     indemnity or liability obligations to one or more recipients of Covered
+     Software. However, You may do so only on Your own behalf, and not on behalf
+     of any Contributor. You must make it absolutely clear that any such
+     warranty, support, indemnity, or liability obligation is offered by You
+     alone, and You hereby agree to indemnify every Contributor for any
+     liability incurred by such Contributor as a result of warranty, support,
+     indemnity or liability terms You offer. You may include additional
+     disclaimers of warranty and limitations of liability specific to any
+     jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+
+   If it is impossible for You to comply with any of the terms of this License
+   with respect to some or all of the Covered Software due to statute, judicial
+   order, or regulation then You must: (a) comply with the terms of this License
+   to the maximum extent possible; and (b) describe the limitations and the code
+   they affect. Such description must be placed in a text file included with all
+   distributions of the Covered Software under this License. Except to the
+   extent prohibited by statute or regulation, such description must be
+   sufficiently detailed for a recipient of ordinary skill to be able to
+   understand it.
+
+5. Termination
+
+5.1. The rights granted under this License will terminate automatically if You
+     fail to comply with any of its terms. However, if You become compliant,
+     then the rights granted under this License from a particular Contributor
+     are reinstated (a) provisionally, unless and until such Contributor
+     explicitly and finally terminates Your grants, and (b) on an ongoing basis,
+     if such Contributor fails to notify You of the non-compliance by some
+     reasonable means prior to 60 days after You have come back into compliance.
+     Moreover, Your grants from a particular Contributor are reinstated on an
+     ongoing basis if such Contributor notifies You of the non-compliance by
+     some reasonable means, this is the first time You have received notice of
+     non-compliance with this License from such Contributor, and You become
+     compliant prior to 30 days after Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+     infringement claim (excluding declaratory judgment actions, counter-claims,
+     and cross-claims) alleging that a Contributor Version directly or
+     indirectly infringes any patent, then the rights granted to You by any and
+     all Contributors for the Covered Software under Section 2.1 of this License
+     shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
+     license agreements (excluding distributors and resellers) which have been
+     validly granted by You or Your distributors under this License prior to
+     termination shall survive termination.
+
+6. Disclaimer of Warranty
+
+   Covered Software is provided under this License on an “as is” basis, without
+   warranty of any kind, either expressed, implied, or statutory, including,
+   without limitation, warranties that the Covered Software is free of defects,
+   merchantable, fit for a particular purpose or non-infringing. The entire
+   risk as to the quality and performance of the Covered Software is with You.
+   Should any Covered Software prove defective in any respect, You (not any
+   Contributor) assume the cost of any necessary servicing, repair, or
+   correction. This disclaimer of warranty constitutes an essential part of this
+   License. No use of  any Covered Software is authorized under this License
+   except under this disclaimer.
+
+7. Limitation of Liability
+
+   Under no circumstances and under no legal theory, whether tort (including
+   negligence), contract, or otherwise, shall any Contributor, or anyone who
+   distributes Covered Software as permitted above, be liable to You for any
+   direct, indirect, special, incidental, or consequential damages of any
+   character including, without limitation, damages for lost profits, loss of
+   goodwill, work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses, even if such party shall have been
+   informed of the possibility of such damages. This limitation of liability
+   shall not apply to liability for death or personal injury resulting from such
+   party’s negligence to the extent applicable law prohibits such limitation.
+   Some jurisdictions do not allow the exclusion or limitation of incidental or
+   consequential damages, so this exclusion and limitation may not apply to You.
+
+8. Litigation
+
+   Any litigation relating to this License may be brought only in the courts of
+   a jurisdiction where the defendant maintains its principal place of business
+   and such litigation shall be governed by laws of that jurisdiction, without
+   reference to its conflict-of-law provisions. Nothing in this Section shall
+   prevent a party’s ability to bring cross-claims or counter-claims.
+
+9. Miscellaneous
+
+   This License represents the complete agreement concerning the subject matter
+   hereof. If any provision of this License is held to be unenforceable, such
+   provision shall be reformed only to the extent necessary to make it
+   enforceable. Any law or regulation which provides that the language of a
+   contract shall be construed against the drafter shall not be used to construe
+   this License against a Contributor.
+
+
+10. Versions of the License
+
+10.1. New Versions
+
+      Mozilla Foundation is the license steward. Except as provided in Section
+      10.3, no one other than the license steward has the right to modify or
+      publish new versions of this License. Each version will be given a
+      distinguishing version number.
+
+10.2. Effect of New Versions
+
+      You may distribute the Covered Software under the terms of the version of
+      the License under which You originally received the Covered Software, or
+      under the terms of any subsequent version published by the license
+      steward.
+
+10.3. Modified Versions
+
+      If you create software not governed by this License, and you want to
+      create a new license for such software, you may create and use a modified
+      version of this License if you rename the license and remove any
+      references to the name of the license steward (except to note that such
+      modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses
+      If You choose to distribute Source Code Form that is Incompatible With
+      Secondary Licenses under the terms of this version of the License, the
+      notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+
+      This Source Code Form is subject to the
+      terms of the Mozilla Public License, v.
+      2.0. If a copy of the MPL was not
+      distributed with this file, You can
+      obtain one at
+      http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular file, then
+You may include the notice in a location (such as a LICENSE file in a relevant
+directory) where a recipient would be likely to look for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - “Incompatible With Secondary Licenses” Notice
+
+      This Source Code Form is “Incompatible
+      With Secondary Licenses”, as defined by
+      the Mozilla Public License, v. 2.0.
+
diff --git a/third_party/github.com/hashicorp/errwrap/README.md b/third_party/github.com/hashicorp/errwrap/README.md
new file mode 100644
index 000000000..444df08f8
--- /dev/null
+++ b/third_party/github.com/hashicorp/errwrap/README.md
@@ -0,0 +1,89 @@
+# errwrap
+
+`errwrap` is a package for Go that formalizes the pattern of wrapping errors
+and checking if an error contains another error.
+
+There is a common pattern in Go of taking a returned `error` value and
+then wrapping it (such as with `fmt.Errorf`) before returning it. The problem
+with this pattern is that you completely lose the original `error` structure.
+
+Arguably the _correct_ approach is that you should make a custom structure
+implementing the `error` interface, and have the original error as a field
+on that structure, such [as this example](http://golang.org/pkg/os/#PathError).
+This is a good approach, but you have to know the entire chain of possible
+rewrapping that happens, when you might just care about one.
+
+`errwrap` formalizes this pattern (it doesn't matter what approach you use
+above) by giving a single interface for wrapping errors, checking if a specific
+error is wrapped, and extracting that error.
+
+## Installation and Docs
+
+Install using `go get github.com/hashicorp/errwrap`.
+
+Full documentation is available at
+http://godoc.org/github.com/hashicorp/errwrap
+
+## Usage
+
+#### Basic Usage
+
+Below is a very basic example of its usage:
+
+```go
+// A function that always returns an error, but wraps it, like a real
+// function might.
+func tryOpen() error {
+	_, err := os.Open("/i/dont/exist")
+	if err != nil {
+		return errwrap.Wrapf("Doesn't exist: {{err}}", err)
+	}
+
+	return nil
+}
+
+func main() {
+	err := tryOpen()
+
+	// We can use the Contains helpers to check if an error contains
+	// another error. It is safe to do this with a nil error, or with
+	// an error that doesn't even use the errwrap package.
+	if errwrap.Contains(err, "does not exist") {
+		// Do something
+	}
+	if errwrap.ContainsType(err, new(os.PathError)) {
+		// Do something
+	}
+
+	// Or we can use the associated `Get` functions to just extract
+	// a specific error. This would return nil if that specific error doesn't
+	// exist.
+	perr := errwrap.GetType(err, new(os.PathError))
+}
+```
+
+#### Custom Types
+
+If you're already making custom types that properly wrap errors, then
+you can get all the functionality of `errwraps.Contains` and such by
+implementing the `Wrapper` interface with just one function. Example:
+
+```go
+type AppError {
+  Code ErrorCode
+  Err  error
+}
+
+func (e *AppError) WrappedErrors() []error {
+  return []error{e.Err}
+}
+```
+
+Now this works:
+
+```go
+err := &AppError{Err: fmt.Errorf("an error")}
+if errwrap.ContainsType(err, fmt.Errorf("")) {
+	// This will work!
+}
+```
diff --git a/third_party/github.com/hashicorp/errwrap/errwrap.go b/third_party/github.com/hashicorp/errwrap/errwrap.go
new file mode 100644
index 000000000..a733bef18
--- /dev/null
+++ b/third_party/github.com/hashicorp/errwrap/errwrap.go
@@ -0,0 +1,169 @@
+// Package errwrap implements methods to formalize error wrapping in Go.
+//
+// All of the top-level functions that take an `error` are built to be able
+// to take any error, not just wrapped errors. This allows you to use errwrap
+// without having to type-check and type-cast everywhere.
+package errwrap
+
+import (
+	"errors"
+	"reflect"
+	"strings"
+)
+
+// WalkFunc is the callback called for Walk.
+type WalkFunc func(error)
+
+// Wrapper is an interface that can be implemented by custom types to
+// have all the Contains, Get, etc. functions in errwrap work.
+//
+// When Walk reaches a Wrapper, it will call the callback for every
+// wrapped error in addition to the wrapper itself. Since all the top-level
+// functions in errwrap use Walk, this means that all those functions work
+// with your custom type.
+type Wrapper interface {
+	WrappedErrors() []error
+}
+
+// Wrap defines that outer wraps inner, returning an error type that
+// can be cleanly used with the other methods in this package, such as
+// Contains, GetAll, etc.
+//
+// This function won't modify the error message at all (the outer message
+// will be used).
+func Wrap(outer, inner error) error {
+	return &wrappedError{
+		Outer: outer,
+		Inner: inner,
+	}
+}
+
+// Wrapf wraps an error with a formatting message. This is similar to using
+// `fmt.Errorf` to wrap an error. If you're using `fmt.Errorf` to wrap
+// errors, you should replace it with this.
+//
+// format is the format of the error message. The string '{{err}}' will
+// be replaced with the original error message.
+func Wrapf(format string, err error) error {
+	outerMsg := "<nil>"
+	if err != nil {
+		outerMsg = err.Error()
+	}
+
+	outer := errors.New(strings.Replace(
+		format, "{{err}}", outerMsg, -1))
+
+	return Wrap(outer, err)
+}
+
+// Contains checks if the given error contains an error with the
+// message msg. If err is not a wrapped error, this will always return
+// false unless the error itself happens to match this msg.
+func Contains(err error, msg string) bool {
+	return len(GetAll(err, msg)) > 0
+}
+
+// ContainsType checks if the given error contains an error with
+// the same concrete type as v. If err is not a wrapped error, this will
+// check the err itself.
+func ContainsType(err error, v interface{}) bool {
+	return len(GetAllType(err, v)) > 0
+}
+
+// Get is the same as GetAll but returns the deepest matching error.
+func Get(err error, msg string) error {
+	es := GetAll(err, msg)
+	if len(es) > 0 {
+		return es[len(es)-1]
+	}
+
+	return nil
+}
+
+// GetType is the same as GetAllType but returns the deepest matching error.
+func GetType(err error, v interface{}) error {
+	es := GetAllType(err, v)
+	if len(es) > 0 {
+		return es[len(es)-1]
+	}
+
+	return nil
+}
+
+// GetAll gets all the errors that might be wrapped in err with the
+// given message. The order of the errors is such that the outermost
+// matching error (the most recent wrap) is index zero, and so on.
+func GetAll(err error, msg string) []error {
+	var result []error
+
+	Walk(err, func(err error) {
+		if err.Error() == msg {
+			result = append(result, err)
+		}
+	})
+
+	return result
+}
+
+// GetAllType gets all the errors that are the same type as v.
+//
+// The order of the return value is the same as described in GetAll.
+func GetAllType(err error, v interface{}) []error {
+	var result []error
+
+	var search string
+	if v != nil {
+		search = reflect.TypeOf(v).String()
+	}
+	Walk(err, func(err error) {
+		var needle string
+		if err != nil {
+			needle = reflect.TypeOf(err).String()
+		}
+
+		if needle == search {
+			result = append(result, err)
+		}
+	})
+
+	return result
+}
+
+// Walk walks all the wrapped errors in err and calls the callback. If
+// err isn't a wrapped error, this will be called once for err. If err
+// is a wrapped error, the callback will be called for both the wrapper
+// that implements error as well as the wrapped error itself.
+func Walk(err error, cb WalkFunc) {
+	if err == nil {
+		return
+	}
+
+	switch e := err.(type) {
+	case *wrappedError:
+		cb(e.Outer)
+		Walk(e.Inner, cb)
+	case Wrapper:
+		cb(err)
+
+		for _, err := range e.WrappedErrors() {
+			Walk(err, cb)
+		}
+	default:
+		cb(err)
+	}
+}
+
+// wrappedError is an implementation of error that has both the
+// outer and inner errors.
+type wrappedError struct {
+	Outer error
+	Inner error
+}
+
+func (w *wrappedError) Error() string {
+	return w.Outer.Error()
+}
+
+func (w *wrappedError) WrappedErrors() []error {
+	return []error{w.Outer, w.Inner}
+}
diff --git a/third_party/github.com/hashicorp/errwrap/go.mod b/third_party/github.com/hashicorp/errwrap/go.mod
new file mode 100644
index 000000000..c9b84022c
--- /dev/null
+++ b/third_party/github.com/hashicorp/errwrap/go.mod
@@ -0,0 +1,1 @@
+module github.com/hashicorp/errwrap
diff --git a/third_party/github.com/hashicorp/go-multierror/LICENSE b/third_party/github.com/hashicorp/go-multierror/LICENSE
new file mode 100644
index 000000000..82b4de97c
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/LICENSE
@@ -0,0 +1,353 @@
+Mozilla Public License, version 2.0
+
+1. Definitions
+
+1.1. “Contributor”
+
+     means each individual or legal entity that creates, contributes to the
+     creation of, or owns Covered Software.
+
+1.2. “Contributor Version”
+
+     means the combination of the Contributions of others (if any) used by a
+     Contributor and that particular Contributor’s Contribution.
+
+1.3. “Contribution”
+
+     means Covered Software of a particular Contributor.
+
+1.4. “Covered Software”
+
+     means Source Code Form to which the initial Contributor has attached the
+     notice in Exhibit A, the Executable Form of such Source Code Form, and
+     Modifications of such Source Code Form, in each case including portions
+     thereof.
+
+1.5. “Incompatible With Secondary Licenses”
+     means
+
+     a. that the initial Contributor has attached the notice described in
+        Exhibit B to the Covered Software; or
+
+     b. that the Covered Software was made available under the terms of version
+        1.1 or earlier of the License, but not also under the terms of a
+        Secondary License.
+
+1.6. “Executable Form”
+
+     means any form of the work other than Source Code Form.
+
+1.7. “Larger Work”
+
+     means a work that combines Covered Software with other material, in a separate
+     file or files, that is not Covered Software.
+
+1.8. “License”
+
+     means this document.
+
+1.9. “Licensable”
+
+     means having the right to grant, to the maximum extent possible, whether at the
+     time of the initial grant or subsequently, any and all of the rights conveyed by
+     this License.
+
+1.10. “Modifications”
+
+     means any of the following:
+
+     a. any file in Source Code Form that results from an addition to, deletion
+        from, or modification of the contents of Covered Software; or
+
+     b. any new file in Source Code Form that contains any Covered Software.
+
+1.11. “Patent Claims” of a Contributor
+
+      means any patent claim(s), including without limitation, method, process,
+      and apparatus claims, in any patent Licensable by such Contributor that
+      would be infringed, but for the grant of the License, by the making,
+      using, selling, offering for sale, having made, import, or transfer of
+      either its Contributions or its Contributor Version.
+
+1.12. “Secondary License”
+
+      means either the GNU General Public License, Version 2.0, the GNU Lesser
+      General Public License, Version 2.1, the GNU Affero General Public
+      License, Version 3.0, or any later versions of those licenses.
+
+1.13. “Source Code Form”
+
+      means the form of the work preferred for making modifications.
+
+1.14. “You” (or “Your”)
+
+      means an individual or a legal entity exercising rights under this
+      License. For legal entities, “You” includes any entity that controls, is
+      controlled by, or is under common control with You. For purposes of this
+      definition, “control” means (a) the power, direct or indirect, to cause
+      the direction or management of such entity, whether by contract or
+      otherwise, or (b) ownership of more than fifty percent (50%) of the
+      outstanding shares or beneficial ownership of such entity.
+
+
+2. License Grants and Conditions
+
+2.1. Grants
+
+     Each Contributor hereby grants You a world-wide, royalty-free,
+     non-exclusive license:
+
+     a. under intellectual property rights (other than patent or trademark)
+        Licensable by such Contributor to use, reproduce, make available,
+        modify, display, perform, distribute, and otherwise exploit its
+        Contributions, either on an unmodified basis, with Modifications, or as
+        part of a Larger Work; and
+
+     b. under Patent Claims of such Contributor to make, use, sell, offer for
+        sale, have made, import, and otherwise transfer either its Contributions
+        or its Contributor Version.
+
+2.2. Effective Date
+
+     The licenses granted in Section 2.1 with respect to any Contribution become
+     effective for each Contribution on the date the Contributor first distributes
+     such Contribution.
+
+2.3. Limitations on Grant Scope
+
+     The licenses granted in this Section 2 are the only rights granted under this
+     License. No additional rights or licenses will be implied from the distribution
+     or licensing of Covered Software under this License. Notwithstanding Section
+     2.1(b) above, no patent license is granted by a Contributor:
+
+     a. for any code that a Contributor has removed from Covered Software; or
+
+     b. for infringements caused by: (i) Your and any other third party’s
+        modifications of Covered Software, or (ii) the combination of its
+        Contributions with other software (except as part of its Contributor
+        Version); or
+
+     c. under Patent Claims infringed by Covered Software in the absence of its
+        Contributions.
+
+     This License does not grant any rights in the trademarks, service marks, or
+     logos of any Contributor (except as may be necessary to comply with the
+     notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+     No Contributor makes additional grants as a result of Your choice to
+     distribute the Covered Software under a subsequent version of this License
+     (see Section 10.2) or under the terms of a Secondary License (if permitted
+     under the terms of Section 3.3).
+
+2.5. Representation
+
+     Each Contributor represents that the Contributor believes its Contributions
+     are its original creation(s) or it has sufficient rights to grant the
+     rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+     This License is not intended to limit any rights You have under applicable
+     copyright doctrines of fair use, fair dealing, or other equivalents.
+
+2.7. Conditions
+
+     Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
+     Section 2.1.
+
+
+3. Responsibilities
+
+3.1. Distribution of Source Form
+
+     All distribution of Covered Software in Source Code Form, including any
+     Modifications that You create or to which You contribute, must be under the
+     terms of this License. You must inform recipients that the Source Code Form
+     of the Covered Software is governed by the terms of this License, and how
+     they can obtain a copy of this License. You may not attempt to alter or
+     restrict the recipients’ rights in the Source Code Form.
+
+3.2. Distribution of Executable Form
+
+     If You distribute Covered Software in Executable Form then:
+
+     a. such Covered Software must also be made available in Source Code Form,
+        as described in Section 3.1, and You must inform recipients of the
+        Executable Form how they can obtain a copy of such Source Code Form by
+        reasonable means in a timely manner, at a charge no more than the cost
+        of distribution to the recipient; and
+
+     b. You may distribute such Executable Form under the terms of this License,
+        or sublicense it under different terms, provided that the license for
+        the Executable Form does not attempt to limit or alter the recipients’
+        rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+     You may create and distribute a Larger Work under terms of Your choice,
+     provided that You also comply with the requirements of this License for the
+     Covered Software. If the Larger Work is a combination of Covered Software
+     with a work governed by one or more Secondary Licenses, and the Covered
+     Software is not Incompatible With Secondary Licenses, this License permits
+     You to additionally distribute such Covered Software under the terms of
+     such Secondary License(s), so that the recipient of the Larger Work may, at
+     their option, further distribute the Covered Software under the terms of
+     either this License or such Secondary License(s).
+
+3.4. Notices
+
+     You may not remove or alter the substance of any license notices (including
+     copyright notices, patent notices, disclaimers of warranty, or limitations
+     of liability) contained within the Source Code Form of the Covered
+     Software, except that You may alter any license notices to the extent
+     required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+     You may choose to offer, and to charge a fee for, warranty, support,
+     indemnity or liability obligations to one or more recipients of Covered
+     Software. However, You may do so only on Your own behalf, and not on behalf
+     of any Contributor. You must make it absolutely clear that any such
+     warranty, support, indemnity, or liability obligation is offered by You
+     alone, and You hereby agree to indemnify every Contributor for any
+     liability incurred by such Contributor as a result of warranty, support,
+     indemnity or liability terms You offer. You may include additional
+     disclaimers of warranty and limitations of liability specific to any
+     jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+
+   If it is impossible for You to comply with any of the terms of this License
+   with respect to some or all of the Covered Software due to statute, judicial
+   order, or regulation then You must: (a) comply with the terms of this License
+   to the maximum extent possible; and (b) describe the limitations and the code
+   they affect. Such description must be placed in a text file included with all
+   distributions of the Covered Software under this License. Except to the
+   extent prohibited by statute or regulation, such description must be
+   sufficiently detailed for a recipient of ordinary skill to be able to
+   understand it.
+
+5. Termination
+
+5.1. The rights granted under this License will terminate automatically if You
+     fail to comply with any of its terms. However, if You become compliant,
+     then the rights granted under this License from a particular Contributor
+     are reinstated (a) provisionally, unless and until such Contributor
+     explicitly and finally terminates Your grants, and (b) on an ongoing basis,
+     if such Contributor fails to notify You of the non-compliance by some
+     reasonable means prior to 60 days after You have come back into compliance.
+     Moreover, Your grants from a particular Contributor are reinstated on an
+     ongoing basis if such Contributor notifies You of the non-compliance by
+     some reasonable means, this is the first time You have received notice of
+     non-compliance with this License from such Contributor, and You become
+     compliant prior to 30 days after Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+     infringement claim (excluding declaratory judgment actions, counter-claims,
+     and cross-claims) alleging that a Contributor Version directly or
+     indirectly infringes any patent, then the rights granted to You by any and
+     all Contributors for the Covered Software under Section 2.1 of this License
+     shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
+     license agreements (excluding distributors and resellers) which have been
+     validly granted by You or Your distributors under this License prior to
+     termination shall survive termination.
+
+6. Disclaimer of Warranty
+
+   Covered Software is provided under this License on an “as is” basis, without
+   warranty of any kind, either expressed, implied, or statutory, including,
+   without limitation, warranties that the Covered Software is free of defects,
+   merchantable, fit for a particular purpose or non-infringing. The entire
+   risk as to the quality and performance of the Covered Software is with You.
+   Should any Covered Software prove defective in any respect, You (not any
+   Contributor) assume the cost of any necessary servicing, repair, or
+   correction. This disclaimer of warranty constitutes an essential part of this
+   License. No use of  any Covered Software is authorized under this License
+   except under this disclaimer.
+
+7. Limitation of Liability
+
+   Under no circumstances and under no legal theory, whether tort (including
+   negligence), contract, or otherwise, shall any Contributor, or anyone who
+   distributes Covered Software as permitted above, be liable to You for any
+   direct, indirect, special, incidental, or consequential damages of any
+   character including, without limitation, damages for lost profits, loss of
+   goodwill, work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses, even if such party shall have been
+   informed of the possibility of such damages. This limitation of liability
+   shall not apply to liability for death or personal injury resulting from such
+   party’s negligence to the extent applicable law prohibits such limitation.
+   Some jurisdictions do not allow the exclusion or limitation of incidental or
+   consequential damages, so this exclusion and limitation may not apply to You.
+
+8. Litigation
+
+   Any litigation relating to this License may be brought only in the courts of
+   a jurisdiction where the defendant maintains its principal place of business
+   and such litigation shall be governed by laws of that jurisdiction, without
+   reference to its conflict-of-law provisions. Nothing in this Section shall
+   prevent a party’s ability to bring cross-claims or counter-claims.
+
+9. Miscellaneous
+
+   This License represents the complete agreement concerning the subject matter
+   hereof. If any provision of this License is held to be unenforceable, such
+   provision shall be reformed only to the extent necessary to make it
+   enforceable. Any law or regulation which provides that the language of a
+   contract shall be construed against the drafter shall not be used to construe
+   this License against a Contributor.
+
+
+10. Versions of the License
+
+10.1. New Versions
+
+      Mozilla Foundation is the license steward. Except as provided in Section
+      10.3, no one other than the license steward has the right to modify or
+      publish new versions of this License. Each version will be given a
+      distinguishing version number.
+
+10.2. Effect of New Versions
+
+      You may distribute the Covered Software under the terms of the version of
+      the License under which You originally received the Covered Software, or
+      under the terms of any subsequent version published by the license
+      steward.
+
+10.3. Modified Versions
+
+      If you create software not governed by this License, and you want to
+      create a new license for such software, you may create and use a modified
+      version of this License if you rename the license and remove any
+      references to the name of the license steward (except to note that such
+      modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses
+      If You choose to distribute Source Code Form that is Incompatible With
+      Secondary Licenses under the terms of this version of the License, the
+      notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+
+      This Source Code Form is subject to the
+      terms of the Mozilla Public License, v.
+      2.0. If a copy of the MPL was not
+      distributed with this file, You can
+      obtain one at
+      http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular file, then
+You may include the notice in a location (such as a LICENSE file in a relevant
+directory) where a recipient would be likely to look for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - “Incompatible With Secondary Licenses” Notice
+
+      This Source Code Form is “Incompatible
+      With Secondary Licenses”, as defined by
+      the Mozilla Public License, v. 2.0.
diff --git a/third_party/github.com/hashicorp/go-multierror/Makefile b/third_party/github.com/hashicorp/go-multierror/Makefile
new file mode 100644
index 000000000..b97cd6ed0
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/Makefile
@@ -0,0 +1,31 @@
+TEST?=./...
+
+default: test
+
+# test runs the test suite and vets the code.
+test: generate
+	@echo "==> Running tests..."
+	@go list $(TEST) \
+		| grep -v "/vendor/" \
+		| xargs -n1 go test -timeout=60s -parallel=10 ${TESTARGS}
+
+# testrace runs the race checker
+testrace: generate
+	@echo "==> Running tests (race)..."
+	@go list $(TEST) \
+		| grep -v "/vendor/" \
+		| xargs -n1 go test -timeout=60s -race ${TESTARGS}
+
+# updatedeps installs all the dependencies needed to run and build.
+updatedeps:
+	@sh -c "'${CURDIR}/scripts/deps.sh' '${NAME}'"
+
+# generate runs `go generate` to build the dynamically generated source files.
+generate:
+	@echo "==> Generating..."
+	@find . -type f -name '.DS_Store' -delete
+	@go list ./... \
+		| grep -v "/vendor/" \
+		| xargs -n1 go generate
+
+.PHONY: default test testrace updatedeps generate
diff --git a/third_party/github.com/hashicorp/go-multierror/README.md b/third_party/github.com/hashicorp/go-multierror/README.md
new file mode 100644
index 000000000..71dd308ed
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/README.md
@@ -0,0 +1,150 @@
+# go-multierror
+
+[![CircleCI](https://img.shields.io/circleci/build/github/hashicorp/go-multierror/master)](https://circleci.com/gh/hashicorp/go-multierror)
+[![Go Reference](https://pkg.go.dev/badge/github.com/hashicorp/go-multierror.svg)](https://pkg.go.dev/github.com/hashicorp/go-multierror)
+![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/hashicorp/go-multierror)
+
+[circleci]: https://app.circleci.com/pipelines/github/hashicorp/go-multierror
+[godocs]: https://pkg.go.dev/github.com/hashicorp/go-multierror
+
+`go-multierror` is a package for Go that provides a mechanism for
+representing a list of `error` values as a single `error`.
+
+This allows a function in Go to return an `error` that might actually
+be a list of errors. If the caller knows this, they can unwrap the
+list and access the errors. If the caller doesn't know, the error
+formats to a nice human-readable format.
+
+`go-multierror` is fully compatible with the Go standard library
+[errors](https://golang.org/pkg/errors/) package, including the
+functions `As`, `Is`, and `Unwrap`. This provides a standardized approach
+for introspecting on error values.
+
+## Installation and Docs
+
+Install using `go get github.com/hashicorp/go-multierror`.
+
+Full documentation is available at
+https://pkg.go.dev/github.com/hashicorp/go-multierror
+
+### Requires go version 1.13 or newer
+
+`go-multierror` requires go version 1.13 or newer. Go 1.13 introduced
+[error wrapping](https://golang.org/doc/go1.13#error_wrapping), which
+this library takes advantage of.
+
+If you need to use an earlier version of go, you can use the
+[v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0)
+tag, which doesn't rely on features in go 1.13.
+
+If you see compile errors that look like the below, it's likely that
+you're on an older version of go:
+
+```
+/go/src/github.com/hashicorp/go-multierror/multierror.go:112:9: undefined: errors.As
+/go/src/github.com/hashicorp/go-multierror/multierror.go:117:9: undefined: errors.Is
+```
+
+## Usage
+
+go-multierror is easy to use and purposely built to be unobtrusive in
+existing Go applications/libraries that may not be aware of it.
+
+**Building a list of errors**
+
+The `Append` function is used to create a list of errors. This function
+behaves a lot like the Go built-in `append` function: it doesn't matter
+if the first argument is nil, a `multierror.Error`, or any other `error`,
+the function behaves as you would expect.
+
+```go
+var result error
+
+if err := step1(); err != nil {
+	result = multierror.Append(result, err)
+}
+if err := step2(); err != nil {
+	result = multierror.Append(result, err)
+}
+
+return result
+```
+
+**Customizing the formatting of the errors**
+
+By specifying a custom `ErrorFormat`, you can customize the format
+of the `Error() string` function:
+
+```go
+var result *multierror.Error
+
+// ... accumulate errors here, maybe using Append
+
+if result != nil {
+	result.ErrorFormat = func([]error) string {
+		return "errors!"
+	}
+}
+```
+
+**Accessing the list of errors**
+
+`multierror.Error` implements `error` so if the caller doesn't know about
+multierror, it will work just fine. But if you're aware a multierror might
+be returned, you can use type switches to access the list of errors:
+
+```go
+if err := something(); err != nil {
+	if merr, ok := err.(*multierror.Error); ok {
+		// Use merr.Errors
+	}
+}
+```
+
+You can also use the standard [`errors.Unwrap`](https://golang.org/pkg/errors/#Unwrap)
+function. This will continue to unwrap into subsequent errors until none exist.
+
+**Extracting an error**
+
+The standard library [`errors.As`](https://golang.org/pkg/errors/#As)
+function can be used directly with a multierror to extract a specific error:
+
+```go
+// Assume err is a multierror value
+err := somefunc()
+
+// We want to know if "err" has a "RichErrorType" in it and extract it.
+var errRich RichErrorType
+if errors.As(err, &errRich) {
+	// It has it, and now errRich is populated.
+}
+```
+
+**Checking for an exact error value**
+
+Some errors are returned as exact errors such as the [`ErrNotExist`](https://golang.org/pkg/os/#pkg-variables)
+error in the `os` package. You can check if this error is present by using
+the standard [`errors.Is`](https://golang.org/pkg/errors/#Is) function.
+
+```go
+// Assume err is a multierror value
+err := somefunc()
+if errors.Is(err, os.ErrNotExist) {
+	// err contains os.ErrNotExist
+}
+```
+
+**Returning a multierror only if there are errors**
+
+If you build a `multierror.Error`, you can use the `ErrorOrNil` function
+to return an `error` implementation only if there are errors to return:
+
+```go
+var result *multierror.Error
+
+// ... accumulate errors here
+
+// Return the `error` only if errors were added to the multierror, otherwise
+// return nil since there are no errors.
+return result.ErrorOrNil()
+```
diff --git a/third_party/github.com/hashicorp/go-multierror/append.go b/third_party/github.com/hashicorp/go-multierror/append.go
new file mode 100644
index 000000000..3e2589bfd
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/append.go
@@ -0,0 +1,43 @@
+package multierror
+
+// Append is a helper function that will append more errors
+// onto an Error in order to create a larger multi-error.
+//
+// If err is not a multierror.Error, then it will be turned into
+// one. If any of the errs are multierr.Error, they will be flattened
+// one level into err.
+// Any nil errors within errs will be ignored. If err is nil, a new
+// *Error will be returned.
+func Append(err error, errs ...error) *Error {
+	switch err := err.(type) {
+	case *Error:
+		// Typed nils can reach here, so initialize if we are nil
+		if err == nil {
+			err = new(Error)
+		}
+
+		// Go through each error and flatten
+		for _, e := range errs {
+			switch e := e.(type) {
+			case *Error:
+				if e != nil {
+					err.Errors = append(err.Errors, e.Errors...)
+				}
+			default:
+				if e != nil {
+					err.Errors = append(err.Errors, e)
+				}
+			}
+		}
+
+		return err
+	default:
+		newErrs := make([]error, 0, len(errs)+1)
+		if err != nil {
+			newErrs = append(newErrs, err)
+		}
+		newErrs = append(newErrs, errs...)
+
+		return Append(&Error{}, newErrs...)
+	}
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/flatten.go b/third_party/github.com/hashicorp/go-multierror/flatten.go
new file mode 100644
index 000000000..aab8e9abe
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/flatten.go
@@ -0,0 +1,26 @@
+package multierror
+
+// Flatten flattens the given error, merging any *Errors together into
+// a single *Error.
+func Flatten(err error) error {
+	// If it isn't an *Error, just return the error as-is
+	if _, ok := err.(*Error); !ok {
+		return err
+	}
+
+	// Otherwise, make the result and flatten away!
+	flatErr := new(Error)
+	flatten(err, flatErr)
+	return flatErr
+}
+
+func flatten(err error, flatErr *Error) {
+	switch err := err.(type) {
+	case *Error:
+		for _, e := range err.Errors {
+			flatten(e, flatErr)
+		}
+	default:
+		flatErr.Errors = append(flatErr.Errors, err)
+	}
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/format.go b/third_party/github.com/hashicorp/go-multierror/format.go
new file mode 100644
index 000000000..47f13c49a
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/format.go
@@ -0,0 +1,27 @@
+package multierror
+
+import (
+	"fmt"
+	"strings"
+)
+
+// ErrorFormatFunc is a function callback that is called by Error to
+// turn the list of errors into a string.
+type ErrorFormatFunc func([]error) string
+
+// ListFormatFunc is a basic formatter that outputs the number of errors
+// that occurred along with a bullet point list of the errors.
+func ListFormatFunc(es []error) string {
+	if len(es) == 1 {
+		return fmt.Sprintf("1 error occurred:\n\t* %s\n\n", es[0])
+	}
+
+	points := make([]string, len(es))
+	for i, err := range es {
+		points[i] = fmt.Sprintf("* %s", err)
+	}
+
+	return fmt.Sprintf(
+		"%d errors occurred:\n\t%s\n\n",
+		len(es), strings.Join(points, "\n\t"))
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/go.mod b/third_party/github.com/hashicorp/go-multierror/go.mod
new file mode 100644
index 000000000..141cc4ccb
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/go.mod
@@ -0,0 +1,5 @@
+module github.com/hashicorp/go-multierror
+
+go 1.13
+
+require github.com/hashicorp/errwrap v1.0.0
diff --git a/third_party/github.com/hashicorp/go-multierror/go.sum b/third_party/github.com/hashicorp/go-multierror/go.sum
new file mode 100644
index 000000000..e8238e9ec
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/go.sum
@@ -0,0 +1,2 @@
+github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
+github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
diff --git a/third_party/github.com/hashicorp/go-multierror/group.go b/third_party/github.com/hashicorp/go-multierror/group.go
new file mode 100644
index 000000000..9c29efb7f
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/group.go
@@ -0,0 +1,38 @@
+package multierror
+
+import "sync"
+
+// Group is a collection of goroutines which return errors that need to be
+// coalesced.
+type Group struct {
+	mutex sync.Mutex
+	err   *Error
+	wg    sync.WaitGroup
+}
+
+// Go calls the given function in a new goroutine.
+//
+// If the function returns an error it is added to the group multierror which
+// is returned by Wait.
+func (g *Group) Go(f func() error) {
+	g.wg.Add(1)
+
+	go func() {
+		defer g.wg.Done()
+
+		if err := f(); err != nil {
+			g.mutex.Lock()
+			g.err = Append(g.err, err)
+			g.mutex.Unlock()
+		}
+	}()
+}
+
+// Wait blocks until all function calls from the Go method have returned, then
+// returns the multierror.
+func (g *Group) Wait() *Error {
+	g.wg.Wait()
+	g.mutex.Lock()
+	defer g.mutex.Unlock()
+	return g.err
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/multierror.go b/third_party/github.com/hashicorp/go-multierror/multierror.go
new file mode 100644
index 000000000..f54574326
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/multierror.go
@@ -0,0 +1,121 @@
+package multierror
+
+import (
+	"errors"
+	"fmt"
+)
+
+// Error is an error type to track multiple errors. This is used to
+// accumulate errors in cases and return them as a single "error".
+type Error struct {
+	Errors      []error
+	ErrorFormat ErrorFormatFunc
+}
+
+func (e *Error) Error() string {
+	fn := e.ErrorFormat
+	if fn == nil {
+		fn = ListFormatFunc
+	}
+
+	return fn(e.Errors)
+}
+
+// ErrorOrNil returns an error interface if this Error represents
+// a list of errors, or returns nil if the list of errors is empty. This
+// function is useful at the end of accumulation to make sure that the value
+// returned represents the existence of errors.
+func (e *Error) ErrorOrNil() error {
+	if e == nil {
+		return nil
+	}
+	if len(e.Errors) == 0 {
+		return nil
+	}
+
+	return e
+}
+
+func (e *Error) GoString() string {
+	return fmt.Sprintf("*%#v", *e)
+}
+
+// WrappedErrors returns the list of errors that this Error is wrapping. It is
+// an implementation of the errwrap.Wrapper interface so that multierror.Error
+// can be used with that library.
+//
+// This method is not safe to be called concurrently. Unlike accessing the
+// Errors field directly, this function also checks if the multierror is nil to
+// prevent a null-pointer panic. It satisfies the errwrap.Wrapper interface.
+func (e *Error) WrappedErrors() []error {
+	if e == nil {
+		return nil
+	}
+	return e.Errors
+}
+
+// Unwrap returns an error from Error (or nil if there are no errors).
+// This error returned will further support Unwrap to get the next error,
+// etc. The order will match the order of Errors in the multierror.Error
+// at the time of calling.
+//
+// The resulting error supports errors.As/Is/Unwrap so you can continue
+// to use the stdlib errors package to introspect further.
+//
+// This will perform a shallow copy of the errors slice. Any errors appended
+// to this error after calling Unwrap will not be available until a new
+// Unwrap is called on the multierror.Error.
+func (e *Error) Unwrap() error {
+	// If we have no errors then we do nothing
+	if e == nil || len(e.Errors) == 0 {
+		return nil
+	}
+
+	// If we have exactly one error, we can just return that directly.
+	if len(e.Errors) == 1 {
+		return e.Errors[0]
+	}
+
+	// Shallow copy the slice
+	errs := make([]error, len(e.Errors))
+	copy(errs, e.Errors)
+	return chain(errs)
+}
+
+// chain implements the interfaces necessary for errors.Is/As/Unwrap to
+// work in a deterministic way with multierror. A chain tracks a list of
+// errors while accounting for the current represented error. This lets
+// Is/As be meaningful.
+//
+// Unwrap returns the next error. In the cleanest form, Unwrap would return
+// the wrapped error here but we can't do that if we want to properly
+// get access to all the errors. Instead, users are recommended to use
+// Is/As to get the correct error type out.
+//
+// Precondition: []error is non-empty (len > 0)
+type chain []error
+
+// Error implements the error interface
+func (e chain) Error() string {
+	return e[0].Error()
+}
+
+// Unwrap implements errors.Unwrap by returning the next error in the
+// chain or nil if there are no more errors.
+func (e chain) Unwrap() error {
+	if len(e) == 1 {
+		return nil
+	}
+
+	return e[1:]
+}
+
+// As implements errors.As by attempting to map to the current value.
+func (e chain) As(target interface{}) bool {
+	return errors.As(e[0], target)
+}
+
+// Is implements errors.Is by comparing the current value directly.
+func (e chain) Is(target error) bool {
+	return errors.Is(e[0], target)
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/prefix.go b/third_party/github.com/hashicorp/go-multierror/prefix.go
new file mode 100644
index 000000000..5c477abe4
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/prefix.go
@@ -0,0 +1,37 @@
+package multierror
+
+import (
+	"fmt"
+
+	"github.com/hashicorp/errwrap"
+)
+
+// Prefix is a helper function that will prefix some text
+// to the given error. If the error is a multierror.Error, then
+// it will be prefixed to each wrapped error.
+//
+// This is useful to use when appending multiple multierrors
+// together in order to give better scoping.
+func Prefix(err error, prefix string) error {
+	if err == nil {
+		return nil
+	}
+
+	format := fmt.Sprintf("%s {{err}}", prefix)
+	switch err := err.(type) {
+	case *Error:
+		// Typed nils can reach here, so initialize if we are nil
+		if err == nil {
+			err = new(Error)
+		}
+
+		// Wrap each of the errors
+		for i, e := range err.Errors {
+			err.Errors[i] = errwrap.Wrapf(format, e)
+		}
+
+		return err
+	default:
+		return errwrap.Wrapf(format, err)
+	}
+}
diff --git a/third_party/github.com/hashicorp/go-multierror/sort.go b/third_party/github.com/hashicorp/go-multierror/sort.go
new file mode 100644
index 000000000..fecb14e81
--- /dev/null
+++ b/third_party/github.com/hashicorp/go-multierror/sort.go
@@ -0,0 +1,16 @@
+package multierror
+
+// Len implements sort.Interface function for length
+func (err Error) Len() int {
+	return len(err.Errors)
+}
+
+// Swap implements sort.Interface function for swapping elements
+func (err Error) Swap(i, j int) {
+	err.Errors[i], err.Errors[j] = err.Errors[j], err.Errors[i]
+}
+
+// Less implements sort.Interface function for determining order
+func (err Error) Less(i, j int) bool {
+	return err.Errors[i].Error() < err.Errors[j].Error()
+}
diff --git a/third_party/google.golang.org/grpc/NOTICE.txt b/third_party/google.golang.org/grpc/NOTICE.txt
new file mode 100644
index 000000000..530197749
--- /dev/null
+++ b/third_party/google.golang.org/grpc/NOTICE.txt
@@ -0,0 +1,13 @@
+Copyright 2014 gRPC authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.


Test patch:
diff --git a/pkg/apis/operator/v1alpha1/tektonaddon_default_test.go b/pkg/apis/operator/v1alpha1/tektonaddon_default_test.go
index ac70b3b4c..8e8dbfea5 100644
--- a/pkg/apis/operator/v1alpha1/tektonaddon_default_test.go
+++ b/pkg/apis/operator/v1alpha1/tektonaddon_default_test.go
@@ -58,10 +58,12 @@ func Test_AddonSetDefaults_ClusterTaskIsFalse(t *testing.T) {
 			CommonSpec: CommonSpec{
 				TargetNamespace: "namespace",
 			},
-			Params: []Param{
-				{
-					Name:  "clusterTasks",
-					Value: "false",
+			Addon: Addon{
+				Params: []Param{
+					{
+						Name:  "clusterTasks",
+						Value: "false",
+					},
 				},
 			},
 		},
diff --git a/pkg/apis/operator/v1alpha1/tektonaddon_validation_test.go b/pkg/apis/operator/v1alpha1/tektonaddon_validation_test.go
index 02a30fd7c..f37d19c65 100644
--- a/pkg/apis/operator/v1alpha1/tektonaddon_validation_test.go
+++ b/pkg/apis/operator/v1alpha1/tektonaddon_validation_test.go
@@ -56,10 +56,12 @@ func Test_ValidateTektonAddon_InvalidParam(t *testing.T) {
 			CommonSpec: CommonSpec{
 				TargetNamespace: "namespace",
 			},
-			Params: []Param{
-				{
-					Name:  "foo",
-					Value: "test",
+			Addon: Addon{
+				Params: []Param{
+					{
+						Name:  "foo",
+						Value: "test",
+					},
 				},
 			},
 		},
@@ -80,10 +82,12 @@ func Test_ValidateTektonAddon_InvalidParamValue(t *testing.T) {
 			CommonSpec: CommonSpec{
 				TargetNamespace: "namespace",
 			},
-			Params: []Param{
-				{
-					Name:  "clusterTasks",
-					Value: "test",
+			Addon: Addon{
+				Params: []Param{
+					{
+						Name:  "clusterTasks",
+						Value: "test",
+					},
 				},
 			},
 		},
@@ -104,14 +108,16 @@ func Test_ValidateTektonAddon_ClusterTaskIsFalseAndPipelineTemplateIsTrue(t *tes
 			CommonSpec: CommonSpec{
 				TargetNamespace: "namespace",
 			},
-			Params: []Param{
-				{
-					Name:  "clusterTasks",
-					Value: "false",
-				},
-				{
-					Name:  "pipelineTemplates",
-					Value: "true",
+			Addon: Addon{
+				Params: []Param{
+					{
+						Name:  "clusterTasks",
+						Value: "false",
+					},
+					{
+						Name:  "pipelineTemplates",
+						Value: "true",
+					},
 				},
 			},
 		},
diff --git a/pkg/reconciler/common/labels_test.go b/pkg/reconciler/common/labels_test.go
index 669fff780..7b3c9b16b 100644
--- a/pkg/reconciler/common/labels_test.go
+++ b/pkg/reconciler/common/labels_test.go
@@ -35,11 +35,10 @@ func TestLabelSelector(t *testing.T) {
 		name: "non empty label selector",
 		ls: metav1.LabelSelector{
 			MatchLabels: map[string]string{
-				"createdByKey":     "createdByValue",
 				"installerSetType": "pipelineResourceName",
 			},
 		},
-		want: "createdByKey=createdByValue,installerSetType=pipelineResourceName",
+		want: "installerSetType=pipelineResourceName",
 	}} {
 		t.Run(c.name, func(t *testing.T) {
 			got, _ := LabelSelector(c.ls)
diff --git a/test/e2e/common/tektonconfigdeployment_test.go b/test/e2e/common/tektonconfigdeployment_test.go
index 00fcae9dd..7af80fe30 100644
--- a/test/e2e/common/tektonconfigdeployment_test.go
+++ b/test/e2e/common/tektonconfigdeployment_test.go
@@ -312,9 +312,8 @@ func runAddonTest(t *testing.T, clients *utils.Clients, tc *v1alpha1.TektonConfi
 			t.Fatalf("failed to get InstallerSet: %v", err)
 		}
 
-		// there must be 5 installerSet created for addons
-		if len(addonsIS.Items) != 6 {
-			t.Fatalf("expected 6 installerSets for Addon but got %v", len(addonsIS.Items))
+		if len(addonsIS.Items) != 7 {
+			t.Fatalf("expected 7 installerSets for Addon but got %v", len(addonsIS.Items))
 		}
 
 		// Now, disable clusterTasks and pipelineTemplates through TektonConfig
@@ -346,9 +345,48 @@ func runAddonTest(t *testing.T, clients *utils.Clients, tc *v1alpha1.TektonConfi
 		if err != nil {
 			t.Fatalf("failed to get InstallerSet: %v", err)
 		}
+		// Now, there must be 4 installerSet
+		if len(addonsIS.Items) != 4 {
+			t.Fatalf("expected 4 installerSets after disabling params for Addon but got %v", len(addonsIS.Items))
+		}
+	})
+
+	t.Run("disable-pac", func(t *testing.T) {
+
+		tc, err := clients.Operator.TektonConfigs().Get(context.TODO(), v1alpha1.ConfigResourceName, metav1.GetOptions{})
+		if err != nil {
+			t.Fatalf("failed to get tektonconfig: %v", err)
+		}
+
+		// Now, disable pipelinesAsCode
+		tc.Spec.Addon.EnablePAC = ptr.Bool(false)
+
+		tc, err = clients.Operator.TektonConfigs().Update(context.TODO(), tc, metav1.UpdateOptions{})
+		if err != nil {
+			t.Fatalf("failed to update tektonconfig: %v", err)
+		}
+
+		// wait till the installer set is deleted
+		time.Sleep(time.Second * 10)
+
+		ls := metav1.LabelSelector{
+			MatchLabels: map[string]string{
+				v1alpha1.CreatedByKey: "TektonAddon",
+			},
+		}
+		labelSelector, err := common.LabelSelector(ls)
+		if err != nil {
+			t.Fatal(err)
+		}
+		addonsIS, err := clients.Operator.TektonInstallerSets().List(context.TODO(), metav1.ListOptions{
+			LabelSelector: labelSelector,
+		})
+		if err != nil {
+			t.Fatalf("failed to get InstallerSet: %v", err)
+		}
 		// Now, there must be 3 installerSet
 		if len(addonsIS.Items) != 3 {
-			t.Fatalf("expected 3 installerSets after disabling params for Addon but got %v", len(addonsIS.Items))
+			t.Fatalf("expected 3 installerSets after disabling pac for Addon but got %v", len(addonsIS.Items))
 		}
 	})
 }
diff --git a/test/e2e/kubernetes/tektondashboarddeployment_test.go b/test/e2e/kubernetes/tektondashboarddeployment_test.go
index f6bd0aee5..ad26ec092 100644
--- a/test/e2e/kubernetes/tektondashboarddeployment_test.go
+++ b/test/e2e/kubernetes/tektondashboarddeployment_test.go
@@ -74,6 +74,11 @@ func TestTektonDashboardsDeployment(t *testing.T) {
 		resources.AssertTektonDashboardCRReadyStatus(t, clients, crNames)
 	})
 
+	// Test if TektonInstallerSets are created.
+	t.Run("verify-dashboard-installersets", func(t *testing.T) {
+		resources.AssertDashboardInstallerSets(t, clients)
+	})
+
 	// Delete the TektonDashboard CR instance to see if all resources will be removed
 	t.Run("delete-dashboard", func(t *testing.T) {
 		resources.AssertTektonDashboardCRReadyStatus(t, clients, crNames)
diff --git a/test/resources/tektonaddons.go b/test/resources/tektonaddons.go
index a522e8286..1854d3a73 100644
--- a/test/resources/tektonaddons.go
+++ b/test/resources/tektonaddons.go
@@ -96,15 +96,15 @@ func AssertTektonAddonCRReadyStatus(t *testing.T, clients *utils.Clients, names
 
 // AssertTektonInstallerSets verifies if the TektonInstallerSets are created.
 func AssertTektonInstallerSets(t *testing.T, clients *utils.Clients) {
-	assertInstallerSetsForAddon(t, clients, tektonaddon.ClusterTaskInstallerSet)
-	assertInstallerSetsForAddon(t, clients, tektonaddon.VersionedClusterTaskInstallerSet)
-	assertInstallerSetsForAddon(t, clients, tektonaddon.PipelinesTemplateInstallerSet)
-	assertInstallerSetsForAddon(t, clients, tektonaddon.TriggersResourcesInstallerSet)
-	assertInstallerSetsForAddon(t, clients, tektonaddon.ConsoleCLIInstallerSet)
-	assertInstallerSetsForAddon(t, clients, tektonaddon.MiscellaneousResourcesInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.ClusterTaskInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.VersionedClusterTaskInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.PipelinesTemplateInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.TriggersResourcesInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.ConsoleCLIInstallerSet)
+	assertInstallerSets(t, clients, tektonaddon.MiscellaneousResourcesInstallerSet)
 }
 
-func assertInstallerSetsForAddon(t *testing.T, clients *utils.Clients, component string) {
+func assertInstallerSets(t *testing.T, clients *utils.Clients, component string) {
 	ls := metav1.LabelSelector{
 		MatchLabels: map[string]string{
 			v1alpha1.InstallerSetType: component,
@@ -114,13 +114,13 @@ func assertInstallerSetsForAddon(t *testing.T, clients *utils.Clients, component
 	if err != nil {
 		t.Fatal(err)
 	}
-	clusterTaskIS, err := clients.TektonInstallerSet().List(context.TODO(), metav1.ListOptions{
+	installerSets, err := clients.TektonInstallerSet().List(context.TODO(), metav1.ListOptions{
 		LabelSelector: labelSelector,
 	})
 	if err != nil {
 		t.Fatalf("failed to get TektonInstallerSet for %s : %v", component, err)
 	}
-	if len(clusterTaskIS.Items) > 1 {
+	if len(installerSets.Items) > 1 {
 		t.Fatalf("multiple installer sets for %s TektonInstallerSet", component)
 	}
 }
diff --git a/test/resources/tektondashboards.go b/test/resources/tektondashboards.go
index 85bf4bd91..50c6507e8 100644
--- a/test/resources/tektondashboards.go
+++ b/test/resources/tektondashboards.go
@@ -83,6 +83,11 @@ func IsTektonDashboardReady(s *v1alpha1.TektonDashboard, err error) (bool, error
 	return s.Status.IsReady(), err
 }
 
+// AssertTektonInstallerSets verifies if the TektonInstallerSets are created.
+func AssertDashboardInstallerSets(t *testing.T, clients *utils.Clients) {
+	assertInstallerSets(t, clients, v1alpha1.DashboardResourceName)
+}
+
 // AssertTektonDashboardCRReadyStatus verifies if the TektonDashboard reaches the READY status.
 func AssertTektonDashboardCRReadyStatus(t *testing.T, clients *utils.Clients, names utils.ResourceNames) {
 	if _, err := WaitForTektonDashboardState(clients.TektonDashboard(), names.TektonDashboard,