Repository: tektoncd/operator
Pull request #1029 (2022-08-30 08:27:25)

TektonInstallerSet Improvement: uses hash to decide updating a resource
EnsureResources now creates a resource if doesn't exist.
while creating the resources it adds the hash of the resource
as an annotation.

in further reconcilation it checks if that hash is changed if not
then ignore the resource.

lets say a feature flag is changed then resources will be updated in
installerset spec and its hash will be different from previous.
so now it updates the resource.

this also disables the manifestival logger and log explicitly.

Signed-off-by: Shivam Mukhade <smukhade@redhat.com>

# Changes

<!-- 🎉🎉🎉 Thank you for the PR!!! 🎉🎉🎉 -->

<!-- Describe your changes here- ideally you can get that description straight from
your descriptive commit message(s)! -->

# Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

- [x] Run `make test lint` before submitting a PR
- [x] Includes [tests](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if functionality changed/added)
- [x] Includes [docs](https://github.com/tektoncd/community/blob/master/standards.md#principles) (if user facing)
- [x] Commit messages follow [commit message best practices](https://github.com/tektoncd/community/blob/master/standards.md#commit-messages)

_See [the contribution guide](https://github.com/tektoncd/operator/blob/master/CONTRIBUTING.md) for more details._

# Release Notes

<!--
Describe any user facing changes here, or delete this block.

Examples of user facing changes:
- API changes
- Bug fixes
- Any changes in behavior
- Changes requiring upgrade notices or deprecation warnings

For pull requests with a release note:

```release-note
Your release note here
```

For pull requests that require additional action from users switching to the new release, include the string "action required" (case insensitive) in the release note:

```release-note
action required: your release note here
```

For pull requests that don't need to be mentioned at release time, use the `/release-note-none` Prow command to add the `release-note-none` label to the PR. You can also write the string "NONE" as a release note in your PR description:

```release-note
NONE
```
-->


tekton-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please ask for approval from **sm43** after the PR has been reviewed.

The full list of commands accepted by this bot can be found [here](https://prow.tekton.dev/command-help?repo=tektoncd%2Foperator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/tektoncd/operator/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["sm43"]} -->
tekton-robot: The following is the coverage report on the affected files.
Say `/test pull-tekton-operator-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/reconciler/kubernetes/tektoninstallerset/install.go](https://storage.cloud.google.com/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-go-coverage/1564530220743004160/artifacts/line-cov.html#file0) | 77.8% | 40.8% | -37.0

tekton-robot: @sm43: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-tekton-operator-build-tests | 9a88d7c6c54152ab495997c94cfe51fb11d39b51 | [link](https://prow.tekton.dev/view/gs/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-build-tests/1564530220596203520) | true | `/test pull-tekton-operator-build-tests`

[Full PR test history](https://prow.tekton.dev/pr-history?org=tektoncd&repo=operator&pr=1029). [Your PR dashboard](https://prow.tekton.dev/pr?query=is%3Apr%20state%3Aopen%20author%3Asm43).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: The following is the coverage report on the affected files.
Say `/test pull-tekton-operator-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/reconciler/kubernetes/tektoninstallerset/install.go](https://storage.cloud.google.com/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-go-coverage/1564563809903841280/artifacts/line-cov.html#file0) | 77.8% | 40.8% | -37.0

tekton-robot: @sm43: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-tekton-operator-build-tests | d5509bc50dce72b21fb723e08cc0ca350bbcc219 | [link](https://prow.tekton.dev/view/gs/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-build-tests/1564563809769623552) | true | `/test pull-tekton-operator-build-tests`

[Full PR test history](https://prow.tekton.dev/pr-history?org=tektoncd&repo=operator&pr=1029). [Your PR dashboard](https://prow.tekton.dev/pr?query=is%3Apr%20state%3Aopen%20author%3Asm43).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: The following is the coverage report on the affected files.
Say `/test pull-tekton-operator-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/reconciler/kubernetes/tektoninstallerset/install.go](https://storage.cloud.google.com/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-go-coverage/1564569391431946240/artifacts/line-cov.html#file0) | 77.8% | 42.8% | -35.0

tekton-robot: @sm43: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-tekton-operator-build-tests | f3d5f6655bc02daca3c7049a66f26e67d8b33cf1 | [link](https://prow.tekton.dev/view/gs/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-build-tests/1564569391327088640) | true | `/test pull-tekton-operator-build-tests`

[Full PR test history](https://prow.tekton.dev/pr-history?org=tektoncd&repo=operator&pr=1029). [Your PR dashboard](https://prow.tekton.dev/pr?query=is%3Apr%20state%3Aopen%20author%3Asm43).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tekton-robot: The following is the coverage report on the affected files.
Say `/test pull-tekton-operator-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/reconciler/kubernetes/tektoninstallerset/install.go](https://storage.cloud.google.com/tekton-prow/pr-logs/pull/tektoncd_operator/1029/pull-tekton-operator-go-coverage/1564575748247785472/artifacts/line-cov.html#file0) | 77.8% | 43.9% | -33.9


Patch:
diff --git a/components.yaml b/components.yaml
index ff464a4fc..9f4065f68 100644
--- a/components.yaml
+++ b/components.yaml
@@ -1,6 +1,6 @@
 chains:
   github: tektoncd/chains
-  version: v0.11.0
+  version: v0.12.0
 dashboard:
   github: tektoncd/dashboard
   version: v0.28.0
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/install.go b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
index d32313554..b27e4e1ba 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/install.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/install.go
@@ -24,6 +24,7 @@ import (
 	mf "github.com/manifestival/manifestival"
 	"github.com/tektoncd/operator/pkg/apis/operator/v1alpha1"
 	"github.com/tektoncd/operator/pkg/reconciler/shared/hash"
+	"go.uber.org/zap"
 	appsv1 "k8s.io/api/apps/v1"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
@@ -77,42 +78,72 @@ var (
 
 type installer struct {
 	Manifest mf.Manifest
+	MfClient mf.Client
+	Logger   *zap.SugaredLogger
 }
 
-func ensureResources(mani *mf.Manifest) error {
-	freshCreate := true
-	// check if all resources in a given set of resources exists
-	ok, err := allResourcesExists(mani)
-	if err != nil {
-		if !apierrs.IsNotFound(err) {
+func (i *installer) ensureResources(manifest *mf.Manifest) error {
+	for _, r := range manifest.Resources() {
+		expectedHash, err := hash.Compute(r.Object)
+		if err != nil {
 			return err
 		}
-	}
-	// if error == NotFound error or !ok
-	// then Apply all resources
-	// if ok, that means the reosource already exists
-	// but this reconcile could be an modification eg: concig-defaults change
-	// set freshCreate flag to false, and then Apply all
-	// so that we can skip (break) RECONCILE_AGAIN loop path
-	if ok {
-		freshCreate = false
-	}
 
-	if err := mani.Apply(); err != nil {
-		return err
-	}
-	// on err == nil after Apply() return RECONCILE_AGAIN
-	// if freshCreate == true return RECONCILE_AGAIN
-	// this ensures freshly created resources are in place before proceeding to next stages of reconciler
-	if freshCreate {
-		return v1alpha1.RECONCILE_AGAIN_ERR
+		i.Logger.Infof("fetching resource %s: %s/%s", r.GetKind(), r.GetNamespace(), r.GetName())
+
+		res, err := i.MfClient.Get(&r)
+		if err != nil {
+			if apierrs.IsNotFound(err) {
+				i.Logger.Infof("resource not found, creating %s: %s/%s", r.GetKind(), r.GetNamespace(), r.GetName())
+				// add hash on the resource of expected manifest and create
+				anno := r.GetAnnotations()
+				if anno == nil {
+					anno = map[string]string{}
+				}
+				anno[v1alpha1.LastAppliedHashKey] = expectedHash
+				r.SetAnnotations(anno)
+				err = i.MfClient.Create(&r)
+				if err != nil {
+					return err
+				}
+				continue
+			}
+			return err
+		}
+
+		i.Logger.Infof("found resource %s: %s/%s, checking for update!", r.GetKind(), r.GetNamespace(), r.GetName())
+
+		// if resource exist then check if expected hash is different from the one
+		// on the resource
+		hashOnResource := res.GetAnnotations()[v1alpha1.LastAppliedHashKey]
+
+		if expectedHash == hashOnResource {
+			continue
+		}
+
+		i.Logger.Infof("updating resource %s: %s/%s", r.GetKind(), r.GetNamespace(), r.GetName())
+
+		anno := r.GetAnnotations()
+		if anno == nil {
+			anno = map[string]string{}
+		}
+		anno[v1alpha1.LastAppliedHashKey] = expectedHash
+		r.SetAnnotations(anno)
+
+		installManifests, err := mf.ManifestFrom(mf.Slice([]unstructured.Unstructured{r}), mf.UseClient(i.MfClient))
+		if err != nil {
+			return err
+		}
+		if err := installManifests.Apply(); err != nil {
+			return err
+		}
 	}
 	return nil
 }
 
 func (i *installer) EnsureCRDs() error {
 	resourceList := i.Manifest.Filter(mf.Any(mf.CRDs))
-	return ensureResources(&resourceList)
+	return i.ensureResources(&resourceList)
 }
 
 func (i *installer) EnsureClusterScopedResources() error {
@@ -131,7 +162,7 @@ func (i *installer) EnsureClusterScopedResources() error {
 			ConsoleYAMLSamplePred,
 			securityContextConstraints,
 		))
-	return ensureResources(&resourceList)
+	return i.ensureResources(&resourceList)
 }
 
 func (i *installer) EnsureNamespaceScopedResources() error {
@@ -156,7 +187,7 @@ func (i *installer) EnsureNamespaceScopedResources() error {
 			routePred,
 			statefulSetPred,
 		))
-	return ensureResources(&resourceList)
+	return i.ensureResources(&resourceList)
 }
 
 func (i *installer) EnsureDeploymentResources() error {
@@ -420,28 +451,6 @@ func isDeploymentAvailable(d *appsv1.Deployment) bool {
 	return false
 }
 
-func allResourcesExists(m *mf.Manifest) (bool, error) {
-	c := m.Client
-	for _, item := range m.Resources() {
-		ok, err := resourceExists(c, &item)
-		if err != nil {
-			return false, err
-		}
-		if !ok {
-			return false, nil
-		}
-	}
-	return true, nil
-}
-
-func resourceExists(c mf.Client, u *unstructured.Unstructured) (bool, error) {
-	_, err := c.Get(u)
-	if err != nil {
-		return false, err
-	}
-	return true, nil
-}
-
 func isJobCompleted(d *batchv1.Job) bool {
 	for _, c := range d.Status.Conditions {
 		if c.Type == batchv1.JobComplete && c.Status == corev1.ConditionTrue {
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go b/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
index ef92b03ea..c7fbccb40 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/tektoninstallerset.go
@@ -74,9 +74,9 @@ func getReference(tis *v1alpha1.TektonInstallerSet) []v1.OwnerReference {
 // converge the two.
 func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.TektonInstallerSet) pkgreconciler.Event {
 	installerSet.Status.InitializeConditions()
-	logger := logging.FromContext(ctx)
+	logger := logging.FromContext(ctx).With("installerSet", fmt.Sprintf("%s/%s", installerSet.Namespace, installerSet.Name))
 
-	installManifests, err := mf.ManifestFrom(installerSet.Spec.Manifests, mf.UseClient(r.mfClient), mf.UseLogger(r.mfLogger))
+	installManifests, err := mf.ManifestFrom(installerSet.Spec.Manifests, mf.UseClient(r.mfClient))
 	if err != nil {
 		logger.Error("Error creating initial manifest: ", err)
 		installerSet.Status.MarkNotReady(fmt.Sprintf("Internal Error: failed to create manifest: %s", err.Error()))
@@ -100,6 +100,8 @@ func (r *Reconciler) ReconcileKind(ctx context.Context, installerSet *v1alpha1.T
 
 	installer := installer{
 		Manifest: installManifests,
+		MfClient: r.mfClient,
+		Logger:   logger,
 	}
 
 	// Install CRDs
diff --git a/third_party/github.com/hashicorp/go-retryablehttp/README.md b/third_party/github.com/hashicorp/go-retryablehttp/README.md
index 8943becf1..09f5eaf22 100644
--- a/third_party/github.com/hashicorp/go-retryablehttp/README.md
+++ b/third_party/github.com/hashicorp/go-retryablehttp/README.md
@@ -45,6 +45,25 @@ The returned response object is an `*http.Response`, the same thing you would
 usually get from `net/http`. Had the request failed one or more times, the above
 call would block and retry with exponential backoff.
 
+## Retrying cases that fail after a seeming success
+
+It's possible for a request to succeed in the sense that the expected response headers are received, but then to encounter network-level errors while reading the response body. In go-retryablehttp's most basic usage, this error would not be retryable, due to the out-of-band handling of the response body. In some cases it may be desirable to handle the response body as part of the retryable operation.
+
+A toy example (which will retry the full request and succeed on the second attempt) is shown below:
+
+```go
+c := retryablehttp.NewClient()
+r := retryablehttp.NewRequest("GET", "://foo", nil)
+handlerShouldRetry := true
+r.SetResponseHandler(func(*http.Response) error {
+    if !handlerShouldRetry {
+        return nil
+    }
+    handlerShouldRetry = false
+    return errors.New("retryable error")
+})
+```
+
 ## Getting a stdlib `*http.Client` with retries
 
 It's possible to convert a `*retryablehttp.Client` directly to a `*http.Client`.
diff --git a/third_party/github.com/hashicorp/go-retryablehttp/client.go b/third_party/github.com/hashicorp/go-retryablehttp/client.go
index adbdd92e3..57116e960 100644
--- a/third_party/github.com/hashicorp/go-retryablehttp/client.go
+++ b/third_party/github.com/hashicorp/go-retryablehttp/client.go
@@ -69,11 +69,21 @@ var (
 	// scheme specified in the URL is invalid. This error isn't typed
 	// specifically so we resort to matching on the error string.
 	schemeErrorRe = regexp.MustCompile(`unsupported protocol scheme`)
+
+	// A regular expression to match the error returned by net/http when the
+	// TLS certificate is not trusted. This error isn't typed
+	// specifically so we resort to matching on the error string.
+	notTrustedErrorRe = regexp.MustCompile(`certificate is not trusted`)
 )
 
 // ReaderFunc is the type of function that can be given natively to NewRequest
 type ReaderFunc func() (io.Reader, error)
 
+// ResponseHandlerFunc is a type of function that takes in a Response, and does something with it.
+// It only runs if the initial part of the request was successful.
+// If an error is returned, the client's retry policy will be used to determine whether to retry the whole request.
+type ResponseHandlerFunc func(*http.Response) error
+
 // LenReader is an interface implemented by many in-memory io.Reader's. Used
 // for automatically sending the right Content-Length header when possible.
 type LenReader interface {
@@ -86,6 +96,8 @@ type Request struct {
 	// used to rewind the request data in between retries.
 	body ReaderFunc
 
+	responseHandler ResponseHandlerFunc
+
 	// Embed an HTTP request directly. This makes a *Request act exactly
 	// like an *http.Request so that all meta methods are supported.
 	*http.Request
@@ -94,8 +106,16 @@ type Request struct {
 // WithContext returns wrapped Request with a shallow copy of underlying *http.Request
 // with its context changed to ctx. The provided ctx must be non-nil.
 func (r *Request) WithContext(ctx context.Context) *Request {
-	r.Request = r.Request.WithContext(ctx)
-	return r
+	return &Request{
+		body:            r.body,
+		responseHandler: r.responseHandler,
+		Request:         r.Request.WithContext(ctx),
+	}
+}
+
+// SetResponseHandler allows setting the response handler.
+func (r *Request) SetResponseHandler(fn ResponseHandlerFunc) {
+	r.responseHandler = fn
 }
 
 // BodyBytes allows accessing the request body. It is an analogue to
@@ -252,23 +272,31 @@ func FromRequest(r *http.Request) (*Request, error) {
 		return nil, err
 	}
 	// Could assert contentLength == r.ContentLength
-	return &Request{bodyReader, r}, nil
+	return &Request{body: bodyReader, Request: r}, nil
 }
 
 // NewRequest creates a new wrapped request.
 func NewRequest(method, url string, rawBody interface{}) (*Request, error) {
+	return NewRequestWithContext(context.Background(), method, url, rawBody)
+}
+
+// NewRequestWithContext creates a new wrapped request with the provided context.
+//
+// The context controls the entire lifetime of a request and its response:
+// obtaining a connection, sending the request, and reading the response headers and body.
+func NewRequestWithContext(ctx context.Context, method, url string, rawBody interface{}) (*Request, error) {
 	bodyReader, contentLength, err := getBodyReaderAndContentLength(rawBody)
 	if err != nil {
 		return nil, err
 	}
 
-	httpReq, err := http.NewRequest(method, url, nil)
+	httpReq, err := http.NewRequestWithContext(ctx, method, url, nil)
 	if err != nil {
 		return nil, err
 	}
 	httpReq.ContentLength = contentLength
 
-	return &Request{bodyReader, httpReq}, nil
+	return &Request{body: bodyReader, Request: httpReq}, nil
 }
 
 // Logger interface allows to use other loggers than
@@ -435,6 +463,9 @@ func baseRetryPolicy(resp *http.Response, err error) (bool, error) {
 			}
 
 			// Don't retry if the error was due to TLS cert verification failure.
+			if notTrustedErrorRe.MatchString(v.Error()) {
+				return false, v
+			}
 			if _, ok := v.Err.(x509.UnknownAuthorityError); ok {
 				return false, v
 			}
@@ -455,7 +486,7 @@ func baseRetryPolicy(resp *http.Response, err error) (bool, error) {
 	// the server time to recover, as 500's are typically not permanent
 	// errors and may relate to outages on the server side. This will catch
 	// invalid response codes as well, like 0 and 999.
-	if resp.StatusCode == 0 || (resp.StatusCode >= 500 && resp.StatusCode != 501) {
+	if resp.StatusCode == 0 || (resp.StatusCode >= 500 && resp.StatusCode != http.StatusNotImplemented) {
 		return true, fmt.Errorf("unexpected HTTP status %s", resp.Status)
 	}
 
@@ -555,13 +586,12 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
 	var resp *http.Response
 	var attempt int
 	var shouldRetry bool
-	var doErr, checkErr error
+	var doErr, respErr, checkErr error
 
 	for i := 0; ; i++ {
+		doErr, respErr = nil, nil
 		attempt++
 
-		var code int // HTTP response code
-
 		// Always rewind the request body when non-nil.
 		if req.body != nil {
 			body, err := req.body()
@@ -589,19 +619,24 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
 
 		// Attempt the request
 		resp, doErr = c.HTTPClient.Do(req.Request)
-		if resp != nil {
-			code = resp.StatusCode
-		}
 
 		// Check if we should continue with retries.
 		shouldRetry, checkErr = c.CheckRetry(req.Context(), resp, doErr)
+		if !shouldRetry && doErr == nil && req.responseHandler != nil {
+			respErr = req.responseHandler(resp)
+			shouldRetry, checkErr = c.CheckRetry(req.Context(), resp, respErr)
+		}
 
-		if doErr != nil {
+		err := doErr
+		if respErr != nil {
+			err = respErr
+		}
+		if err != nil {
 			switch v := logger.(type) {
 			case LeveledLogger:
-				v.Error("request failed", "error", doErr, "method", req.Method, "url", req.URL)
+				v.Error("request failed", "error", err, "method", req.Method, "url", req.URL)
 			case Logger:
-				v.Printf("[ERR] %s %s request failed: %v", req.Method, req.URL, doErr)
+				v.Printf("[ERR] %s %s request failed: %v", req.Method, req.URL, err)
 			}
 		} else {
 			// Call this here to maintain the behavior of logging all requests,
@@ -636,11 +671,11 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
 		}
 
 		wait := c.Backoff(c.RetryWaitMin, c.RetryWaitMax, i, resp)
-		desc := fmt.Sprintf("%s %s", req.Method, req.URL)
-		if code > 0 {
-			desc = fmt.Sprintf("%s (status: %d)", desc, code)
-		}
 		if logger != nil {
+			desc := fmt.Sprintf("%s %s", req.Method, req.URL)
+			if resp != nil {
+				desc = fmt.Sprintf("%s (status: %d)", desc, resp.StatusCode)
+			}
 			switch v := logger.(type) {
 			case LeveledLogger:
 				v.Debug("retrying request", "request", desc, "timeout", wait, "remaining", remain)
@@ -648,11 +683,13 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
 				v.Printf("[DEBUG] %s: retrying in %s (%d left)", desc, wait, remain)
 			}
 		}
+		timer := time.NewTimer(wait)
 		select {
 		case <-req.Context().Done():
+			timer.Stop()
 			c.HTTPClient.CloseIdleConnections()
 			return nil, req.Context().Err()
-		case <-time.After(wait):
+		case <-timer.C:
 		}
 
 		// Make shallow copy of http Request so that we can modify its body
@@ -662,15 +699,19 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
 	}
 
 	// this is the closest we have to success criteria
-	if doErr == nil && checkErr == nil && !shouldRetry {
+	if doErr == nil && respErr == nil && checkErr == nil && !shouldRetry {
 		return resp, nil
 	}
 
 	defer c.HTTPClient.CloseIdleConnections()
 
-	err := doErr
+	var err error
 	if checkErr != nil {
 		err = checkErr
+	} else if respErr != nil {
+		err = respErr
+	} else {
+		err = doErr
 	}
 
 	if c.ErrorHandler != nil {
diff --git a/third_party/github.com/hashicorp/go-rootcerts/rootcerts_base.go b/third_party/github.com/hashicorp/go-rootcerts/rootcerts_base.go
index ba58717f5..66b1472c4 100644
--- a/third_party/github.com/hashicorp/go-rootcerts/rootcerts_base.go
+++ b/third_party/github.com/hashicorp/go-rootcerts/rootcerts_base.go
@@ -1,4 +1,3 @@
-//go:build !darwin
 // +build !darwin
 
 package rootcerts
diff --git a/third_party/github.com/hashicorp/go-secure-stdlib/parseutil/parsepath.go b/third_party/github.com/hashicorp/go-secure-stdlib/parseutil/parsepath.go
index 45e1497ca..d59ecbb2b 100644
--- a/third_party/github.com/hashicorp/go-secure-stdlib/parseutil/parsepath.go
+++ b/third_party/github.com/hashicorp/go-secure-stdlib/parseutil/parsepath.go
@@ -9,7 +9,10 @@ import (
 	"strings"
 )
 
-var ErrNotAUrl = errors.New("not a url")
+var (
+	ErrNotAUrl   = errors.New("not a url")
+	ErrNotParsed = errors.New("not a parsed value")
+)
 
 // ParsePath parses a URL with schemes file://, env://, or any other. Depending
 // on the scheme it will return specific types of data:
@@ -18,7 +21,10 @@ var ErrNotAUrl = errors.New("not a url")
 //
 // * env:// will return a string with the env var's contents
 //
-// * Anything else will return the string as it was
+// * Anything else will return the string as it was. Functionally this means
+// anything for which Go's `url.Parse` function does not throw an error. If you
+// want to ensure that this function errors if a known scheme is not found, use
+// MustParsePath.
 //
 // On error, we return the original string along with the error. The caller can
 // switch on errors.Is(err, ErrNotAUrl) to understand whether it was the parsing
@@ -26,6 +32,16 @@ var ErrNotAUrl = errors.New("not a url")
 // useful to attempt to read a non-URL string from some resource, but where the
 // original input may simply be a valid string of that type.
 func ParsePath(path string) (string, error) {
+	return parsePath(path, false)
+}
+
+// MustParsePath behaves like ParsePath but will return ErrNotAUrl if the value
+// is not a URL with a scheme that can be parsed by this function.
+func MustParsePath(path string) (string, error) {
+	return parsePath(path, true)
+}
+
+func parsePath(path string, mustParse bool) (string, error) {
 	path = strings.TrimSpace(path)
 	parsed, err := url.Parse(path)
 	if err != nil {
@@ -40,7 +56,10 @@ func ParsePath(path string) (string, error) {
 		return strings.TrimSpace(string(contents)), nil
 	case "env":
 		return strings.TrimSpace(os.Getenv(strings.TrimPrefix(path, "env://"))), nil
+	default:
+		if mustParse {
+			return "", ErrNotParsed
+		}
+		return path, nil
 	}
-
-	return path, nil
 }
diff --git a/third_party/github.com/hashicorp/go-sockaddr/ifaddrs.go b/third_party/github.com/hashicorp/go-sockaddr/ifaddrs.go
index d58cb50af..80f61bef6 100644
--- a/third_party/github.com/hashicorp/go-sockaddr/ifaddrs.go
+++ b/third_party/github.com/hashicorp/go-sockaddr/ifaddrs.go
@@ -1214,7 +1214,7 @@ func parseDefaultIfNameFromIPCmd(routeOut string) (string, error) {
 // Android.
 func parseDefaultIfNameFromIPCmdAndroid(routeOut string) (string, error) {
 	parsedLines := parseIfNameFromIPCmd(routeOut)
-	if len(parsedLines) > 0 {
+	if (len(parsedLines) > 0) {
 		ifName := strings.TrimSpace(parsedLines[0][4])
 		return ifName, nil
 	}
@@ -1222,6 +1222,7 @@ func parseDefaultIfNameFromIPCmdAndroid(routeOut string) (string, error) {
 	return "", errors.New("No default interface found")
 }
 
+
 // parseIfNameFromIPCmd parses interfaces from ip(8) for
 // Linux.
 func parseIfNameFromIPCmd(routeOut string) [][]string {
diff --git a/third_party/github.com/hashicorp/go-sockaddr/route_info_android.go b/third_party/github.com/hashicorp/go-sockaddr/route_info_android.go
index 6f484cc44..9885915a6 100644
--- a/third_party/github.com/hashicorp/go-sockaddr/route_info_android.go
+++ b/third_party/github.com/hashicorp/go-sockaddr/route_info_android.go
@@ -25,6 +25,7 @@ func (ri routeInfo) GetDefaultInterfaceName() (string, error) {
 		return "", err
 	}
 
+
 	var ifName string
 	if ifName, err = parseDefaultIfNameFromIPCmdAndroid(string(out)); err != nil {
 		return "", errors.New("No default interface found")
diff --git a/third_party/github.com/hashicorp/go-sockaddr/route_info_bsd.go b/third_party/github.com/hashicorp/go-sockaddr/route_info_bsd.go
index 89308cf7c..705757abc 100644
--- a/third_party/github.com/hashicorp/go-sockaddr/route_info_bsd.go
+++ b/third_party/github.com/hashicorp/go-sockaddr/route_info_bsd.go
@@ -1,4 +1,3 @@
-//go:build darwin || dragonfly || freebsd || netbsd || openbsd
 // +build darwin dragonfly freebsd netbsd openbsd
 
 package sockaddr
diff --git a/third_party/github.com/hashicorp/go-sockaddr/route_info_default.go b/third_party/github.com/hashicorp/go-sockaddr/route_info_default.go
index fe291fb0d..d1b009f65 100644
--- a/third_party/github.com/hashicorp/go-sockaddr/route_info_default.go
+++ b/third_party/github.com/hashicorp/go-sockaddr/route_info_default.go
@@ -1,4 +1,3 @@
-//go:build android || nacl || plan9
 // +build android nacl plan9
 
 package sockaddr
diff --git a/third_party/github.com/hashicorp/go-sockaddr/route_info_linux.go b/third_party/github.com/hashicorp/go-sockaddr/route_info_linux.go
index 1b2720488..b62ce6ecb 100644
--- a/third_party/github.com/hashicorp/go-sockaddr/route_info_linux.go
+++ b/third_party/github.com/hashicorp/go-sockaddr/route_info_linux.go
@@ -1,4 +1,3 @@
-//go:build !android
 // +build !android
 
 package sockaddr
diff --git a/third_party/github.com/hashicorp/golang-lru/simplelru/lru_interface.go b/third_party/github.com/hashicorp/golang-lru/simplelru/lru_interface.go
index a0b97e3f7..92d70934d 100644
--- a/third_party/github.com/hashicorp/golang-lru/simplelru/lru_interface.go
+++ b/third_party/github.com/hashicorp/golang-lru/simplelru/lru_interface.go
@@ -34,6 +34,6 @@ type LRUCache interface {
 	// Clears all cache entries.
 	Purge()
 
-	// Resizes cache, returning number evicted
-	Resize(int) int
+  // Resizes cache, returning number evicted
+  Resize(int) int
 }
diff --git a/third_party/github.com/thlib/go-timezone-local/tzlocal/LICENSE b/third_party/github.com/thlib/go-timezone-local/tzlocal/LICENSE
new file mode 100644
index 000000000..fdddb29aa
--- /dev/null
+++ b/third_party/github.com/thlib/go-timezone-local/tzlocal/LICENSE
@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <https://unlicense.org>
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 35fa00d89..93f38d491 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -1177,6 +1177,7 @@ go.uber.org/zap/internal/ztest
 go.uber.org/zap/zapcore
 go.uber.org/zap/zapgrpc
 go.uber.org/zap/zaptest
+go.uber.org/zap/zaptest/observer
 # golang.org/x/crypto v0.0.0-20220214200702-86341886e292
 ## explicit; go 1.17
 golang.org/x/crypto/bcrypt


Test patch:
diff --git a/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go b/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
index 84e8cde5e..6bf600e47 100644
--- a/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
+++ b/pkg/reconciler/kubernetes/tektoninstallerset/install_test.go
@@ -18,81 +18,27 @@ package tektoninstallerset
 
 import (
 	"context"
-	"fmt"
 	"testing"
 
-	"github.com/google/go-cmp/cmp"
 	mf "github.com/manifestival/manifestival"
 	"github.com/manifestival/manifestival/fake"
 	"github.com/tektoncd/operator/pkg/apis/operator/v1alpha1"
+	"github.com/tektoncd/operator/pkg/reconciler/shared/hash"
+	"go.uber.org/zap"
+	zapobserver "go.uber.org/zap/zaptest/observer"
 	"gotest.tools/v3/assert"
 	appsv1 "k8s.io/api/apps/v1"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 )
 
 var (
-	namespace          = clusterScopedResource("v1", "Namespace", "test-ns")
-	clusterRole        = clusterScopedResource("rbac.authorization.k8s.io/v1", "ClusterRole", "test-cluster-role")
-	role               = namespacedResource("rbac.authorization.k8s.io/v1", "Role", "test", "test-role")
-	serviceAccount     = namespacedResource("v1", "ServiceAccount", "test", "test-service-account")
-	clusterRoleBinding = clusterScopedResource("rbac.authorization.k8s.io/v1", "ClusterRoleBinding", "test-cluster-role-binding")
-	roleBinding        = namespacedResource("rbac.authorization.k8s.io/v1", "RoleBinding", "test", "test-role-binding")
-	crd                = clusterScopedResource("apiextensions.k8s.io/v1", "CustomResourceDefinition", "test-crd")
-	secret             = namespacedResource("v1", "Secret", "test", "test-secret")
-	validatingWebhook  = clusterScopedResource("admissionregistration.k8s.io/v1", "ValidatingWebhookConfiguration", "test-validating-webhook")
-	mutatingWebhook    = clusterScopedResource("admissionregistration.k8s.io/v1", "MutatingWebhookConfiguration", "test-mutating-webhook")
-	configMap          = namespacedResource("v1", "ConfigMap", "test", "test-configmap")
-	deployment         = namespacedResource("apps/v1", "Deployment", "test", "test-deployment")
-	service            = namespacedResource("v1", "Service", "test", "test-service")
-	hpa                = namespacedResource("autoscaling/v2beta1", "HorizontalPodAutoscaler", "test", "test-hpa")
-	pvc                = namespacedResource("v1", "PersistentVolumeClaim", "test", "test-pvc")
-	job                = namespacedResource("batch/v1", "Job", "test", "test-job")
+	serviceAccount = namespacedResource("v1", "ServiceAccount", "test", "test-service-account")
 )
 
-type fakeClient struct {
-	err            error
-	getErr         error
-	createErr      error
-	resourcesExist bool
-	gets           []unstructured.Unstructured
-	creates        []unstructured.Unstructured
-	deletes        []unstructured.Unstructured
-}
-
-func (f *fakeClient) Get(obj *unstructured.Unstructured) (*unstructured.Unstructured, error) {
-	var resource *unstructured.Unstructured
-	if f.resourcesExist {
-		for _, item := range f.gets {
-			if obj.GetKind() == item.GetKind() && obj.GetName() == item.GetName() {
-				return &item, nil
-			}
-		}
-
-	}
-	return resource, f.getErr
-}
-
-func (f *fakeClient) Delete(obj *unstructured.Unstructured, options ...mf.DeleteOption) error {
-	f.deletes = append(f.deletes, *obj)
-	return f.err
-}
-
-func (f *fakeClient) Create(obj *unstructured.Unstructured, options ...mf.ApplyOption) error {
-	obj.SetAnnotations(nil) // Deleting the extra annotation. Irrelevant for the test.
-	f.creates = append(f.creates, *obj)
-	return f.createErr
-}
-
-func (f *fakeClient) Update(obj *unstructured.Unstructured, options ...mf.ApplyOption) error {
-	return f.err
-}
-
 // namespacedResource is an unstructured resource with the given apiVersion, kind, ns and name.
 func namespacedResource(apiVersion, kind, ns, name string) unstructured.Unstructured {
 	resource := unstructured.Unstructured{}
@@ -103,88 +49,68 @@ func namespacedResource(apiVersion, kind, ns, name string) unstructured.Unstruct
 	return resource
 }
 
-// clusterScopedResource is an unstructured resource with the given apiVersion, kind and name.
-func clusterScopedResource(apiVersion, kind, name string) unstructured.Unstructured {
-	return namespacedResource(apiVersion, kind, "", name)
-}
-
-func TestInstaller(t *testing.T) {
-	crd.SetDeletionTimestamp(&metav1.Time{})
-	in := []unstructured.Unstructured{namespace, deployment, clusterRole, role,
-		roleBinding, clusterRoleBinding, serviceAccount, crd, validatingWebhook, mutatingWebhook, configMap, service, hpa, secret, pvc, job}
+func TestEnsureResources_CreateResource(t *testing.T) {
+	fakeClient := fake.New()
+	observer, _ := zapobserver.New(zap.InfoLevel)
+	logger := zap.New(observer).Sugar()
 
-	client := &fakeClient{}
-	manifest, err := mf.ManifestFrom(mf.Slice(in), mf.UseClient(client))
+	manifest, err := mf.ManifestFrom(mf.Slice([]unstructured.Unstructured{serviceAccount}))
 	if err != nil {
 		t.Fatalf("Failed to generate manifest: %v", err)
 	}
 
 	i := installer{
 		Manifest: manifest,
+		MfClient: fakeClient,
+		Logger:   logger,
 	}
 
-	want := []unstructured.Unstructured{crd}
-
-	err = i.EnsureCRDs()
-	if err != nil {
-		t.Fatal("Unexpected Error while installing resources: ", err)
-	}
-
-	if len(want) != len(client.creates) {
-		t.Fatalf("Unexpected creates: %s", fmt.Sprintf("(-got, +want): %s", cmp.Diff(client.creates, want)))
-	}
-
-	// reset created array
-	client.creates = []unstructured.Unstructured{}
+	err = i.ensureResources(&manifest)
+	assert.NilError(t, err)
 
-	want = []unstructured.Unstructured{namespace, clusterRole, validatingWebhook, mutatingWebhook}
+	res, err := fakeClient.Get(&serviceAccount)
+	assert.NilError(t, err)
 
-	err = i.EnsureClusterScopedResources()
-	if err != nil {
-		t.Fatal("Unexpected Error while installing resources: ", err)
-	}
+	assert.Equal(t, res.GetNamespace(), serviceAccount.GetNamespace())
+	assert.Equal(t, res.GetName(), serviceAccount.GetName())
+}
 
-	if len(want) != len(client.creates) {
-		t.Fatalf("Unexpected creates: %s", fmt.Sprintf("(-got, +want): %s", cmp.Diff(client.creates, want)))
-	}
+func TestEnsureResources_UpdateResource(t *testing.T) {
+	// service account already exist on cluster
+	sa := serviceAccount
+	sa.SetAnnotations(map[string]string{
+		v1alpha1.LastAppliedHashKey: "abcd",
+	})
 
-	// reset created array
-	client.creates = []unstructured.Unstructured{}
+	fakeClient := fake.New(&sa)
+	observer, _ := zapobserver.New(zap.InfoLevel)
+	logger := zap.New(observer).Sugar()
 
-	want = []unstructured.Unstructured{serviceAccount, clusterRoleBinding, role,
-		roleBinding, configMap, secret, hpa, job, pvc, service}
+	// pass an updated sa with some different hash than existing
+	newSa := serviceAccount
 
-	err = i.EnsureNamespaceScopedResources()
+	manifest, err := mf.ManifestFrom(mf.Slice([]unstructured.Unstructured{newSa}))
 	if err != nil {
-		t.Fatal("Unexpected Error while installing resources: ", err)
+		t.Fatalf("Failed to generate manifest: %v", err)
 	}
 
-	if len(want) != len(client.creates) {
-		t.Fatalf("Unexpected creates: %s", fmt.Sprintf("(-got, +want): %s", cmp.Diff(client.creates, want)))
+	i := installer{
+		Manifest: manifest,
+		MfClient: fakeClient,
+		Logger:   logger,
 	}
 
-	// reset created array
-	client.creates = []unstructured.Unstructured{}
-	client.resourcesExist = false
-	client.getErr = errors.NewNotFound(schema.GroupResource{
-		Group:    "apps/v1",
-		Resource: "Deployment",
-	}, "test-deployment")
+	err = i.ensureResources(&manifest)
+	assert.NilError(t, err)
 
-	err = i.EnsureDeploymentResources()
-	assert.Error(t, err, v1alpha1.RECONCILE_AGAIN_ERR.Error())
+	res, err := fakeClient.Get(&serviceAccount)
+	assert.NilError(t, err)
 
-	want = []unstructured.Unstructured{deployment}
-	if len(want) != len(client.creates) {
-		t.Fatalf("Unexpected creates: %s", fmt.Sprintf("(-got, +want): %s", cmp.Diff(client.creates, want)))
-	}
-
-	client.resourcesExist = true
-	client.gets = []unstructured.Unstructured{deployment}
-	err = i.EnsureDeploymentResources()
-	if err != nil {
-		t.Fatal("Unexpected Error while installing resources: ", err)
-	}
+	assert.Equal(t, res.GetNamespace(), serviceAccount.GetNamespace())
+	assert.Equal(t, res.GetName(), serviceAccount.GetName())
+	expectedHash, err := hash.Compute(serviceAccount.Object)
+	assert.NilError(t, err)
+	assert.Equal(t, res.GetAnnotations()[v1alpha1.LastAppliedHashKey], expectedHash)
 }
 
 var (
diff --git a/vendor/go.uber.org/zap/zaptest/observer/logged_entry.go b/vendor/go.uber.org/zap/zaptest/observer/logged_entry.go
new file mode 100644
index 000000000..a4ea7ec36
--- /dev/null
+++ b/vendor/go.uber.org/zap/zaptest/observer/logged_entry.go
@@ -0,0 +1,39 @@
+// Copyright (c) 2017 Uber Technologies, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.
+
+package observer
+
+import "go.uber.org/zap/zapcore"
+
+// An LoggedEntry is an encoding-agnostic representation of a log message.
+// Field availability is context dependant.
+type LoggedEntry struct {
+	zapcore.Entry
+	Context []zapcore.Field
+}
+
+// ContextMap returns a map for all fields in Context.
+func (e LoggedEntry) ContextMap() map[string]interface{} {
+	encoder := zapcore.NewMapObjectEncoder()
+	for _, f := range e.Context {
+		f.AddTo(encoder)
+	}
+	return encoder.Fields
+}
diff --git a/vendor/go.uber.org/zap/zaptest/observer/observer.go b/vendor/go.uber.org/zap/zaptest/observer/observer.go
new file mode 100644
index 000000000..f77f1308b
--- /dev/null
+++ b/vendor/go.uber.org/zap/zaptest/observer/observer.go
@@ -0,0 +1,196 @@
+// Copyright (c) 2016-2022 Uber Technologies, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.
+
+// Package observer provides a zapcore.Core that keeps an in-memory,
+// encoding-agnostic representation of log entries. It's useful for
+// applications that want to unit test their log output without tying their
+// tests to a particular output encoding.
+package observer // import "go.uber.org/zap/zaptest/observer"
+
+import (
+	"strings"
+	"sync"
+	"time"
+
+	"go.uber.org/zap/internal"
+	"go.uber.org/zap/zapcore"
+)
+
+// ObservedLogs is a concurrency-safe, ordered collection of observed logs.
+type ObservedLogs struct {
+	mu   sync.RWMutex
+	logs []LoggedEntry
+}
+
+// Len returns the number of items in the collection.
+func (o *ObservedLogs) Len() int {
+	o.mu.RLock()
+	n := len(o.logs)
+	o.mu.RUnlock()
+	return n
+}
+
+// All returns a copy of all the observed logs.
+func (o *ObservedLogs) All() []LoggedEntry {
+	o.mu.RLock()
+	ret := make([]LoggedEntry, len(o.logs))
+	copy(ret, o.logs)
+	o.mu.RUnlock()
+	return ret
+}
+
+// TakeAll returns a copy of all the observed logs, and truncates the observed
+// slice.
+func (o *ObservedLogs) TakeAll() []LoggedEntry {
+	o.mu.Lock()
+	ret := o.logs
+	o.logs = nil
+	o.mu.Unlock()
+	return ret
+}
+
+// AllUntimed returns a copy of all the observed logs, but overwrites the
+// observed timestamps with time.Time's zero value. This is useful when making
+// assertions in tests.
+func (o *ObservedLogs) AllUntimed() []LoggedEntry {
+	ret := o.All()
+	for i := range ret {
+		ret[i].Time = time.Time{}
+	}
+	return ret
+}
+
+// FilterLevelExact filters entries to those logged at exactly the given level.
+func (o *ObservedLogs) FilterLevelExact(level zapcore.Level) *ObservedLogs {
+	return o.Filter(func(e LoggedEntry) bool {
+		return e.Level == level
+	})
+}
+
+// FilterMessage filters entries to those that have the specified message.
+func (o *ObservedLogs) FilterMessage(msg string) *ObservedLogs {
+	return o.Filter(func(e LoggedEntry) bool {
+		return e.Message == msg
+	})
+}
+
+// FilterMessageSnippet filters entries to those that have a message containing the specified snippet.
+func (o *ObservedLogs) FilterMessageSnippet(snippet string) *ObservedLogs {
+	return o.Filter(func(e LoggedEntry) bool {
+		return strings.Contains(e.Message, snippet)
+	})
+}
+
+// FilterField filters entries to those that have the specified field.
+func (o *ObservedLogs) FilterField(field zapcore.Field) *ObservedLogs {
+	return o.Filter(func(e LoggedEntry) bool {
+		for _, ctxField := range e.Context {
+			if ctxField.Equals(field) {
+				return true
+			}
+		}
+		return false
+	})
+}
+
+// FilterFieldKey filters entries to those that have the specified key.
+func (o *ObservedLogs) FilterFieldKey(key string) *ObservedLogs {
+	return o.Filter(func(e LoggedEntry) bool {
+		for _, ctxField := range e.Context {
+			if ctxField.Key == key {
+				return true
+			}
+		}
+		return false
+	})
+}
+
+// Filter returns a copy of this ObservedLogs containing only those entries
+// for which the provided function returns true.
+func (o *ObservedLogs) Filter(keep func(LoggedEntry) bool) *ObservedLogs {
+	o.mu.RLock()
+	defer o.mu.RUnlock()
+
+	var filtered []LoggedEntry
+	for _, entry := range o.logs {
+		if keep(entry) {
+			filtered = append(filtered, entry)
+		}
+	}
+	return &ObservedLogs{logs: filtered}
+}
+
+func (o *ObservedLogs) add(log LoggedEntry) {
+	o.mu.Lock()
+	o.logs = append(o.logs, log)
+	o.mu.Unlock()
+}
+
+// New creates a new Core that buffers logs in memory (without any encoding).
+// It's particularly useful in tests.
+func New(enab zapcore.LevelEnabler) (zapcore.Core, *ObservedLogs) {
+	ol := &ObservedLogs{}
+	return &contextObserver{
+		LevelEnabler: enab,
+		logs:         ol,
+	}, ol
+}
+
+type contextObserver struct {
+	zapcore.LevelEnabler
+	logs    *ObservedLogs
+	context []zapcore.Field
+}
+
+var (
+	_ zapcore.Core            = (*contextObserver)(nil)
+	_ internal.LeveledEnabler = (*contextObserver)(nil)
+)
+
+func (co *contextObserver) Level() zapcore.Level {
+	return zapcore.LevelOf(co.LevelEnabler)
+}
+
+func (co *contextObserver) Check(ent zapcore.Entry, ce *zapcore.CheckedEntry) *zapcore.CheckedEntry {
+	if co.Enabled(ent.Level) {
+		return ce.AddCore(ent, co)
+	}
+	return ce
+}
+
+func (co *contextObserver) With(fields []zapcore.Field) zapcore.Core {
+	return &contextObserver{
+		LevelEnabler: co.LevelEnabler,
+		logs:         co.logs,
+		context:      append(co.context[:len(co.context):len(co.context)], fields...),
+	}
+}
+
+func (co *contextObserver) Write(ent zapcore.Entry, fields []zapcore.Field) error {
+	all := make([]zapcore.Field, 0, len(fields)+len(co.context))
+	all = append(all, co.context...)
+	all = append(all, fields...)
+	co.logs.add(LoggedEntry{ent, all})
+	return nil
+}
+
+func (co *contextObserver) Sync() error {
+	return nil
+}