Repository: knative/pkg
Pull request #1712 (2020-09-17 09:08:28)

add subresource reconciler for bindings BaseReconciler
The aim of this PR is allowing for custom code to reconcile possible sub-resources related to a `Bindable` resource.

The concrete use case is for a binding for VPA (`VPABinding`) where other than `PodSpecable` target reference we need to modify other resources (`VerticalPodAutoscaler` and one other custom resource).

Sample usage:
```
type VPASubresourcesReconciler struct {
}

func (vsr *VPASubresourcesReconciler) Reconcile(ctx context.Context, fb psbinding.Bindable) error {
   // reconcile VPA resource
}

func (vsr *VPASubresourcesReconciler) ReconcileDeletion(ctx context.Context, fb psbinding.Bindable) error {
  // delete related resources
}
```

```
vpaReconciler := vpa.NewVPASubresourcesReconciler(...t)

c := &psbinding.BaseReconciler{
   ...,
   SubResourcesReconciler: &vpaReconciler,
}
```

Once we agree on the valid approach, I would add documentation to `pkg/webhook/psbinding/README.md`


googlebot: 
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: **Please visit <https://cla.developers.google.com/> to sign.**

Once you've signed (or fixed any issues), please reply here with `@googlebot I signed it!` and we'll verify it.

----

#### What to do if you already signed the CLA

##### Individual signers

*   It's possible we don't have your GitHub username or you're using a different email address on your commit. Check [your existing CLA data](https://cla.developers.google.com/clas) and verify that your [email is set on your git commits](https://help.github.com/articles/setting-your-email-in-git/).

##### Corporate signers

*   Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to [go/cla#troubleshoot](http://go/cla#troubleshoot) ([Public version](https://opensource.google/docs/cla/#troubleshoot)).
*   The email used to register you as an authorized contributor must be the email used for the Git commit. Check [your existing CLA data](https://cla.developers.google.com/clas) and verify that your [email is set on your git commits](https://help.github.com/articles/setting-your-email-in-git/).
*   The email used to register you as an authorized contributor must also be [attached to your GitHub account](https://github.com/settings/emails).
		

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fknative%2Fpkg%2Fpull%2F1712) for more info**.

<!-- need_sender_cla -->
knative-prow-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/pkg/pull/1712#" title="Author self-approved">rccrdpccl</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **dprotaso**
You can assign the PR to them by writing `/assign @dprotaso` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fpkg).

<details open>
Needs approval from an approver in each of these files:

- **[webhook/OWNERS](https://github.com/knative/pkg/blob/master/webhook/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["dprotaso"]} -->
knative-prow-robot: Welcome @rccrdpccl! It looks like this is your first PR to knative/pkg 🎉
knative-prow-robot: Hi @rccrdpccl. Thanks for your PR.

I'm waiting for a [knative](https://github.com/orgs/knative/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/knative/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=knative%2Fpkg).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

googlebot: CLAs look good, thanks!

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fknative%2Fpkg%2Fpull%2F1712) for more info**.

<!-- ok -->

Patch:
diff --git a/.codecov.yaml b/.codecov.yaml
new file mode 100644
index 000000000..a0a3ae073
--- /dev/null
+++ b/.codecov.yaml
@@ -0,0 +1,8 @@
+ignore:
+  - "**/zz_generated*.go" # Ignore generated files.
+  - "client"
+  - "hack"
+  - "reconciler/testing"
+  - "testing"
+  - "third_party"
+  - "vendor"
diff --git a/.github/workflows/knative-donotsubmit.yaml b/.github/workflows/knative-donotsubmit.yaml
new file mode 100644
index 000000000..dcfd1ed86
--- /dev/null
+++ b/.github/workflows/knative-donotsubmit.yaml
@@ -0,0 +1,64 @@
+# Copyright 2020 The Knative Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is automagically synced here from github.com/knative-sandbox/.github
+# repo by knobots: https://github.com/mattmoor/knobots and will be overwritten.
+
+name: Do Not Submit
+
+on:
+  pull_request:
+    branches: [ 'master', 'release-*' ]
+
+jobs:
+
+  donotsubmit:
+    name: Do Not Submit
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v2
+
+      - name: Do Not Submit
+        shell: bash
+        env:
+          REVIEWDOG_GITHUB_API_TOKEN: ${{ github.token }}
+        run: |
+          set -e
+          cd "${GITHUB_WORKSPACE}" || exit 1
+
+          TEMP_PATH="$(mktemp -d)"
+          PATH="${TEMP_PATH}:$PATH"
+
+          echo '::group::🐶 Installing reviewdog ... https://github.com/reviewdog/reviewdog'
+          curl -sfL https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh | sh -s -- -b "${TEMP_PATH}" 2>&1
+          echo '::endgroup::'
+
+          echo '::group:: Running DO NOT SUBMIT with reviewdog 🐶 ...'
+          # Don't fail because of grep
+          set +o pipefail
+          find . -path './vendor' -prune \
+                 -o -path './third_party' -prune \
+                 -o -path './.github/workflows' -prune \
+                 -o -type f |
+          xargs grep -n "DO NOT SUBMIT" |
+          reviewdog -efm="%f:%l:%m" \
+                -name="DO NOT SUBMIT" \
+                -reporter="github-pr-check" \
+                -filter-mode="added" \
+                -fail-on-error="true" \
+                -level="error"
+
+          echo '::endgroup::'
diff --git a/.github/workflows/knative-go-build.yaml b/.github/workflows/knative-go-build.yaml
index 7a6894f17..98c9e411a 100644
--- a/.github/workflows/knative-go-build.yaml
+++ b/.github/workflows/knative-go-build.yaml
@@ -18,8 +18,6 @@
 name: Build
 
 on:
-  push:
-    branches: [ 'master', 'release-*' ]
   pull_request:
     branches: [ 'master', 'release-*' ]
 
@@ -44,8 +42,6 @@ jobs:
 
       - name: Check out code
         uses: actions/checkout@v2
-        with:
-          fetch-depth: 1
 
       - name: Build
         run: |
diff --git a/.github/workflows/knative-stale.yaml b/.github/workflows/knative-stale.yaml
index 3c13bf9a3..1b4e4334b 100644
--- a/.github/workflows/knative-stale.yaml
+++ b/.github/workflows/knative-stale.yaml
@@ -16,6 +16,7 @@
 # repo by knobots: https://github.com/mattmoor/knobots and will be overwritten.
 
 name: 'Close stale'
+
 on:
   schedule:
   - cron: '0 1 * * *'
diff --git a/.github/workflows/knative-style.yaml b/.github/workflows/knative-style.yaml
index 3501fdc2c..687f4b195 100644
--- a/.github/workflows/knative-style.yaml
+++ b/.github/workflows/knative-style.yaml
@@ -18,15 +18,13 @@
 name: Code Style
 
 on:
-  push:
-    branches: [ 'master', 'release-*' ]
   pull_request:
     branches: [ 'master', 'release-*' ]
 
 jobs:
 
-  lint:
-    name: Lint
+  autoformat:
+    name: Auto-format and Check
     runs-on: ubuntu-latest
 
     steps:
@@ -39,17 +37,14 @@ jobs:
 
       - name: Check out code
         uses: actions/checkout@v2
-        with:
-          fetch-depth: 1
 
+      # Run this last because it alters the workspace.
       # TODO: add prettier step
       # TODO: add goimports step
-
       - name: Go Format
         shell: bash
         run: |
           gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)
-
       - name: Verify
         shell: bash
         run: |
@@ -61,6 +56,21 @@ jobs:
           fi
           echo "${{ github.repository }} is formatted correctly."
 
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+
+    steps:
+
+      - name: Set up Go 1.14.x
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.14.x
+        id: go
+
+      - name: Check out code
+        uses: actions/checkout@v2
+
       - id: golangci_configuration
         uses: andstor/file-existence-action@v1
         with:
@@ -73,9 +83,46 @@ jobs:
           version: v1.30
           only-new-issues: true # for initial defensiveness
 
+      - name: misspell
+        shell: bash
+        if: ${{ always() }}
+        env:
+          REVIEWDOG_GITHUB_API_TOKEN: ${{ github.token }}
+        run: |
+          set -e
+          cd "${GITHUB_WORKSPACE}" || exit 1
+
+          TEMP_PATH="$(mktemp -d)"
+          PATH="${TEMP_PATH}:$PATH"
+
+          echo '::group:: Installing misspell ... https://github.com/client9/misspell'
+          go get github.com/client9/misspell/cmd/misspell
+          echo '::endgroup::'
+
+          echo '::group::🐶 Installing reviewdog ... https://github.com/reviewdog/reviewdog'
+          curl -sfL https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh | sh -s -- -b "${TEMP_PATH}" 2>&1
+          echo '::endgroup::'
+
+          echo '::group:: Running github.com/client9/misspell with reviewdog 🐶 ...'
+          # Don't fail because of misspell
+          set +o pipefail
+          find . -path './vendor' -prune \
+                 -o -path './third_party' -prune \
+                 -o -type f |
+          xargs misspell -error |
+          reviewdog -efm="%f:%l:%c: %m" \
+                -name="github.com/client9/misspell" \
+                -reporter="github-pr-check" \
+                -filter-mode="added" \
+                -fail-on-error="true" \
+                -level="error"
+
+          echo '::endgroup::'
+
       # This is mostly copied from https://github.com/get-woke/woke-action-reviewdog/blob/main/entrypoint.sh
       # since their action is not yet released under a stable version.
       - name: Language
+        if: ${{ always() && github.event_name == 'pull_request' }}
         shell: bash
         env:
           REVIEWDOG_GITHUB_API_TOKEN: ${{ github.token }}
@@ -107,5 +154,5 @@ jobs:
                 -reporter="github-pr-check" \
                 -filter-mode="added" \
                 -fail-on-error="true" \
-                -level="error" \
+                -level="error"
           echo '::endgroup::'
diff --git a/.github/workflows/knative-verify.yaml b/.github/workflows/knative-verify.yaml
index d90641152..c29a6a04d 100644
--- a/.github/workflows/knative-verify.yaml
+++ b/.github/workflows/knative-verify.yaml
@@ -18,8 +18,6 @@
 name: Verify
 
 on:
-  push:
-    branches: [ 'master', 'release-*' ]
   pull_request:
     branches: [ 'master', 'release-*' ]
 
@@ -53,7 +51,6 @@ jobs:
     - name: Check out code onto GOPATH
       uses: actions/checkout@v2
       with:
-        fetch-depth: 1
         path: ./src/knative.dev/${{ github.event.repository.name }}
 
     - name: Update Codegen
diff --git a/go.mod b/go.mod
index a3eb31560..653f96c97 100644
--- a/go.mod
+++ b/go.mod
@@ -52,7 +52,7 @@ require (
 	k8s.io/code-generator v0.18.8
 	k8s.io/gengo v0.0.0-20200205140755-e0e292d8aa12
 	k8s.io/klog v1.0.0
-	knative.dev/test-infra v0.0.0-20200916223841-d362984af929
+	knative.dev/test-infra v0.0.0-20200918021244-22269d552166
 	sigs.k8s.io/boskos v0.0.0-20200729174948-794df80db9c9
 )
 
diff --git a/go.sum b/go.sum
index 87509757d..c9b861e28 100644
--- a/go.sum
+++ b/go.sum
@@ -1814,8 +1814,8 @@ knative.dev/test-infra v0.0.0-20200505052144-5ea2f705bb55/go.mod h1:WqF1Azka+FxP
 knative.dev/test-infra v0.0.0-20200513011557-d03429a76034/go.mod h1:aMif0KXL4g19YCYwsy4Ocjjz5xgPlseYV+B95Oo4JGE=
 knative.dev/test-infra v0.0.0-20200519015156-82551620b0a9/go.mod h1:A5b2OAXTOeHT3hHhVQm3dmtbuWvIDP7qzgtqxA3/2pE=
 knative.dev/test-infra v0.0.0-20200707183444-aed09e56ddc7/go.mod h1:RjYAhXnZqeHw9+B0zsbqSPlae0lCvjekO/nw5ZMpLCs=
-knative.dev/test-infra v0.0.0-20200916223841-d362984af929 h1:jCHClMehwRQJ1BU4YPWX5eHjnJNFhYHJNsYIbJtHGZA=
-knative.dev/test-infra v0.0.0-20200916223841-d362984af929/go.mod h1:Pmg2c7Z7q7BGFUV/GOpU5BlrD3ePJft4MPqx8AYBplc=
+knative.dev/test-infra v0.0.0-20200918021244-22269d552166 h1:7Ls02/98BkpQbCItaIjkpNDKGy5oOam1SsGS6U0zl54=
+knative.dev/test-infra v0.0.0-20200918021244-22269d552166/go.mod h1:Pmg2c7Z7q7BGFUV/GOpU5BlrD3ePJft4MPqx8AYBplc=
 mvdan.cc/interfacer v0.0.0-20180901003855-c20040233aed/go.mod h1:Xkxe497xwlCKkIaQYRfC7CSLworTXY9RMqwhhCm+8Nc=
 mvdan.cc/lint v0.0.0-20170908181259-adc824a0674b/go.mod h1:2odslEg/xrtNQqCYg2/jCoyKnw3vv5biOc3JnIcYfL4=
 mvdan.cc/unparam v0.0.0-20190720180237-d51796306d8f/go.mod h1:4G1h5nDURzA3bwVMZIVpwbkw+04kSxk3rAtzlimaUJw=
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 8de97b5db..f37ef67de 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -972,7 +972,7 @@ k8s.io/utils/buffer
 k8s.io/utils/integer
 k8s.io/utils/pointer
 k8s.io/utils/trace
-# knative.dev/test-infra v0.0.0-20200916223841-d362984af929
+# knative.dev/test-infra v0.0.0-20200918021244-22269d552166
 ## explicit
 knative.dev/test-infra/scripts
 # sigs.k8s.io/boskos v0.0.0-20200729174948-794df80db9c9
diff --git a/webhook/admission.go b/webhook/admission.go
index 46d4c08c4..8a4d64d00 100644
--- a/webhook/admission.go
+++ b/webhook/admission.go
@@ -30,6 +30,23 @@ import (
 	"knative.dev/pkg/logging/logkey"
 )
 
+const (
+	// AdmissionReviewUID is the key used to represent the admission review
+	// request/response UID in logs
+	AdmissionReviewUID = "admissionreview/uid"
+
+	// AdmissionReviewAllowed is the key used to represent whether or not
+	// the admission request was permitted in logs
+	AdmissionReviewAllowed = "admissionreview/allowed"
+
+	// AdmissionReviewResult is the key used to represent extra details into
+	// why an admission request was denied in logs
+	AdmissionReviewResult = "admissionreview/result"
+
+	// AdmissionReviewPatchType is the key used to represent the type of Patch in logs
+	AdmissionReviewPatchType = "admissionreview/patchtype"
+)
+
 // AdmissionController provides the interface for different admission controllers
 type AdmissionController interface {
 	// Path returns the path that this particular admission controller serves on.
@@ -78,13 +95,13 @@ func admissionHandler(rootLogger *zap.SugaredLogger, stats StatsReporter, c Admi
 		}
 
 		logger = logger.With(
-			zap.String(logkey.Kind, review.Request.Kind.String()),
-			zap.String(logkey.Namespace, review.Request.Namespace),
-			zap.String(logkey.Name, review.Request.Name),
-			zap.String(logkey.Operation, string(review.Request.Operation)),
-			zap.String(logkey.Resource, review.Request.Resource.String()),
-			zap.String(logkey.SubResource, review.Request.SubResource),
-			zap.String(logkey.UserInfo, fmt.Sprint(review.Request.UserInfo)))
+			logkey.Kind, review.Request.Kind.String(),
+			logkey.Namespace, review.Request.Namespace,
+			logkey.Name, review.Request.Name,
+			logkey.Operation, string(review.Request.Operation),
+			logkey.Resource, review.Request.Resource.String(),
+			logkey.SubResource, review.Request.SubResource,
+			logkey.UserInfo, fmt.Sprint(review.Request.UserInfo))
 
 		ctx := logging.WithLogger(r.Context(), logger)
 
@@ -101,15 +118,19 @@ func admissionHandler(rootLogger *zap.SugaredLogger, stats StatsReporter, c Admi
 			patchType = string(*reviewResponse.PatchType)
 		}
 
-		logger.Infof("AdmissionReview for %#v: %s/%s response={ UID: %#v, Allowed: %t, Status: %#v, Patch: %s, PatchType: %s, AuditAnnotations: %#v}",
-			review.Request.Kind, review.Request.Namespace, review.Request.Name,
-			reviewResponse.UID, reviewResponse.Allowed, reviewResponse.Result, string(reviewResponse.Patch), patchType, reviewResponse.AuditAnnotations)
-
 		if !reviewResponse.Allowed || reviewResponse.PatchType != nil || response.Response == nil {
 			response.Response = reviewResponse
 		}
 		response.Response.UID = review.Request.UID
 
+		logger = logger.With(
+			AdmissionReviewUID, string(reviewResponse.UID),
+			AdmissionReviewAllowed, reviewResponse.Allowed,
+			AdmissionReviewResult, reviewResponse.Result.String())
+
+		logger.Infof("remote admission controller audit annotations=%#v", reviewResponse.AuditAnnotations)
+		logger.Debugf("AdmissionReview patch={ type: %s, body: %s }", patchType, string(reviewResponse.Patch))
+
 		if err := json.NewEncoder(w).Encode(response); err != nil {
 			http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError)
 			return
diff --git a/webhook/psbinding/README.md b/webhook/psbinding/README.md
index f7294997c..4083093b3 100644
--- a/webhook/psbinding/README.md
+++ b/webhook/psbinding/README.md
@@ -231,3 +231,44 @@ With the above defined, then in our webhook's `main.go` we invoke
 		githubbinding.NewController, githubbinding.NewWebhook,
 	)
 ```
+
+### Subresource reconciler
+
+Sometimes we might find the need for controlling not only `psbinding.Bindable`
+and `duckv1.WithPod`, but also other resources. We can achieve this by
+implementing `psbinding.SubResourcesReconcilerInterface` and injecting it in the
+`psbinding.BaseReconciler`.
+
+For example we can implement a SubResourcesReconciler to create/delete k8s
+resources:
+
+```go
+type FooBindingSubResourcesReconciler struct {
+    Client kubernetes.Interface
+}
+
+func (fr *FooBindingSubresourcesReconciler) Reconcile(ctx context.Context, fb psbinding.Bindable) error {
+    // Logic to create k8s resources here
+    return err
+}
+
+func (fr *FooBindingSubresourcesReconciler) ReconcileDeletion(ctx context.Context, fb psbinding.Bindable) error {
+    // Logic to delete k8s resources related to our Bindable
+    return err
+}
+
+```
+
+The SubResourcesReconciler can be then injected in the
+`psbinding.BaseReconciler` as follows:
+
+```go
+kclient := kubeclient.Get(ctx)
+srr := FooBindingSubResourcesReconciler{
+    Client: kclient,
+}
+c := &psbinding.BaseReconciler{
+		...
+        SubresourcesReconciler: srr
+	}
+```
diff --git a/webhook/psbinding/reconciler.go b/webhook/psbinding/reconciler.go
index ba9f35c5f..1bea867c9 100644
--- a/webhook/psbinding/reconciler.go
+++ b/webhook/psbinding/reconciler.go
@@ -45,6 +45,14 @@ import (
 	"knative.dev/pkg/tracker"
 )
 
+// SubResourcesReconcilerInterface is used to reconcile binding related
+// sub-resources. Reconcile is executed after Binding's ReconcileSubject
+// and ReconcileDeletion will be executed before Binding's ReconcileDeletion
+type SubResourcesReconcilerInterface interface {
+	Reconcile(ctx context.Context, fb Bindable) error
+	ReconcileDeletion(ctx context.Context, fb Bindable) error
+}
+
 var jsonLabelPatch = map[string]interface{}{
 	"metadata": map[string]interface{}{
 		"labels": map[string]string{duck.BindingIncludeLabel: "true"},
@@ -88,6 +96,9 @@ type BaseReconciler struct {
 
 	// Namespace Lister
 	NamespaceLister corev1listers.NamespaceLister
+
+	// Sub-resources reconciler. Used to reconcile Binding related resources
+	SubResourcesReconciler SubResourcesReconcilerInterface
 }
 
 // Check that our Reconciler implements controller.Reconciler
@@ -151,6 +162,12 @@ func (r *BaseReconciler) reconcile(ctx context.Context, fb Bindable) error {
 	if fb.GetDeletionTimestamp() != nil {
 		// Check for a DeletionTimestamp.  If present, elide the normal
 		// reconcile logic and do our finalizer handling.
+		if r.SubResourcesReconciler != nil {
+			// If a SubResourceReconciler is defined, finalize related resources
+			if err := r.SubResourcesReconciler.ReconcileDeletion(ctx, fb); err != nil {
+				return err
+			}
+		}
 		return r.ReconcileDeletion(ctx, fb)
 	}
 	// Make sure that our conditions have been initialized.
@@ -166,6 +183,12 @@ func (r *BaseReconciler) reconcile(ctx context.Context, fb Bindable) error {
 	if err := r.ReconcileSubject(ctx, fb, fb.Do); err != nil {
 		return err
 	}
+	if r.SubResourcesReconciler != nil {
+		// If a SubResourceReconciler is defined, reconcile related resources
+		if err := r.SubResourcesReconciler.Reconcile(ctx, fb); err != nil {
+			return err
+		}
+	}
 
 	// Update the observed generation once we have successfully reconciled
 	// our spec.


Test patch:
diff --git a/.github/workflows/knative-go-test.yaml b/.github/workflows/knative-go-test.yaml
index 675f15d4d..f8dcfd88d 100644
--- a/.github/workflows/knative-go-test.yaml
+++ b/.github/workflows/knative-go-test.yaml
@@ -18,15 +18,17 @@
 name: Test
 
 on:
+
   push:
-    branches: [ 'master', 'release-*' ]
+    branches: [ 'master' ]
+
   pull_request:
     branches: [ 'master', 'release-*' ]
 
 jobs:
 
   test:
-    name: Test
+    name: Unit Tests
     strategy:
       matrix:
         go-version: [1.14.x]
@@ -44,8 +46,20 @@ jobs:
 
       - name: Check out code
         uses: actions/checkout@v2
+
+      - name: Check for .codecov.yaml
+        id: codecov-enabled
+        uses: andstor/file-existence-action@v1
         with:
-          fetch-depth: 1
+          files: .codecov.yaml
+
+      - if: steps.codecov-enabled.outputs.files_exists == 'true'
+        name: Produce Go Coverage
+        run: echo '::set-env name=COVER_OPTS::-coverprofile=coverage.txt -covermode=atomic'
 
       - name: Test
-        run: go test -race ./...
+        run: go test -race $COVER_OPTS ./...
+
+      - if: steps.codecov-enabled.outputs.files_exists == 'true'
+        name: Codecov
+        uses: codecov/codecov-action@v1
diff --git a/test/junit/junit.go b/test/junit/junit.go
deleted file mode 100644
index bc69067be..000000000
--- a/test/junit/junit.go
+++ /dev/null
@@ -1,169 +0,0 @@
-/*
-Copyright 2019 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// junit.go defines types and functions specific to manipulating junit test result XML files
-
-package junit
-
-import (
-	"encoding/xml"
-	"fmt"
-	"io/ioutil"
-	"log"
-)
-
-// TestStatusEnum is a enum for test result status
-type TestStatusEnum string
-
-const (
-	// Failed means junit test failed
-	Failed TestStatusEnum = "failed"
-	// Skipped means junit test skipped
-	Skipped TestStatusEnum = "skipped"
-	// Passed means junit test passed
-	Passed TestStatusEnum = "passed"
-)
-
-// TestSuites holds a <testSuites/> list of TestSuite results
-type TestSuites struct {
-	XMLName xml.Name    `xml:"testsuites"`
-	Suites  []TestSuite `xml:"testsuite"`
-}
-
-// TestSuite holds <testSuite/> results
-type TestSuite struct {
-	XMLName    xml.Name       `xml:"testsuite"`
-	Name       string         `xml:"name,attr"`
-	Time       float64        `xml:"time,attr"` // Seconds
-	Failures   int            `xml:"failures,attr"`
-	Tests      int            `xml:"tests,attr"`
-	TestCases  []TestCase     `xml:"testcase"`
-	Properties TestProperties `xml:"properties"`
-}
-
-// TestCase holds <testcase/> results
-type TestCase struct {
-	Name       string         `xml:"name,attr"`
-	Time       float64        `xml:"time,attr"` // Seconds
-	ClassName  string         `xml:"classname,attr"`
-	Failure    *string        `xml:"failure,omitempty"`
-	Output     *string        `xml:"system-out,omitempty"`
-	Error      *string        `xml:"system-err,omitempty"`
-	Skipped    *string        `xml:"skipped,omitempty"`
-	Properties TestProperties `xml:"properties"`
-}
-
-// TestProperties is an array of test properties
-type TestProperties struct {
-	Properties []TestProperty `xml:"property"`
-}
-
-// TestProperty defines a property of the test
-type TestProperty struct {
-	Name  string `xml:"name,attr"`
-	Value string `xml:"value,attr"`
-}
-
-// GetTestStatus returns the test status as a string
-func (testCase *TestCase) GetTestStatus() TestStatusEnum {
-	testStatus := Passed
-	switch {
-	case testCase.Failure != nil:
-		testStatus = Failed
-	case testCase.Skipped != nil:
-		testStatus = Skipped
-	}
-	return testStatus
-}
-
-// AddProperty adds property to testcase
-func (testCase *TestCase) AddProperty(name, val string) {
-	property := TestProperty{Name: name, Value: val}
-	testCase.Properties.Properties = append(testCase.Properties.Properties, property)
-}
-
-// AddTestCase adds a testcase to the testsuite
-func (ts *TestSuite) AddTestCase(tc TestCase) {
-	ts.Tests++
-	if tc.GetTestStatus() == Failed {
-		ts.Failures++
-	}
-	ts.TestCases = append(ts.TestCases, tc)
-}
-
-// GetTestSuite gets TestSuite struct by name
-func (testSuites *TestSuites) GetTestSuite(suiteName string) (*TestSuite, error) {
-	for _, testSuite := range testSuites.Suites {
-		if testSuite.Name == suiteName {
-			return &testSuite, nil
-		}
-	}
-	return nil, fmt.Errorf("Test suite '%s' not found", suiteName)
-}
-
-// AddTestSuite adds TestSuite to TestSuites
-func (testSuites *TestSuites) AddTestSuite(testSuite *TestSuite) error {
-	if _, err := testSuites.GetTestSuite(testSuite.Name); err == nil {
-		return fmt.Errorf("Test suite '%s' already exists", testSuite.Name)
-	}
-	testSuites.Suites = append(testSuites.Suites, *testSuite)
-	return nil
-}
-
-// ToBytes converts TestSuites struct to bytes array
-func (testSuites *TestSuites) ToBytes(prefix, indent string) ([]byte, error) {
-	return xml.MarshalIndent(testSuites, prefix, indent)
-}
-
-// UnMarshal converts bytes array to TestSuites struct,
-// it works with both TestSuites and TestSuite structs, if
-// input is a TestSuite struct it will still return a TestSuites
-// struct, which is an empty wrapper TestSuites containing only
-// the input Suite
-func UnMarshal(buf []byte) (*TestSuites, error) {
-	var testSuites TestSuites
-	if err := xml.Unmarshal(buf, &testSuites); err == nil {
-		return &testSuites, nil
-	}
-
-	// The input might be a TestSuite if reach here, try parsing with TestSuite
-	testSuites.Suites = append([]TestSuite(nil), TestSuite{})
-	if err := xml.Unmarshal(buf, &testSuites.Suites[0]); err != nil {
-		return nil, err
-	}
-	return &testSuites, nil
-}
-
-// CreateXMLErrorMsg outputs a junit testsuite, testname and error message to the destination path
-// in XML format
-func CreateXMLErrorMsg(testSuite, testName, errMsg, dest string) {
-	suites := TestSuites{}
-	suite := TestSuite{Name: testSuite}
-	var errP *string
-	if errMsg != "" {
-		errP = &errMsg
-	}
-	suite.AddTestCase(TestCase{
-		Name:    testName,
-		Failure: errP,
-	})
-	suites.AddTestSuite(&suite)
-	contents, err := suites.ToBytes("", "")
-	if err != nil {
-		log.Fatal(err)
-	}
-	ioutil.WriteFile(dest, contents, 0644)
-}
diff --git a/test/junit/junit_test.go b/test/junit/junit_test.go
deleted file mode 100644
index 7241b8bea..000000000
--- a/test/junit/junit_test.go
+++ /dev/null
@@ -1,174 +0,0 @@
-/*
-Copyright 2019 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// junit_test.go contains unit tests for junit package
-
-package junit
-
-import (
-	"io/ioutil"
-	"os"
-	"path"
-	"testing"
-)
-
-var emptySuites = `
-<testsuites>
-</testsuites>
-`
-
-var malSuitesString = `
-<testsuites>
-<testsuites>
-`
-
-var validSuiteString = `
-<testsuite name="knative/test-infra">
-	<properties>
-		<property name="go.version" value="go1.6"/>
-	</properties>
-	<testcase name="TestBad" time="0.1">
-		<failure>something bad</failure>
-		<system-out>out: first line</system-out>
-		<system-err>err: first line</system-err>
-		<system-out>out: second line</system-out>
-	</testcase>
-	<testcase name="TestGood" time="0.1">
-	</testcase>
-	<testcase name="TestSkip" time="0.1">
-		<skipped>do not test</skipped>
-	</testcase>
-</testsuite>
-`
-
-var validSuitesString = `
-<testsuites>
-	<testsuite name="knative/test-infra">
-		<properties>
-			<property name="go.version" value="go1.6"/>
-		</properties>
-		<testcase name="TestBad" time="0.1">
-			<failure>something bad</failure>
-			<system-out>out: first line</system-out>
-			<system-err>err: first line</system-err>
-			<system-out>out: second line</system-out>
-		</testcase>
-		<testcase name="TestGood" time="0.1">
-		</testcase>
-		<testcase name="TestSkip" time="0.1">
-			<skipped>do not test</skipped>
-		</testcase>
-	</testsuite>
-</testsuites>
-`
-
-func newTestCase(name string, status TestStatusEnum) *TestCase {
-	testCase := TestCase{
-		Name: name,
-	}
-
-	var tmp string // cast const to string
-	switch {
-	case status == Failed:
-		tmp = string(Failed)
-		testCase.Failure = &tmp
-	case status == Skipped:
-		tmp = string(Skipped)
-		testCase.Skipped = &tmp
-	}
-
-	return &testCase
-}
-
-func TestUnmarshalEmptySuites(t *testing.T) {
-	if _, err := UnMarshal([]byte(emptySuites)); err != nil {
-		t.Errorf("Expected 'succeed', actual: 'failed parsing empty suites, '%s'", err)
-	}
-}
-
-func TestUnmarshalMalFormed(t *testing.T) {
-	if _, err := UnMarshal([]byte(malSuitesString)); err == nil {
-		t.Errorf("Expected: failed, actual: succeeded parsing malformed xml, '%s'", err)
-	}
-}
-
-func TestUnmarshalSuites(t *testing.T) {
-	if _, err := UnMarshal([]byte(validSuitesString)); err != nil {
-		t.Errorf("Expected: succeed, actual: failed parsing suites result, '%s'", err)
-	}
-}
-
-func TestUnmarshalSuite(t *testing.T) {
-	if _, err := UnMarshal([]byte(validSuiteString)); err != nil {
-		t.Errorf("Expected: succeed, actual: failed parsing suite result, '%s'", err)
-	}
-}
-
-func TestGetTestStatus(t *testing.T) {
-	if status := newTestCase("TestGood", Passed).GetTestStatus(); Passed != status {
-		t.Errorf("Expected '%s', actual '%s'", Passed, status)
-	}
-	if status := newTestCase("TestSkip", Skipped).GetTestStatus(); Skipped != status {
-		t.Errorf("Expected '%s', actual '%s'", Skipped, status)
-	}
-	if status := newTestCase("TestBad", Failed).GetTestStatus(); Failed != status {
-		t.Errorf("Expected '%s', actual '%s'", Failed, status)
-	}
-}
-
-func TestAddTestSuite(t *testing.T) {
-	testSuites := TestSuites{}
-	testSuite0 := TestSuite{Name: "suite_0"}
-	testSuite1 := TestSuite{Name: "suite_1"}
-
-	if err := testSuites.AddTestSuite(&testSuite0); err != nil {
-		t.Fatalf("Expected '', actual '%v'", err)
-	}
-
-	expectedErrString := "Test suite 'suite_0' already exists"
-	if err := testSuites.AddTestSuite(&testSuite0); err == nil || err.Error() != expectedErrString {
-		t.Fatalf("Expected: '%s', actual: '%v'", expectedErrString, err)
-	}
-
-	if err := testSuites.AddTestSuite(&testSuite1); err != nil {
-		t.Fatalf("Expected '', actual '%v'", err)
-	}
-
-	if len(testSuites.Suites) != 2 {
-		t.Fatalf("Expected 2, actual %d", len(testSuites.Suites))
-	}
-}
-
-func TestCreateXMLErrorMsg(t *testing.T) {
-	testDir := "test_output"
-	os.RemoveAll(testDir) // clean up in case there were stale side-effects from previous runs
-	if err := os.Mkdir(testDir, 0777); err != nil {
-		t.Fatalf("cannot create directory %q", testDir)
-	}
-	defer os.RemoveAll(testDir) // clean up
-	dest := path.Join(testDir, "TestCreateXMLErrorTestFile")
-	CreateXMLErrorMsg("dummySuite", "dummyTest", "dummyError has occurred", dest)
-	expected := `<testsuites><testsuite name="dummySuite" time="0" failures="1" tests="1"><testcase name="dummyTest" time="0" classname=""><failure>dummyError has occurred</failure><properties></properties></testcase><properties></properties></testsuite></testsuites>`
-
-	got, err := ioutil.ReadFile(dest)
-	if err != nil {
-		t.Fatalf("cannot read %q, error %v", dest, err)
-	}
-
-	if string(got) != expected {
-		t.Fatalf("expected:\n%q\n, got:\n%q", expected, got)
-	}
-}
diff --git a/test/performance/performance.go b/test/performance/performance.go
deleted file mode 100644
index 8023da2dd..000000000
--- a/test/performance/performance.go
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
-Copyright 2020 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package performance
-
-import (
-	"fmt"
-
-	"knative.dev/pkg/test/junit"
-)
-
-const (
-	// Property name of the performance test, it's used by testgrid for visualization
-	perfPropertyName = "perf_latency"
-)
-
-// CreatePerfTestCase creates a perf test case with the provided name and value
-func CreatePerfTestCase(metricValue float32, metricName, testName string) junit.TestCase {
-	tp := []junit.TestProperty{{Name: perfPropertyName, Value: fmt.Sprintf("%f", metricValue)}}
-	tc := junit.TestCase{
-		ClassName:  testName,
-		Name:       fmt.Sprintf("%s/%s", testName, metricName),
-		Properties: junit.TestProperties{Properties: tp}}
-
-	return tc
-}
diff --git a/test/testgrid/testgrid.go b/test/testgrid/testgrid.go
deleted file mode 100644
index 633ff7f6f..000000000
--- a/test/testgrid/testgrid.go
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
-Copyright 2020 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// testgrid.go provides methods to perform action on testgrid.
-
-package testgrid
-
-import (
-	"log"
-	"os"
-	"path"
-
-	"knative.dev/pkg/test/helpers"
-	"knative.dev/pkg/test/junit"
-	"knative.dev/pkg/test/prow"
-)
-
-const (
-	filePrefix = "junit_"
-	extension  = ".xml"
-)
-
-// CreateXMLOutput creates the junit xml file in the provided artifacts directory
-func CreateXMLOutput(tc []junit.TestCase, testName string) error {
-	ts := junit.TestSuites{}
-	ts.AddTestSuite(&junit.TestSuite{Name: testName, TestCases: tc})
-
-	// ensure artifactsDir exist, in case not invoked from this script
-	artifactsDir := prow.GetLocalArtifactsDir()
-	if err := helpers.CreateDir(artifactsDir); err != nil {
-		return err
-	}
-	op, err := ts.ToBytes("", "  ")
-	if err != nil {
-		return err
-	}
-
-	outputFile := path.Join(artifactsDir, filePrefix+testName+extension)
-	log.Printf("Storing output in %s", outputFile)
-	f, err := os.OpenFile(outputFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
-	if err != nil {
-		return err
-	}
-	defer f.Close()
-	if _, err := f.WriteString(string(op) + "\n"); err != nil {
-		return err
-	}
-	return nil
-}
diff --git a/test/testgrid/testgrid_test.go b/test/testgrid/testgrid_test.go
deleted file mode 100644
index f210b8775..000000000
--- a/test/testgrid/testgrid_test.go
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
-Copyright 2020 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package testgrid
-
-import (
-	"io/ioutil"
-	"os"
-	"path"
-	"testing"
-
-	"knative.dev/pkg/test/junit"
-	"knative.dev/pkg/test/prow"
-)
-
-const (
-	filename = "junit_test.xml"
-	name     = "test"
-)
-
-func checkFileText(resultFile, expected string, t *testing.T) {
-	d, err := ioutil.ReadFile(resultFile)
-	s := string(d)
-	if err != nil {
-		t.Fatalf("Failed to open test file: %v", err)
-	}
-	if s != expected {
-		t.Fatalf("Got:\n%s, Want:\n %s", s, expected)
-	}
-}
-
-func TestXMLOutput(t *testing.T) {
-	resultFile := path.Join(prow.GetLocalArtifactsDir(), filename)
-	defer os.Remove(resultFile)
-
-	// Create a test suites
-	tc := []junit.TestCase{}
-	want := `<testsuites>
-  <testsuite name="test" time="0" failures="0" tests="0">
-    <properties></properties>
-  </testsuite>
-</testsuites>
-`
-
-	// Create a test file
-	if err := CreateXMLOutput(tc, name); err != nil {
-		t.Fatalf("Error when creating xml output file: %v", err)
-	}
-	checkFileText(resultFile, want, t)
-}
diff --git a/test/upgrade/asserts_test.go b/test/upgrade/asserts_test.go
new file mode 100644
index 000000000..8286568d6
--- /dev/null
+++ b/test/upgrade/asserts_test.go
@@ -0,0 +1,44 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+)
+
+type assertions struct {
+	t *testing.T
+}
+
+func (a assertions) textContains(haystack string, needles texts) {
+	for _, needle := range needles.elms {
+		if !strings.Contains(haystack, needle) {
+			a.t.Errorf(
+				"expected \"%s\" is not in: `%s`",
+				needle, haystack,
+			)
+		}
+	}
+}
+
+func (a assertions) arraysEqual(actual []string, expected []string) {
+	if !reflect.DeepEqual(actual, expected) {
+		a.t.Errorf("arrays differ:\n  actual: %#v\nexpected: %#v", actual, expected)
+	}
+}
diff --git a/test/upgrade/execute_failures_test.go b/test/upgrade/execute_failures_test.go
new file mode 100644
index 000000000..aae11f1fe
--- /dev/null
+++ b/test/upgrade/execute_failures_test.go
@@ -0,0 +1,90 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"testing"
+)
+
+func TestSuiteExecuteWithFailures(t *testing.T) {
+	for i := 1; i <= 8; i++ {
+		for j := 1; j <= 2; j++ {
+			fp := failurePoint{
+				step:    i,
+				element: j,
+			}
+			testSuiteExecuteWithFailingStep(fp, t)
+		}
+	}
+}
+
+var allTestsFilter = func(_, _ string) (bool, error) { return true, nil }
+
+func testSuiteExecuteWithFailingStep(fp failurePoint, t *testing.T) {
+	assert := assertions{t: t}
+	testName := fmt.Sprintf("FailAt-%d-%d", fp.step, fp.element)
+	t.Run(testName, func(t *testing.T) {
+		var output string
+		suite := completeSuiteExample(fp)
+		txt := expectedTexts(suite, fp)
+		txt.append(upgradeTestRunning, upgradeTestFailure)
+		log, buf := newExampleZap()
+
+		it := []testing.InternalTest{{
+			Name: testName,
+			F: func(t *testing.T) {
+				c, _ := newConfig(t)
+				c.Log = log
+				suite.Execute(c)
+			},
+		}}
+		var ok bool
+		testOutput := captureStdOutput(func() {
+			ok = testing.RunTests(allTestsFilter, it)
+		})
+		output = buf.String()
+
+		if ok {
+			t.Fatal("didn't failed, but should")
+		}
+
+		assert.textContains(output, txt)
+		assert.textContains(testOutput, texts{
+			elms: []string{
+				fmt.Sprintf("--- FAIL: FailAt-%d-%d", fp.step, fp.element),
+			},
+		})
+	})
+}
+
+func captureStdOutput(call func()) string {
+	rescueStdout := os.Stdout
+	r, w, _ := os.Pipe()
+	os.Stdout = w
+	defer func() {
+		os.Stdout = rescueStdout
+	}()
+
+	call()
+
+	_ = w.Close()
+	out, _ := ioutil.ReadAll(r)
+	return string(out)
+}
diff --git a/test/upgrade/functions.go b/test/upgrade/functions.go
new file mode 100644
index 000000000..c6549baf7
--- /dev/null
+++ b/test/upgrade/functions.go
@@ -0,0 +1,155 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import (
+	"time"
+
+	"go.uber.org/zap"
+)
+
+// Execute the Suite of upgrade tests with a Configuration given.
+func (s *Suite) Execute(c Configuration) {
+	l := c.logger()
+	se := suiteExecution{
+		suite:         enrichSuite(s),
+		configuration: c,
+		failed:        false,
+		logger:        l,
+	}
+	l.Info("🏃 Running upgrade test suite...")
+
+	se.execute()
+
+	if !se.failed {
+		l.Info("🥳🎉 Success! Upgrade suite completed without errors.")
+	} else {
+		l.Error("💣🤬💔️ Upgrade suite have failed!")
+	}
+}
+
+// NewOperation creates a new upgrade operation or test.
+func NewOperation(name string, handler func(c Context)) Operation {
+	return &simpleOperation{name: name, handler: handler}
+}
+
+// NewBackgroundVerification is convenience function to easily setup a
+// background operation that will setup environment and then verify environment
+// status after receiving a StopEvent.
+func NewBackgroundVerification(name string, setup func(c Context), verify func(c Context)) BackgroundOperation {
+	return NewBackgroundOperation(name, setup, func(bc BackgroundContext) {
+		WaitForStopEvent(bc, WaitForStopEventConfiguration{
+			Name: name,
+			OnStop: func(event StopEvent) {
+				verify(Context{
+					T:   event.T,
+					Log: bc.Log,
+				})
+			},
+			OnWait:   DefaultOnWait,
+			WaitTime: DefaultWaitTime,
+		})
+	})
+}
+
+// NewBackgroundOperation creates a new background operation or test that can be
+// notified to stop its operation.
+func NewBackgroundOperation(name string, setup func(c Context),
+	handler func(bc BackgroundContext)) BackgroundOperation {
+	return &simpleBackgroundOperation{
+		name:    name,
+		setup:   setup,
+		handler: handler,
+	}
+}
+
+// WaitForStopEvent will wait until upgrade suite sends a stop event to it.
+// After that happen a handler is invoked to verify environment state and report
+// failures.
+func WaitForStopEvent(bc BackgroundContext, w WaitForStopEventConfiguration) {
+	log := bc.Log
+	for {
+		select {
+		case stopEvent := <-bc.Stop:
+			log.Infof("%s have received a stop event: %s", w.Name, stopEvent.Name())
+			w.OnStop(stopEvent)
+			close(stopEvent.Finished)
+			return
+		default:
+			w.OnWait(bc, w)
+		}
+		time.Sleep(w.WaitTime)
+	}
+}
+
+func (c Configuration) logger() *zap.SugaredLogger {
+	return c.Log.Sugar()
+}
+
+// Name returns a friendly human readable text.
+func (s *StopEvent) Name() string {
+	return s.name
+}
+
+func enrichSuite(s *Suite) *enrichedSuite {
+	es := &enrichedSuite{
+		installations: s.Installations,
+		tests: enrichedTests{
+			preUpgrade:    s.Tests.PreUpgrade,
+			postUpgrade:   s.Tests.PostUpgrade,
+			postDowngrade: s.Tests.PostDowngrade,
+			continual:     make([]stoppableOperation, len(s.Tests.Continual)),
+		},
+	}
+	for i, test := range s.Tests.Continual {
+		es.tests.continual[i] = stoppableOperation{
+			BackgroundOperation: test,
+			stop:                make(chan StopEvent),
+		}
+	}
+	return es
+}
+
+// Name is a human readable operation title, and it will be used in t.Run.
+func (h *simpleOperation) Name() string {
+	return h.name
+}
+
+// Handler is a function that will be called to perform an operation.
+func (h *simpleOperation) Handler() func(c Context) {
+	return h.handler
+}
+
+// Name is a human readable operation title, and it will be used in t.Run.
+func (s *simpleBackgroundOperation) Name() string {
+	return s.name
+}
+
+// Setup method may be used to set up environment before upgrade/downgrade is
+// performed.
+func (s *simpleBackgroundOperation) Setup() func(c Context) {
+	return s.setup
+}
+
+// Handler will be executed in background while upgrade/downgrade is being
+// executed. It can be used to constantly validate environment during that
+// time and/or wait for StopEvent being sent. After StopEvent is received
+// user should validate environment, clean up resources, and report found
+// issues to testing.T forwarded in StepEvent.
+func (s *simpleBackgroundOperation) Handler() func(bc BackgroundContext) {
+	return s.handler
+}
diff --git a/test/upgrade/functions_test.go b/test/upgrade/functions_test.go
new file mode 100644
index 000000000..f47cfd948
--- /dev/null
+++ b/test/upgrade/functions_test.go
@@ -0,0 +1,141 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import "testing"
+
+const (
+	upgradeTestRunning = "🏃 Running upgrade test suite..."
+	upgradeTestSuccess = "🥳🎉 Success! Upgrade suite completed without errors."
+	upgradeTestFailure = "💣🤬💔️ Upgrade suite have failed!"
+)
+
+func TestExpectedTextsForEmptySuite(t *testing.T) {
+	assert := assertions{t: t}
+	fp := notFailing
+	suite := emptySuiteExample()
+	txt := expectedTexts(suite, fp)
+	expected := []string{
+		"1) 💿 No base installation registered. Skipping.",
+		"2) ✅️️ No pre upgrade tests registered. Skipping.",
+		"3) 🔄 No continual tests registered. Skipping.",
+		"4) 📀 No upgrade operations registered. Skipping.",
+		"5) ✅️️ No post upgrade tests registered. Skipping.",
+		"6) 💿 No downgrade operations registered. Skipping.",
+		"7) ✅️️ No post downgrade tests registered. Skipping.",
+	}
+	assert.arraysEqual(txt.elms, expected)
+}
+
+func TestExpectedTextsForCompleteSuite(t *testing.T) {
+	assert := assertions{t: t}
+	fp := notFailing
+	suite := completeSuiteExample(fp)
+	txt := expectedTexts(suite, fp)
+	expected := []string{
+		"1) 💿 Installing base installations. 2 are registered.",
+		`1.1) Installing base install of "Serving latest stable release".`,
+		`1.2) Installing base install of "Eventing latest stable release".`,
+		"2) ✅️️ Testing functionality before upgrade is performed. 2 tests are registered.",
+		`2.1) Testing with "Serving pre upgrade test".`,
+		`2.2) Testing with "Eventing pre upgrade test".`,
+		"3) 🔄 Starting continual tests. 2 tests are registered.",
+		`3.1) Starting continual tests of "Serving continual test".`,
+		`3.2) Starting continual tests of "Eventing continual test".`,
+		"4) 📀 Upgrading with 2 registered operations.",
+		`4.1) Upgrading with "Serving HEAD".`,
+		`4.2) Upgrading with "Eventing HEAD".`,
+		"5) ✅️️ Testing functionality after upgrade is performed. 2 tests are registered.",
+		`5.1) Testing with "Serving post upgrade test".`,
+		`5.2) Testing with "Eventing post upgrade test".`,
+		"6) 💿 Downgrading with 2 registered operations.",
+		`6.1) Downgrading with "Serving latest stable release".`,
+		`6.2) Downgrading with "Eventing latest stable release".`,
+		"7) ✅️️ Testing functionality after downgrade is performed. 2 tests are registered.",
+		`7.1) Testing with "Serving post downgrade test".`,
+		`7.2) Testing with "Eventing post downgrade test".`,
+		"8) ✋ Verifying 2 running continual tests.",
+		`8.1) Verifying "Serving continual test".`,
+		`8.2) Verifying "Eventing continual test".`,
+	}
+	assert.arraysEqual(txt.elms, expected)
+}
+
+func TestExpectedTextsForFailingCompleteSuite(t *testing.T) {
+	assert := assertions{t: t}
+	fp := failurePoint{
+		step:    2,
+		element: 1,
+	}
+	suite := completeSuiteExample(fp)
+	txt := expectedTexts(suite, fp)
+	expected := []string{
+		"1) 💿 Installing base installations. 2 are registered.",
+		`1.1) Installing base install of "Serving latest stable release".`,
+		`1.2) Installing base install of "Eventing latest stable release".`,
+		"2) ✅️️ Testing functionality before upgrade is performed. 2 tests are registered.",
+		`2.1) Testing with "FailingOfServing pre upgrade test".`,
+	}
+	assert.arraysEqual(txt.elms, expected)
+}
+
+func TestSuiteExecuteEmpty(t *testing.T) {
+	assert := assertions{t: t}
+	c, buf := newConfig(t)
+	fp := notFailing
+	suite := emptySuiteExample()
+	suite.Execute(c)
+	output := buf.String()
+	if c.T.Failed() {
+		return
+	}
+
+	txt := expectedTexts(suite, fp)
+	txt.append(upgradeTestRunning, upgradeTestSuccess)
+
+	assert.textContains(output, txt)
+}
+
+func TestSuiteExecuteWithComplete(t *testing.T) {
+	assert := assertions{t: t}
+	c, buf := newConfig(t)
+	fp := notFailing
+	suite := completeSuiteExample(fp)
+	suite.Execute(c)
+	output := buf.String()
+	if c.T.Failed() {
+		return
+	}
+	txt := expectedTexts(suite, fp)
+	txt.append(upgradeTestRunning, upgradeTestSuccess)
+	txt.append(
+		"Installing Serving stable 0.17.1",
+		"Installing Eventing stable 0.17.2",
+		"Running Serving continual test",
+		"Stopping and verify of Eventing continual test",
+		"Installing Serving HEAD at e3c4563",
+		"Installing Eventing HEAD at 12f67cc",
+		"Installing Serving stable 0.17.1",
+		"Installing Eventing stable 0.17.2",
+		"Serving have received a stop event",
+		"Eventing continual test have received a stop event",
+		"Serving - probing functionality...",
+		"Eventing continual test - probing functionality...",
+	)
+
+	assert.textContains(output, txt)
+}
diff --git a/test/upgrade/private_types.go b/test/upgrade/private_types.go
new file mode 100644
index 000000000..e34f09005
--- /dev/null
+++ b/test/upgrade/private_types.go
@@ -0,0 +1,63 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import "go.uber.org/zap"
+
+type suiteExecution struct {
+	suite         *enrichedSuite
+	configuration Configuration
+	failed        bool
+	logger        *zap.SugaredLogger
+}
+
+type enrichedSuite struct {
+	installations Installations
+	tests         enrichedTests
+}
+
+type enrichedTests struct {
+	preUpgrade    []Operation
+	postUpgrade   []Operation
+	postDowngrade []Operation
+	continual     []stoppableOperation
+}
+
+type stoppableOperation struct {
+	BackgroundOperation
+	stop chan StopEvent
+}
+
+type operationGroup struct {
+	num                   int
+	operations            []Operation
+	groupName             string
+	groupTemplate         string
+	elementTemplate       string
+	skippingGroupTemplate string
+}
+
+type simpleOperation struct {
+	name    string
+	handler func(c Context)
+}
+
+type simpleBackgroundOperation struct {
+	name    string
+	setup   func(c Context)
+	handler func(bc BackgroundContext)
+}
diff --git a/test/upgrade/steps.go b/test/upgrade/steps.go
new file mode 100644
index 000000000..c50bdcfb6
--- /dev/null
+++ b/test/upgrade/steps.go
@@ -0,0 +1,155 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import (
+	"testing"
+)
+
+const skippingOperationTemplate = `Skipping "%s" as previous operation have failed`
+
+func (se *suiteExecution) installingBase(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.installations.Base,
+		groupName:             "InstallingBase",
+		elementTemplate:       `%d.%d) Installing base install of "%s".`,
+		skippingGroupTemplate: "%d) 💿 No base installation registered. Skipping.",
+		groupTemplate:         "%d) 💿 Installing base installations. %d are registered.",
+	})
+}
+
+func (se *suiteExecution) preUpgradeTests(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.tests.preUpgrade,
+		groupName:             "PreUpgradeTests",
+		elementTemplate:       `%d.%d) Testing with "%s".`,
+		skippingGroupTemplate: "%d) ✅️️ No pre upgrade tests registered. Skipping.",
+		groupTemplate: "%d) ✅️️ Testing functionality before upgrade is performed." +
+			" %d tests are registered.",
+	})
+}
+
+func (se *suiteExecution) startContinualTests(num int) {
+	l := se.logger
+	operations := se.suite.tests.continual
+	groupTemplate := "%d) 🔄 Starting continual tests. " +
+		"%d tests are registered."
+	elementTemplate := `%d.%d) Starting continual tests of "%s".`
+	numOps := len(operations)
+	se.configuration.T.Run("ContinualTests", func(t *testing.T) {
+		if numOps > 0 {
+			l.Infof(groupTemplate, num, numOps)
+			for i := range operations {
+				operation := operations[i]
+				l.Infof(elementTemplate, num, i+1, operation.Name())
+				if se.failed {
+					l.Debugf(skippingOperationTemplate, operation.Name())
+					return
+				}
+				setup := operation.Setup()
+				t.Run("Setup"+operation.Name(), func(t *testing.T) {
+					setup(Context{T: t, Log: l})
+				})
+				handler := operation.Handler()
+				go func() {
+					bc := BackgroundContext{Log: l, Stop: operation.stop}
+					handler(bc)
+				}()
+
+				se.failed = se.failed || t.Failed()
+				if se.failed {
+					return
+				}
+			}
+
+		} else {
+			l.Infof("%d) 🔄 No continual tests registered. Skipping.", num)
+		}
+	})
+}
+
+func (se *suiteExecution) verifyContinualTests(num int) {
+	l := se.logger
+	testsCount := len(se.suite.tests.continual)
+	if testsCount > 0 {
+		se.configuration.T.Run("VerifyContinualTests", func(t *testing.T) {
+			l.Infof("%d) ✋ Verifying %d running continual tests.", num, testsCount)
+			for i, operation := range se.suite.tests.continual {
+				t.Run(operation.Name(), func(t *testing.T) {
+					l.Infof(`%d.%d) Verifying "%s".`, num, i+1, operation.Name())
+					finished := make(chan struct{})
+					operation.stop <- StopEvent{
+						T:        t,
+						Finished: finished,
+						name:     "Stop of " + operation.Name(),
+					}
+					<-finished
+					se.failed = se.failed || t.Failed()
+					l.Debugf(`Finished "%s"`, operation.Name())
+				})
+			}
+		})
+	}
+}
+
+func (se *suiteExecution) upgradeWith(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.installations.UpgradeWith,
+		groupName:             "UpgradeWith",
+		elementTemplate:       `%d.%d) Upgrading with "%s".`,
+		skippingGroupTemplate: "%d) 📀 No upgrade operations registered. Skipping.",
+		groupTemplate:         "%d) 📀 Upgrading with %d registered operations.",
+	})
+}
+
+func (se *suiteExecution) postUpgradeTests(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.tests.postUpgrade,
+		groupName:             "PostUpgradeTests",
+		elementTemplate:       `%d.%d) Testing with "%s".`,
+		skippingGroupTemplate: "%d) ✅️️ No post upgrade tests registered. Skipping.",
+		groupTemplate: "%d) ✅️️ Testing functionality after upgrade is performed." +
+			" %d tests are registered.",
+	})
+}
+
+func (se *suiteExecution) downgradeWith(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.installations.DowngradeWith,
+		groupName:             "DowngradeWith",
+		elementTemplate:       `%d.%d) Downgrading with "%s".`,
+		skippingGroupTemplate: "%d) 💿 No downgrade operations registered. Skipping.",
+		groupTemplate:         "%d) 💿 Downgrading with %d registered operations.",
+	})
+}
+
+func (se *suiteExecution) postDowngradeTests(num int) {
+	se.processOperationGroup(operationGroup{
+		num:                   num,
+		operations:            se.suite.tests.postDowngrade,
+		groupName:             "PostDowngradeTests",
+		elementTemplate:       `%d.%d) Testing with "%s".`,
+		skippingGroupTemplate: "%d) ✅️️ No post downgrade tests registered. Skipping.",
+		groupTemplate: "%d) ✅️️ Testing functionality after downgrade is performed." +
+			" %d tests are registered.",
+	})
+}
diff --git a/test/upgrade/suite_examples_test.go b/test/upgrade/suite_examples_test.go
new file mode 100644
index 000000000..390b9e313
--- /dev/null
+++ b/test/upgrade/suite_examples_test.go
@@ -0,0 +1,162 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import (
+	"time"
+
+	"knative.dev/pkg/test/upgrade"
+)
+
+const (
+	shortWait = 50 * time.Microsecond
+	longWait  = 750 * time.Microsecond
+)
+
+func init() {
+	upgrade.DefaultOnWait = func(bc upgrade.BackgroundContext, self upgrade.WaitForStopEventConfiguration) {
+		bc.Log.Debugf("%s - probing functionality...", self.Name)
+	}
+	upgrade.DefaultWaitTime = shortWait
+}
+
+var (
+	notFailing        = failurePoint{step: -1, element: -1}
+	messageFormatters = messageFormatterRepository{
+		baseInstall: createMessages(formats{
+			starting: "%d) 💿 Installing base installations. %d are registered.",
+			element:  `%d.%d) Installing base install of "%s".`,
+			skipped:  "%d) 💿 No base installation registered. Skipping.",
+		}),
+		preUpgrade: createMessages(formats{
+			starting: "%d) ✅️️ Testing functionality before upgrade is performed. %d tests are registered.",
+			element:  `%d.%d) Testing with "%s".`,
+			skipped:  "%d) ✅️️ No pre upgrade tests registered. Skipping.",
+		}),
+		startContinual: createMessages(formats{
+			starting: "%d) 🔄 Starting continual tests. %d tests are registered.",
+			element:  `%d.%d) Starting continual tests of "%s".`,
+			skipped:  "%d) 🔄 No continual tests registered. Skipping.",
+		}),
+		upgrade: createMessages(formats{
+			starting: "%d) 📀 Upgrading with %d registered operations.",
+			element:  `%d.%d) Upgrading with "%s".`,
+			skipped:  "%d) 📀 No upgrade operations registered. Skipping.",
+		}),
+		postUpgrade: createMessages(formats{
+			starting: "%d) ✅️️ Testing functionality after upgrade is performed. %d tests are registered.",
+			element:  `%d.%d) Testing with "%s".`,
+			skipped:  "%d) ✅️️ No post upgrade tests registered. Skipping.",
+		}),
+		downgrade: createMessages(formats{
+			starting: "%d) 💿 Downgrading with %d registered operations.",
+			element:  `%d.%d) Downgrading with "%s".`,
+			skipped:  "%d) 💿 No downgrade operations registered. Skipping.",
+		}),
+		postDowngrade: createMessages(formats{
+			starting: "%d) ✅️️ Testing functionality after downgrade is performed. %d tests are registered.",
+			element:  `%d.%d) Testing with "%s".`,
+			skipped:  "%d) ✅️️ No post downgrade tests registered. Skipping.",
+		}),
+		verifyContinual: createMessages(formats{
+			starting: "%d) ✋ Verifying %d running continual tests.",
+			element:  `%d.%d) Verifying "%s".`,
+			skipped:  "",
+		}),
+	}
+	serving = component{
+		installs: installs{
+			stable: upgrade.NewOperation("Serving latest stable release", func(c upgrade.Context) {
+				c.Log.Info("Installing Serving stable 0.17.1")
+				time.Sleep(longWait)
+			}),
+			head: upgrade.NewOperation("Serving HEAD", func(c upgrade.Context) {
+				c.Log.Info("Installing Serving HEAD at e3c4563")
+				time.Sleep(longWait)
+			}),
+		},
+		tests: tests{
+			preUpgrade: upgrade.NewOperation("Serving pre upgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Serving pre upgrade test")
+				time.Sleep(shortWait)
+			}),
+			postUpgrade: upgrade.NewOperation("Serving post upgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Serving post upgrade test")
+				time.Sleep(shortWait)
+			}),
+			postDowngrade: upgrade.NewOperation("Serving post downgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Serving post downgrade test")
+				time.Sleep(shortWait)
+			}),
+			continual: upgrade.NewBackgroundOperation("Serving continual test",
+				func(c upgrade.Context) {
+					c.Log.Info("Setup of Serving continual test")
+					time.Sleep(shortWait)
+				},
+				func(bc upgrade.BackgroundContext) {
+					bc.Log.Info("Running Serving continual test")
+					upgrade.WaitForStopEvent(bc, upgrade.WaitForStopEventConfiguration{
+						Name: "Serving",
+						OnStop: func(event upgrade.StopEvent) {
+							bc.Log.Info("Stopping and verify of Serving continual test")
+							time.Sleep(shortWait)
+						},
+						OnWait: func(bc upgrade.BackgroundContext, self upgrade.WaitForStopEventConfiguration) {
+							bc.Log.Debugf("%s - probing functionality...", self.Name)
+						},
+						WaitTime: shortWait,
+					})
+				}),
+		},
+	}
+	eventing = component{
+		installs: installs{
+			stable: upgrade.NewOperation("Eventing latest stable release", func(c upgrade.Context) {
+				c.Log.Info("Installing Eventing stable 0.17.2")
+				time.Sleep(longWait)
+			}),
+			head: upgrade.NewOperation("Eventing HEAD", func(c upgrade.Context) {
+				c.Log.Info("Installing Eventing HEAD at 12f67cc")
+				time.Sleep(longWait)
+			}),
+		},
+		tests: tests{
+			preUpgrade: upgrade.NewOperation("Eventing pre upgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Eventing pre upgrade test")
+				time.Sleep(shortWait)
+			}),
+			postUpgrade: upgrade.NewOperation("Eventing post upgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Eventing post upgrade test")
+				time.Sleep(shortWait)
+			}),
+			postDowngrade: upgrade.NewOperation("Eventing post downgrade test", func(c upgrade.Context) {
+				c.Log.Info("Running Eventing post downgrade test")
+				time.Sleep(shortWait)
+			}),
+			continual: upgrade.NewBackgroundVerification("Eventing continual test",
+				func(c upgrade.Context) {
+					c.Log.Info("Setup of Eventing continual test")
+					time.Sleep(shortWait)
+				},
+				func(c upgrade.Context) {
+					c.Log.Info("Stopping and verify of Eventing continual test")
+					time.Sleep(shortWait)
+				},
+			),
+		},
+	}
+)
diff --git a/test/upgrade/suite_execution.go b/test/upgrade/suite_execution.go
new file mode 100644
index 000000000..70c5ea095
--- /dev/null
+++ b/test/upgrade/suite_execution.go
@@ -0,0 +1,85 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import (
+	"testing"
+)
+
+func (se *suiteExecution) processOperationGroup(op operationGroup) {
+	l := se.logger
+	se.configuration.T.Run(op.groupName, func(t *testing.T) {
+		if len(op.operations) > 0 {
+			l.Infof(op.groupTemplate, op.num, len(op.operations))
+			for i, operation := range op.operations {
+				l.Infof(op.elementTemplate, op.num, i+1, operation.Name())
+				if se.failed {
+					l.Debugf(skippingOperationTemplate, operation.Name())
+					return
+				}
+				handler := operation.Handler()
+				t.Run(operation.Name(), func(t *testing.T) {
+					handler(Context{T: t, Log: l})
+				})
+				se.failed = se.failed || t.Failed()
+				if se.failed {
+					return
+				}
+			}
+		} else {
+			l.Infof(op.skippingGroupTemplate, op.num)
+		}
+	})
+}
+
+func (se *suiteExecution) execute() {
+	idx := 1
+	operations := []func(num int){
+		se.installingBase,
+		se.preUpgradeTests,
+	}
+	for _, operation := range operations {
+		operation(idx)
+		idx++
+		if se.failed {
+			return
+		}
+	}
+
+	se.startContinualTests(idx)
+	idx++
+	if se.failed {
+		return
+	}
+	defer func() {
+		se.verifyContinualTests(idx)
+	}()
+
+	operations = []func(num int){
+		se.upgradeWith,
+		se.postUpgradeTests,
+		se.downgradeWith,
+		se.postDowngradeTests,
+	}
+	for _, operation := range operations {
+		operation(idx)
+		idx++
+		if se.failed {
+			return
+		}
+	}
+}
diff --git a/test/upgrade/testing_operations_test.go b/test/upgrade/testing_operations_test.go
new file mode 100644
index 000000000..79483d7a4
--- /dev/null
+++ b/test/upgrade/testing_operations_test.go
@@ -0,0 +1,293 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"sync"
+	"testing"
+
+	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
+	"go.uber.org/zap/zaptest"
+	"knative.dev/pkg/test/upgrade"
+)
+
+func newConfig(t *testing.T) (upgrade.Configuration, fmt.Stringer) {
+	log, buf := newExampleZap()
+	c := upgrade.Configuration{T: t, Log: log}
+	return c, buf
+}
+
+func newExampleZap() (*zap.Logger, fmt.Stringer) {
+	ec := zap.NewDevelopmentEncoderConfig()
+	ec.TimeKey = ""
+	encoder := zapcore.NewConsoleEncoder(ec)
+	buf := &buffer{
+		Buffer: bytes.Buffer{},
+		Mutex:  sync.Mutex{},
+		Syncer: zaptest.Syncer{},
+	}
+	ws := zapcore.NewMultiWriteSyncer(buf, os.Stdout)
+	core := zapcore.NewCore(encoder, ws, zap.DebugLevel)
+	return zap.New(core).WithOptions(), buf
+}
+
+func createSteps(s upgrade.Suite) []*step {
+	continualTestsGeneralized := generalizeOpsFromBg(s.Tests.Continual)
+	return []*step{{
+		messages: messageFormatters.baseInstall,
+		ops:      generalizeOps(s.Installations.Base),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Installations.Base = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.preUpgrade,
+		ops:      generalizeOps(s.Tests.PreUpgrade),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Tests.PreUpgrade = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.startContinual,
+		ops:      continualTestsGeneralized,
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Tests.Continual = ops.asBackgroundOperation()
+		},
+	}, {
+		messages: messageFormatters.upgrade,
+		ops:      generalizeOps(s.Installations.UpgradeWith),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Installations.UpgradeWith = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.postUpgrade,
+		ops:      generalizeOps(s.Tests.PostUpgrade),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Tests.PostUpgrade = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.downgrade,
+		ops:      generalizeOps(s.Installations.DowngradeWith),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Installations.DowngradeWith = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.postDowngrade,
+		ops:      generalizeOps(s.Tests.PostDowngrade),
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Tests.PostDowngrade = ops.asOperations()
+		},
+	}, {
+		messages: messageFormatters.verifyContinual,
+		ops:      continualTestsGeneralized,
+		updateSuite: func(ops operations, s *upgrade.Suite) {
+			s.Tests.Continual = ops.asBackgroundOperation()
+		},
+	}}
+}
+
+func expectedTexts(s upgrade.Suite, fp failurePoint) texts {
+	steps := createSteps(s)
+	tt := texts{elms: nil}
+	for i, st := range steps {
+		stepIdx := i + 1
+		if st.ops.length() == 0 {
+			tt.append(st.skipped(stepIdx))
+		} else {
+			tt.append(st.starting(stepIdx, st.ops.length()))
+			for j, op := range st.ops.ops {
+				elemIdx := j + 1
+				tt.append(st.element(stepIdx, elemIdx, op.Name()))
+				if fp.step == stepIdx && fp.element == elemIdx {
+					return tt
+				}
+			}
+		}
+	}
+	return tt
+}
+
+func generalizeOps(ops []upgrade.Operation) operations {
+	gen := make([]*operation, len(ops))
+	for idx, op := range ops {
+		gen[idx] = &operation{op: op}
+	}
+	return operations{ops: gen}
+}
+
+func generalizeOpsFromBg(ops []upgrade.BackgroundOperation) operations {
+	gen := make([]*operation, len(ops))
+	for idx, op := range ops {
+		gen[idx] = &operation{bg: op}
+	}
+	return operations{ops: gen}
+}
+
+func createMessages(mf formats) messages {
+	return messages{
+		skipped: func(args ...interface{}) string {
+			empty := ""
+			if mf.skipped == empty {
+				return empty
+			}
+			return fmt.Sprintf(mf.skipped, args...)
+		},
+		starting: func(args ...interface{}) string {
+			return fmt.Sprintf(mf.starting, args...)
+		},
+		element: func(args ...interface{}) string {
+			return fmt.Sprintf(mf.element, args...)
+		},
+	}
+}
+
+func (tt *texts) append(messages ...string) {
+	for _, msg := range messages {
+		if msg == "" {
+			continue
+		}
+		tt.elms = append(tt.elms, msg)
+	}
+}
+
+func completeSuiteExample(fp failurePoint) upgrade.Suite {
+	suite := upgrade.Suite{
+		Tests: upgrade.Tests{
+			PreUpgrade: []upgrade.Operation{
+				serving.tests.preUpgrade, eventing.tests.preUpgrade,
+			},
+			PostUpgrade: []upgrade.Operation{
+				serving.tests.postUpgrade, eventing.tests.postUpgrade,
+			},
+			PostDowngrade: []upgrade.Operation{
+				serving.tests.postDowngrade, eventing.tests.postDowngrade,
+			},
+			Continual: []upgrade.BackgroundOperation{
+				serving.tests.continual, eventing.tests.continual,
+			},
+		},
+		Installations: upgrade.Installations{
+			Base: []upgrade.Operation{
+				serving.installs.stable, eventing.installs.stable,
+			},
+			UpgradeWith: []upgrade.Operation{
+				serving.installs.head, eventing.installs.head,
+			},
+			DowngradeWith: []upgrade.Operation{
+				serving.installs.stable, eventing.installs.stable,
+			},
+		},
+	}
+	return enrichSuiteWithFailures(suite, fp)
+}
+
+func emptySuiteExample() upgrade.Suite {
+	return upgrade.Suite{
+		Tests:         upgrade.Tests{},
+		Installations: upgrade.Installations{},
+	}
+}
+
+func enrichSuiteWithFailures(suite upgrade.Suite, fp failurePoint) upgrade.Suite {
+	steps := createSteps(suite)
+	for i, st := range steps {
+		for j, op := range st.ops.ops {
+			if fp.step == i+1 && fp.element == j+1 {
+				op.fail(fp.step == 3)
+			}
+		}
+	}
+	return recreateSuite(steps)
+}
+
+func recreateSuite(steps []*step) upgrade.Suite {
+	suite := &upgrade.Suite{
+		Tests:         upgrade.Tests{},
+		Installations: upgrade.Installations{},
+	}
+	for _, st := range steps {
+		st.updateSuite(st.ops, suite)
+	}
+	return *suite
+}
+
+func (o operation) Name() string {
+	if o.op != nil {
+		return o.op.Name()
+	}
+	return o.bg.Name()
+}
+
+func (o *operation) fail(setupFail bool) {
+	failureTestingMessage := "This error is expected to be seen. Upgrade suite should fail."
+	testName := fmt.Sprintf("FailingOf%s", o.Name())
+	if o.op != nil {
+		prev := o.op
+		o.op = upgrade.NewOperation(testName, func(c upgrade.Context) {
+			handler := prev.Handler()
+			handler(c)
+			c.T.Error(failureTestingMessage)
+			c.Log.Error(failureTestingMessage)
+		})
+	} else {
+		prev := o.bg
+		o.bg = upgrade.NewBackgroundOperation(testName, func(c upgrade.Context) {
+			handler := prev.Setup()
+			handler(c)
+			if setupFail {
+				c.T.Error(failureTestingMessage)
+				c.Log.Error(failureTestingMessage)
+			}
+		}, func(bc upgrade.BackgroundContext) {
+			upgrade.WaitForStopEvent(bc, upgrade.WaitForStopEventConfiguration{
+				Name: testName,
+				OnStop: func(event upgrade.StopEvent) {
+					if !setupFail {
+						event.T.Error(failureTestingMessage)
+						bc.Log.Error(failureTestingMessage)
+					}
+				},
+				OnWait: func(bc upgrade.BackgroundContext, self upgrade.WaitForStopEventConfiguration) {
+					bc.Log.Debugf("%s - probing functionality...", self.Name)
+				},
+				WaitTime: shortWait,
+			})
+		})
+	}
+}
+
+func (o operations) length() int {
+	return len(o.ops)
+}
+
+func (o operations) asOperations() []upgrade.Operation {
+	ops := make([]upgrade.Operation, o.length())
+	for i, op := range o.ops {
+		ops[i] = op.op
+	}
+	return ops
+}
+
+func (o operations) asBackgroundOperation() []upgrade.BackgroundOperation {
+	ops := make([]upgrade.BackgroundOperation, o.length())
+	for i, op := range o.ops {
+		ops[i] = op.bg
+	}
+	return ops
+}
diff --git a/test/upgrade/testing_types_test.go b/test/upgrade/testing_types_test.go
new file mode 100644
index 000000000..0dabca141
--- /dev/null
+++ b/test/upgrade/testing_types_test.go
@@ -0,0 +1,85 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import "knative.dev/pkg/test/upgrade"
+
+type failurePoint struct {
+	step    int
+	element int
+}
+
+type texts struct {
+	elms []string
+}
+
+type messageFormatter func(args ...interface{}) string
+
+type step struct {
+	messages
+	ops         operations
+	updateSuite func(ops operations, s *upgrade.Suite)
+}
+
+type operations struct {
+	ops []*operation
+}
+
+type operation struct {
+	op upgrade.Operation
+	bg upgrade.BackgroundOperation
+}
+
+type formats struct {
+	skipped  string
+	starting string
+	element  string
+}
+
+type messages struct {
+	starting messageFormatter
+	element  messageFormatter
+	skipped  messageFormatter
+}
+
+type messageFormatterRepository struct {
+	baseInstall     messages
+	preUpgrade      messages
+	startContinual  messages
+	upgrade         messages
+	postUpgrade     messages
+	downgrade       messages
+	postDowngrade   messages
+	verifyContinual messages
+}
+
+type component struct {
+	installs
+	tests
+}
+
+type installs struct {
+	stable upgrade.Operation
+	head   upgrade.Operation
+}
+
+type tests struct {
+	preUpgrade    upgrade.Operation
+	postUpgrade   upgrade.Operation
+	continual     upgrade.BackgroundOperation
+	postDowngrade upgrade.Operation
+}
diff --git a/test/upgrade/types.go b/test/upgrade/types.go
new file mode 100644
index 000000000..afcccec33
--- /dev/null
+++ b/test/upgrade/types.go
@@ -0,0 +1,123 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import (
+	"testing"
+	"time"
+
+	"go.uber.org/zap"
+)
+
+// Suite represents a upgrade tests suite that can be executed and will perform
+// execution in predictable manner.
+type Suite struct {
+	Tests         Tests
+	Installations Installations
+}
+
+// Tests holds a list of operations for various part of upgrade suite.
+type Tests struct {
+	PreUpgrade    []Operation
+	PostUpgrade   []Operation
+	PostDowngrade []Operation
+	Continual     []BackgroundOperation
+}
+
+// Installations holds a list of operations that will install Knative components
+// in different versions.
+type Installations struct {
+	Base          []Operation
+	UpgradeWith   []Operation
+	DowngradeWith []Operation
+}
+
+// Operation represents a upgrade test operation like test or installation that
+// can be provided by specific component or reused in aggregating components.
+type Operation interface {
+	// Name is a human readable operation title, and it will be used in t.Run.
+	Name() string
+	// Handler is a function that will be called to perform an operation.
+	Handler() func(c Context)
+}
+
+// BackgroundOperation represents a upgrade test operation that will be
+// performed in background while other operations is running. To achieve that
+// a passed BackgroundContext should be used to synchronize it's operations with
+// Ready and Stop channels.
+type BackgroundOperation interface {
+	// Name is a human readable operation title, and it will be used in t.Run.
+	Name() string
+	// Setup method may be used to set up environment before upgrade/downgrade is
+	// performed.
+	Setup() func(c Context)
+	// Handler will be executed in background while upgrade/downgrade is being
+	// executed. It can be used to constantly validate environment during that
+	// time and/or wait for StopEvent being sent. After StopEvent is received
+	// user should validate environment, clean up resources, and report found
+	// issues to testing.T forwarded in StepEvent.
+	Handler() func(bc BackgroundContext)
+}
+
+// Context is an object that is passed to every operation. It contains testing.T
+// for error reporting and zap.SugaredLogger for unbuffered logging.
+type Context struct {
+	T   *testing.T
+	Log *zap.SugaredLogger
+}
+
+// BackgroundContext is a upgrade test execution context that will be passed
+// down to each handler of BackgroundOperation. It contains a StopEvent channel
+// which end user should use to obtain a testing.T for error reporting. Until
+// StopEvent is sent user may use zap.SugaredLogger to log state of execution if
+// necessary.
+type BackgroundContext struct {
+	Log  *zap.SugaredLogger
+	Stop <-chan StopEvent
+}
+
+// StopEvent represents an event that is to be received by background operation
+// to indicate that is should stop it's operations and validate results using
+// passed T. User should use Finished channel to signalize upgrade suite that
+// all stop & verify operations are finished and it is safe to end tests.
+type StopEvent struct {
+	T        *testing.T
+	Finished chan<- struct{}
+	name     string
+}
+
+// WaitForStopEventConfiguration holds a values to be used be WaitForStopEvent
+// function. OnStop will be called when StopEvent is sent. OnWait will be
+// invoked in a loop while waiting, and each wait act is driven by WaitTime
+// amount.
+type WaitForStopEventConfiguration struct {
+	Name     string
+	OnStop   func(event StopEvent)
+	OnWait   func(bc BackgroundContext, self WaitForStopEventConfiguration)
+	WaitTime time.Duration
+}
+
+// Configuration holds required and optional configuration to run upgrade tests.
+type Configuration struct {
+	T   *testing.T
+	Log *zap.Logger
+}
+
+// SuiteExecutor is to execute upgrade test suite.
+type SuiteExecutor interface {
+	Execute(c Configuration)
+}
diff --git a/test/upgrade/vars.go b/test/upgrade/vars.go
new file mode 100644
index 000000000..8a8ba93e1
--- /dev/null
+++ b/test/upgrade/vars.go
@@ -0,0 +1,32 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade
+
+import "time"
+
+var (
+	// DefaultWaitTime holds a default value for WaitForStopEventConfiguration
+	// when used within a NewBackgroundVerification function.
+	DefaultWaitTime = 20 * time.Millisecond
+
+	// DefaultOnWait is a implementation that will be called by default for each
+	// wait performed by WaitForStopEvent when used within
+	// NewBackgroundVerification function.
+	DefaultOnWait = func(bc BackgroundContext, self WaitForStopEventConfiguration) {
+		// do nothing by default
+	}
+)
diff --git a/test/upgrade/zaptest_buffer_test.go b/test/upgrade/zaptest_buffer_test.go
new file mode 100644
index 000000000..05b373743
--- /dev/null
+++ b/test/upgrade/zaptest_buffer_test.go
@@ -0,0 +1,49 @@
+/*
+ * Copyright 2020 The Knative Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package upgrade_test
+
+import (
+	"bytes"
+	"sync"
+
+	"go.uber.org/zap/zaptest"
+)
+
+// To avoid race condition on zaptest.Buffer, see: https://stackoverflow.com/a/36226525/844449
+type buffer struct {
+	bytes.Buffer
+	sync.Mutex
+	zaptest.Syncer
+}
+
+func (b *buffer) Read(p []byte) (n int, err error) {
+	b.Mutex.Lock()
+	defer b.Mutex.Unlock()
+	return b.Buffer.Read(p)
+}
+
+func (b *buffer) Write(p []byte) (n int, err error) {
+	b.Mutex.Lock()
+	defer b.Mutex.Unlock()
+	return b.Buffer.Write(p)
+}
+
+func (b *buffer) String() string {
+	b.Mutex.Lock()
+	defer b.Mutex.Unlock()
+	return b.Buffer.String()
+}
diff --git a/testutils/junithelper/README.md b/testutils/junithelper/README.md
deleted file mode 100644
index 4f53a84b5..000000000
--- a/testutils/junithelper/README.md
+++ /dev/null
@@ -1,22 +0,0 @@
-## junithelper
-
-junithelper is a tool for creating a simple junit test result, which is used for
-creating a single junit result file with a single test
-
-## Usage
-
-This tool can be invoked from command line with following parameters:
-
-- `--suite`: name of suite
-- `--name`: name of test
-- `--err-msg`: (optional) error message, by default it's empty, means test
-  passed
-- `--dest`: (optional) file path for result to be written to, default
-  `junit_result.xml`
-
-### Example
-
-```
-go run junithelper --suite foo --name TestBar --err-msg "Failed Randomly" --dest
-"/tmp/junit_important_suite.xml"
-```
diff --git a/testutils/junithelper/main.go b/testutils/junithelper/main.go
deleted file mode 100644
index f5f9a833a..000000000
--- a/testutils/junithelper/main.go
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
-Copyright 2019 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package main
-
-import (
-	"flag"
-
-	"knative.dev/pkg/test/junit"
-)
-
-var (
-	suite  string
-	name   string
-	errMsg string
-	dest   string
-)
-
-func main() {
-	flag.StringVar(&suite, "suite", "", "Name of suite")
-	flag.StringVar(&name, "name", "", "Name of test")
-	flag.StringVar(&errMsg, "err-msg", "", "Error message, empty means test passed, default empty")
-	flag.StringVar(&dest, "dest", "junit_result.xml", "Where junit xml writes to")
-	flag.Parse()
-
-	junit.CreateXMLErrorMsg(suite, name, errMsg, dest)
-}
diff --git a/vendor/knative.dev/test-infra/scripts/library.sh b/vendor/knative.dev/test-infra/scripts/library.sh
index 92fccb17a..e7ecbc9be 100644
--- a/vendor/knative.dev/test-infra/scripts/library.sh
+++ b/vendor/knative.dev/test-infra/scripts/library.sh
@@ -69,12 +69,12 @@ function abort() {
 # Parameters: $1 - character to use for the box.
 #             $2 - banner message.
 function make_banner() {
-    local msg="$1$1$1$1 $2 $1$1$1$1"
-    local border="${msg//[-0-9A-Za-z _.,\/()\']/$1}"
-    echo -e "${border}\n${msg}\n${border}"
-    # TODO(adrcunha): Remove once logs have timestamps on Prow
-    # For details, see https://github.com/kubernetes/test-infra/issues/10100
-    echo -e "$1$1$1$1 $(TZ='America/Los_Angeles' date)\n${border}"
+  local msg="$1$1$1$1 $2 $1$1$1$1"
+  local border="${msg//[-0-9A-Za-z _.,\/()\']/$1}"
+  echo -e "${border}\n${msg}\n${border}"
+  # TODO(adrcunha): Remove once logs have timestamps on Prow
+  # For details, see https://github.com/kubernetes/test-infra/issues/10100
+  echo -e "$1$1$1$1 $(TZ='America/Los_Angeles' date)\n${border}"
 }
 
 # Simple header for logging purposes.
@@ -127,70 +127,20 @@ function wait_until_object_does_not_exist() {
 # Waits until all pods are running in the given namespace.
 # Parameters: $1 - namespace.
 function wait_until_pods_running() {
-  echo -n "Waiting until all pods in namespace $1 are up"
-  local failed_pod=""
-  for i in {1..150}; do  # timeout after 5 minutes
-    # List all pods. Ignore Terminating pods as those have either been replaced through
-    # a deployment or terminated on purpose (through chaosduck for example).
-    local pods="$(kubectl get pods --no-headers -n $1 2>/dev/null | grep -v Terminating)"
-    # All pods must be running (ignore ImagePull error to allow the pod to retry)
-    local not_running_pods=$(echo "${pods}" | grep -v Running | grep -v Completed | grep -v ErrImagePull | grep -v ImagePullBackOff)
-    if [[ -n "${pods}" ]] && [[ -z "${not_running_pods}" ]]; then
-      # All Pods are running or completed. Verify the containers on each Pod.
-      local all_ready=1
-      while read pod ; do
-        local status=(`echo -n ${pod} | cut -f2 -d' ' | tr '/' ' '`)
-        # Set this Pod as the failed_pod. If nothing is wrong with it, then after the checks, set
-        # failed_pod to the empty string.
-        failed_pod=$(echo -n "${pod}" | cut -f1 -d' ')
-        # All containers must be ready
-        [[ -z ${status[0]} ]] && all_ready=0 && break
-        [[ -z ${status[1]} ]] && all_ready=0 && break
-        [[ ${status[0]} -lt 1 ]] && all_ready=0 && break
-        [[ ${status[1]} -lt 1 ]] && all_ready=0 && break
-        [[ ${status[0]} -ne ${status[1]} ]] && all_ready=0 && break
-        # All the tests passed, this is not a failed pod.
-        failed_pod=""
-      done <<< "$(echo "${pods}" | grep -v Completed)"
-      if (( all_ready )); then
-        echo -e "\nAll pods are up:\n${pods}"
-        return 0
-      fi
-    elif [[ -n "${not_running_pods}" ]]; then
-      # At least one Pod is not running, just save the first one's name as the failed_pod.
-      failed_pod="$(echo "${not_running_pods}" | head -n 1 | cut -f1 -d' ')"
-    fi
-    echo -n "."
-    sleep 2
-  done
-  echo -e "\n\nERROR: timeout waiting for pods to come up\n${pods}"
-  if [[ -n "${failed_pod}" ]]; then
-    echo -e "\n\nFailed Pod (data in YAML format) - ${failed_pod}\n"
-    kubectl -n $1 get pods "${failed_pod}" -oyaml
-    echo -e "\n\nPod Logs\n"
-    kubectl -n $1 logs "${failed_pod}" --all-containers
+  echo "Waiting until all pods in namespace $1 are up"
+  kubectl wait pod --for=condition=Ready -n "$1" -l '!job-name' --timeout=5m || return 1
+  # Also wait for all the job pods to be completed.
+  # This is mainly for maintaining backward compatibility.
+  if [[ $(kubectl get jobs --ignore-not-found=true -n "$1") ]]; then
+    kubectl wait job --for=condition=Complete --all -n "$1" --timeout=5m || return 1
   fi
-  return 1
 }
 
 # Waits until all batch jobs complete in the given namespace.
 # Parameters: $1 - namespace.
 function wait_until_batch_job_complete() {
-  echo -n "Waiting until all batch jobs in namespace $1 run to completion."
-  for i in {1..150}; do  # timeout after 5 minutes
-    local jobs=$(kubectl get jobs -n $1 --no-headers \
-                 -ocustom-columns='n:{.metadata.name},c:{.spec.completions},s:{.status.succeeded}')
-    # All jobs must be complete
-    local not_complete=$(echo "${jobs}" | awk '{if ($2!=$3) print $0}' | wc -l)
-    if [[ ${not_complete} -eq 0 ]]; then
-      echo -e "\nAll jobs are complete:\n${jobs}"
-      return 0
-    fi
-    echo -n "."
-    sleep 2
-  done
-  echo -e "\n\nERROR: timeout waiting for jobs to complete\n${jobs}"
-  return 1
+  echo "Waiting until all batch jobs in namespace $1 run to completion."
+  kubectl wait job --for=condition=Complete --all -n "$1" --timeout=5m || return 1
 }
 
 # Waits until the given service has an external address (IP/hostname).
diff --git a/webhook/psbinding/table_test.go b/webhook/psbinding/table_test.go
index 1ae11be65..34429fe12 100644
--- a/webhook/psbinding/table_test.go
+++ b/webhook/psbinding/table_test.go
@@ -26,6 +26,7 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	jsonpatch "gomodules.xyz/jsonpatch/v2"
+	fakek8s "k8s.io/client-go/kubernetes/fake"
 	"knative.dev/pkg/apis"
 	"knative.dev/pkg/client/injection/ducks/duck/v1/podspecable"
 	kubeclient "knative.dev/pkg/client/injection/kube/client/fake"
@@ -2008,6 +2009,188 @@ func TestBaseReconcile(t *testing.T) {
 	}))
 }
 
+func TestBaseReconcileWithSubResourcesReconciler(t *testing.T) {
+	table := TableTest{{
+		Name: "create new subresource",
+		Key:  "foo/bar",
+		Objects: []runtime.Object{
+			&TestBindable{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace:  "foo",
+					Name:       "bar",
+					Finalizers: []string{"testbindables.duck.knative.dev"},
+				},
+				Spec: TestBindableSpec{
+					BindingSpec: duckv1alpha1.BindingSpec{
+						Subject: tracker.Reference{
+							APIVersion: "apps/v1",
+							Kind:       "Deployment",
+							Namespace:  "foo",
+							Name:       "on-it",
+						},
+					},
+					Foo: "asdfasdfasdfasdf",
+				},
+				Status: TestBindableStatus{
+					Status: duckv1.Status{
+						Conditions: []apis.Condition{{
+							Type:   "Ready",
+							Status: "True",
+						}},
+					},
+				},
+			},
+			&appsv1.Deployment{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace: "foo",
+					Name:      "on-it",
+				},
+				Spec: appsv1.DeploymentSpec{
+					Template: corev1.PodTemplateSpec{
+						Spec: corev1.PodSpec{
+							Containers: []corev1.Container{{
+								Name:  "foo",
+								Image: "busybox",
+								Env: []corev1.EnvVar{{
+									Name:  "FOO",
+									Value: "asdfasdfasdfasdf",
+								}},
+							}},
+						},
+					},
+				},
+			},
+			&corev1.Namespace{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "foo",
+				},
+			},
+		},
+		WantPatches: []clientgotesting.PatchActionImpl{
+			patchAddLabel("foo"),
+		},
+		WantCreates: []runtime.Object{
+			&corev1.ConfigMap{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "bar",
+				},
+			},
+		},
+	},
+		{
+			Name: "delete resource and subresource",
+			Key:  "foo/bar",
+			Objects: []runtime.Object{
+				&TestBindable{
+					ObjectMeta: metav1.ObjectMeta{
+						Namespace:         "foo",
+						DeletionTimestamp: &metav1.Time{Time: time.Now()},
+						Name:              "bar",
+						Finalizers:        []string{"testbindables.duck.knative.dev"},
+					},
+					Spec: TestBindableSpec{
+						BindingSpec: duckv1alpha1.BindingSpec{
+							Subject: tracker.Reference{
+								APIVersion: "apps/v1",
+								Kind:       "Deployment",
+								Namespace:  "foo",
+								Name:       "on-it",
+							},
+						},
+						Foo: "asdfasdfasdfasdf",
+					},
+					Status: TestBindableStatus{
+						Status: duckv1.Status{
+							Conditions: []apis.Condition{{
+								Type:   "Ready",
+								Status: "True",
+							}},
+						},
+					},
+				},
+				&appsv1.Deployment{
+					ObjectMeta: metav1.ObjectMeta{
+						Namespace: "foo",
+						Name:      "on-it",
+					},
+					Spec: appsv1.DeploymentSpec{
+						Template: corev1.PodTemplateSpec{
+							Spec: corev1.PodSpec{
+								Containers: []corev1.Container{{
+									Name:  "foo",
+									Image: "busybox",
+									Env: []corev1.EnvVar{{
+										Name:  "FOO",
+										Value: "asdfasdfasdfasdf",
+									}},
+								}},
+							},
+						},
+					},
+				},
+				&corev1.Namespace{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "foo",
+					},
+				},
+				&corev1.ConfigMap{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "bar",
+						Namespace: "foo",
+					},
+				},
+			},
+			WantPatches: []clientgotesting.PatchActionImpl{
+				patchAddLabel("foo"),
+				patchRemoveEnv("foo", "on-it"),
+				patchRemoveFinalizer("foo", "bar", "" /* resource version */),
+			},
+			WantDeletes: []clientgotesting.DeleteActionImpl{
+				deletedConfigmap("foo", "bar"),
+			},
+		},
+	}
+
+	table.Test(t, MakeFactory(func(ctx context.Context, listers *Listers, cmw configmap.Watcher) controller.Reconciler {
+		gvr := SchemeGroupVersion.WithResource("testbindables")
+		ctx = podspecable.WithDuck(ctx)
+
+		dc := dynamicclient.Get(ctx)
+
+		kc := kubeclient.Get(ctx)
+		srr := &dummySubResourcesReconciler{
+			client: kc,
+		}
+		return &BaseReconciler{
+			GVR: gvr,
+
+			DynamicClient: dc,
+			Factory:       podspecable.Get(ctx),
+
+			Tracker: &FakeTracker{},
+
+			Recorder: record.NewFakeRecorder(20),
+
+			Get: func(namespace, name string) (Bindable, error) {
+				for _, elt := range listers.GetDuckObjects() {
+					b, ok := elt.(*TestBindable)
+					if !ok {
+						continue
+					}
+					if b.Namespace != namespace || b.Name != name {
+						continue
+					}
+					return b, nil
+				}
+				return nil, apierrs.NewNotFound(gvr.GroupResource(), name)
+			},
+			NamespaceLister: listers.GetNamespaceLister(),
+
+			SubResourcesReconciler: srr,
+		}
+	}))
+}
+
 func mustTU(t *testing.T, ro duck.OneOfOurs) *unstructured.Unstructured {
 	u, err := duck.ToUnstructured(ro)
 	if err != nil {
@@ -2080,3 +2263,40 @@ func patchRemoveEnv(namespace, name string) clientgotesting.PatchActionImpl {
 	action.Patch = []byte(patch)
 	return action
 }
+
+func deletedConfigmap(namespace string, name string) clientgotesting.DeleteActionImpl {
+	action := clientgotesting.DeleteActionImpl{}
+	resource := schema.GroupVersionResource{
+		Group:    "core",
+		Version:  "v1",
+		Resource: "configmaps",
+	}
+	actionImpl := clientgotesting.ActionImpl{
+		Namespace:   namespace,
+		Verb:        "delete",
+		Resource:    resource,
+		Subresource: "",
+	}
+	action.ActionImpl = actionImpl
+	action.Name = name
+	return action
+}
+
+type dummySubResourcesReconciler struct {
+	client *fakek8s.Clientset
+}
+
+func (d *dummySubResourcesReconciler) Reconcile(ctx context.Context, fb Bindable) error {
+	cfg := corev1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: fb.GetName(),
+		},
+	}
+
+	_, err := d.client.CoreV1().ConfigMaps(fb.GetNamespace()).Create(ctx, &cfg, metav1.CreateOptions{})
+	return err
+}
+
+func (d *dummySubResourcesReconciler) ReconcileDeletion(ctx context.Context, fb Bindable) error {
+	return d.client.CoreV1().ConfigMaps(fb.GetNamespace()).Delete(ctx, fb.GetName(), metav1.DeleteOptions{})
+}