Repository: knative/serving
Pull request #11909 (2021-09-03 07:01:45)

Speed up concurrency state implementation
<!-- Please include the 'why' behind your changes if no issue exists -->
## Proposed Changes

**Note:** Please take your time reviewing this. I might totally have missed a race condition somewhere and this might be utter garbage. If so, I'm super happy to bin this. This was mostly some friday "let's solve a trickyish concurrency problem quicker" type fun :joy:.

This rewrites the ConcurrencyStateHandler from a channel based approach
into an atomic/mux based approach, to minimize contention and allow for
more concurrent processing.

An RWMutex is used to make sure that calling either pause or resume is
mutually exclusive to each other and also mutually exclusive to the
requests themselves, though the requests can happily run in parallel.

An important race to consider is a request entering into the pausing
logic while another request enters the resume block. Before pausing,
it's important to recheck if `inFlight` is still 0 **after** obtaining
the lock to make sure no other request has raced obtaining the lock. The
race can happen because the increment/decrement of `inFlight` is not
atomic to getting the lock so the following order of operations could
happen:

1. A: inFlight.Dec() == 0 --> true
2. B: inFlight.Inc()
3. B: mux.RLock()
4. B: !paused --> true
5. A: !paused --> true
6. A: pause()
7. B: h.ServeHTTP --> Boom.

In step 6, the pause would not have been excecuted in the actual
implementation though, because we doublecheck the state of `inFlight`,
which wouldn't have been 0 anymore.

## Benchmark diff

```
benchmark                                                                             old ns/op     new ns/op     delta
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-no-reports-16             8123          1770          -78.21%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-no-reports-16               1919          672           -64.98%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-no-reports-16       8093          1319          -83.70%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-no-reports-16         1886          631           -66.55%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-many-reports-16           7657          1854          -75.79%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-many-reports-16             1889          693           -63.34%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-many-reports-16     7149          1428          -80.03%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-many-reports-16       1839          673           -63.38%

benchmark                                                                             old allocs     new allocs     delta
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-no-reports-16             11             10             -9.09%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-no-reports-16               11             10             -9.09%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-no-reports-16       7              6              -14.29%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-no-reports-16         7              6              -14.29%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-many-reports-16           11             10             -9.09%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-many-reports-16             11             10             -9.09%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-many-reports-16     7              6              -14.29%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-many-reports-16       7              6              -14.29%

benchmark                                                                             old bytes     new bytes     delta
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-no-reports-16             912           816           -10.53%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-no-reports-16               912           816           -10.53%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-no-reports-16       704           608           -13.64%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-no-reports-16         704           608           -13.64%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-10-many-reports-16           912           816           -10.53%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-10-many-reports-16             912           816           -10.53%
BenchmarkConcurrencyStateProxyHandler/sequential-breaker-infinite-many-reports-16     704           608           -13.64%
BenchmarkConcurrencyStateProxyHandler/parallel-breaker-infinite-many-reports-16       704           608           -13.64%
```

/assign @psschwei @julz 

knative-prow-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/serving/pull/11909#" title="Author self-approved">markusthoemmes</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[pkg/queue/OWNERS](https://github.com/knative/serving/blob/main/pkg/queue/OWNERS)~~ [markusthoemmes]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov[bot]: # [Codecov](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative) Report
> Merging [#11909](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative) (ae4412d) into [main](https://codecov.io/gh/knative/serving/commit/cefe5ed204e2ec0d1c179e82da360443dea84b08?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative) (cefe5ed) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/knative/serving/pull/11909/graphs/tree.svg?width=650&height=150&src=pr&token=MPpDfAj4Ui&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative)](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative)

```diff
@@            Coverage Diff             @@
##             main   #11909      +/-   ##
==========================================
+ Coverage   87.69%   87.70%   +0.01%     
==========================================
  Files         197      197              
  Lines        9467     9461       -6     
==========================================
- Hits         8302     8298       -4     
+ Misses        901      900       -1     
+ Partials      264      263       -1     
```


| [Impacted Files](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative) | Coverage Δ | |
|---|---|---|
| [pkg/queue/concurrency\_state.go](https://codecov.io/gh/knative/serving/pull/11909/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative#diff-cGtnL3F1ZXVlL2NvbmN1cnJlbmN5X3N0YXRlLmdv) | `85.18% <100.00%> (-2.70%)` | :arrow_down: |
| [pkg/reconciler/revision/reconcile\_resources.go](https://codecov.io/gh/knative/serving/pull/11909/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative#diff-cGtnL3JlY29uY2lsZXIvcmV2aXNpb24vcmVjb25jaWxlX3Jlc291cmNlcy5nbw==) | `83.13% <0.00%> (+2.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative). Last update [cefe5ed...ae4412d](https://codecov.io/gh/knative/serving/pull/11909?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative).

markusthoemmes: /hold

Actually found a race that's not handled properly
markusthoemmes: Might be fixed, but this is getting complex :joy: 
markusthoemmes: Almost sure I can actually replace the waiters stuff with a doubly checked lock and allowing the requests to race to resume 🤔
psschwei: Interesting... I'm going to check how this plays with the pause/resume functions from #11802 (which is awaiting review ;) ). I don't anticipate any issues, just want to poke around with it a bit. Then will give a review, probably after the US holiday weekend. 

(The performance improvement is quite nice!)
psschwei: One interesting thing I noticed is that, in certain cases, the pause/resume functions are making two HTTP calls. 

Here's what I did:

* I cloned your branch and added pause/resume functions from #11802 (and made some minor changes to function signatures): https://github.com/psschwei/serving/tree/rewrite-concurrencystate
* Set up a dev environment on minikube using this branch (w/ kourier networking)
* Deployed the [pod-freezer](https://github.com/psschwei/pod-freezer) service, which as of now just logs when a pause/resume request comes in
* Deployed julz's [sleeptalker app](https://github.com/psschwei/pod-freezer), which takes two seconds to run its main task (and thus doesn't result in pause being called immediately after resume)
* Used hey to simulate multiple requests: `hey -c 7 -n 10 http://example.default.10.98.123.235.sslip.io`

Here's the interesting part:

logs for queue-proxy:
```
{"severity":"INFO","timestamp":"2021-09-04T00:19:07.441246031Z","logger":"queueproxy","caller":"queue/concurrency_state.go:79","message":"Requests increased from zero","commit":"f4b0ad7","knative.dev/key":"default/example-00001","knative.dev/pod":"example-00001-deployment-678bd67c76-xd6w4"}
{"severity":"INFO","timestamp":"2021-09-04T00:19:09.445299796Z","logger":"queueproxy","caller":"queue/concurrency_state.go:52","message":"Requests dropped to zero","commit":"f4b0ad7","knative.dev/key":"default/example-00001","knative.dev/pod":"example-00001-deployment-678bd67c76-xd6w4"}

```

logs for pod-freezer:
```
{"severity":"INFO","timestamp":"2021-09-04T00:19:07.44293111Z","caller":"pod-freezer/main.go:34","message":"resume request received, thawing pod..."}
{"severity":"INFO","timestamp":"2021-09-04T00:19:07.444625075Z","caller":"pod-freezer/main.go:34","message":"resume request received, thawing pod..."}
{"severity":"INFO","timestamp":"2021-09-04T00:19:09.445622954Z","caller":"pod-freezer/main.go:32","message":"pause request received, freezing pod..."}
{"severity":"INFO","timestamp":"2021-09-04T00:19:09.447745441Z","caller":"pod-freezer/main.go:32","message":"pause request received, freezing pod..."}
```

So the pod-freezer service got two resume/pause requests, but QP only sent one of each....

This doesn't happen every time, and fwiw even when it does the end result seems to be what we want (i.e. the pod was resumed and then paused only after all requests finished). I won't be able to dig into it more until next week, but figured it was worth noting before then. 

It also could very well just be some issue with the pod-freezer and not QP, in which case we can pretend this comment was never made :smile: ...


psschwei: @julz just checked it, and yes the duplicate logs also occur in the current code (i.e. #11802)

Patch:
diff --git a/go.mod b/go.mod
index bffa8b611..dfe9d3e11 100644
--- a/go.mod
+++ b/go.mod
@@ -45,9 +45,9 @@ require (
 	k8s.io/code-generator v0.21.4
 	k8s.io/kube-openapi v0.0.0-20210305001622-591a79e4bda7
 	k8s.io/utils v0.0.0-20210111153108-fddb29f9d009 // indirect
-	knative.dev/caching v0.0.0-20210909065659-e5d224326175
+	knative.dev/caching v0.0.0-20210910063632-a51cf6760e00
 	knative.dev/hack v0.0.0-20210806075220-815cd312d65c
-	knative.dev/networking v0.0.0-20210909065659-060e46eeddbd
-	knative.dev/pkg v0.0.0-20210909102158-d569db39a812
+	knative.dev/networking v0.0.0-20210910120730-eb37d69eb87f
+	knative.dev/pkg v0.0.0-20210909165259-d4505c660535
 	sigs.k8s.io/yaml v1.2.0
 )
diff --git a/go.sum b/go.sum
index c4f5bd5e3..dd9fa56c8 100644
--- a/go.sum
+++ b/go.sum
@@ -1526,15 +1526,14 @@ k8s.io/legacy-cloud-providers v0.21.0/go.mod h1:bNxo7gDg+PGkBmT/MFZswLTWdSWK9kAl
 k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
 k8s.io/utils v0.0.0-20210111153108-fddb29f9d009 h1:0T5IaWHO3sJTEmCP6mUlBvMukxPKUQWqiI/YuiBNMiQ=
 k8s.io/utils v0.0.0-20210111153108-fddb29f9d009/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
-knative.dev/caching v0.0.0-20210909065659-e5d224326175 h1:fJVzj33VnyLaQ41hKglWJbWX3R92UnnuMzK0FkP+ZnQ=
-knative.dev/caching v0.0.0-20210909065659-e5d224326175/go.mod h1:XWnA7pdlpf5b8h2PEQN0PhL/h+i9Iv4w10BLcS5Tz8A=
+knative.dev/caching v0.0.0-20210910063632-a51cf6760e00 h1:BObI5QX32nefcKWZlJATp0sWyj4LjDq596AlwCVpC7k=
+knative.dev/caching v0.0.0-20210910063632-a51cf6760e00/go.mod h1:PMt1PGfOsjGpIj0m6QWIfutn+5mY4WPGf3q4MP6oFf0=
 knative.dev/hack v0.0.0-20210806075220-815cd312d65c h1:nOXoDWAAItwr4o0dp3nHr6skgpVD4IvME/UX84YNl5k=
 knative.dev/hack v0.0.0-20210806075220-815cd312d65c/go.mod h1:PHt8x8yX5Z9pPquBEfIj0X66f8iWkWfR0S/sarACJrI=
-knative.dev/networking v0.0.0-20210909065659-060e46eeddbd h1:wQqcXHW9hAI8I9zKIzIjX3Vu1qCKcs7hT/AgcGwe6OA=
-knative.dev/networking v0.0.0-20210909065659-060e46eeddbd/go.mod h1:viMddr/gluqUQfSAvhJs0OxKBzh0Df5bELo8A/HS5qc=
-knative.dev/pkg v0.0.0-20210908202858-9a4b6128207c/go.mod h1:jMSqkNMsrzuy+XR4Yr/BMy7SDVbUOl3KKB6+5MR+ZU8=
-knative.dev/pkg v0.0.0-20210909102158-d569db39a812 h1:8EQzOnAxqP1inXNlfeFrjcRUbuwZ7HldgJLqNQnuOxY=
-knative.dev/pkg v0.0.0-20210909102158-d569db39a812/go.mod h1:jMSqkNMsrzuy+XR4Yr/BMy7SDVbUOl3KKB6+5MR+ZU8=
+knative.dev/networking v0.0.0-20210910120730-eb37d69eb87f h1:FHPmpIrUnJjoI2xBIpxTAvCoh9Y1nNHAOWRSJRJECww=
+knative.dev/networking v0.0.0-20210910120730-eb37d69eb87f/go.mod h1:Bq/n7aJlzPiieLpB/ndIgZXUJeGf50LXN7AQ/UaL6KA=
+knative.dev/pkg v0.0.0-20210909165259-d4505c660535 h1:1bbKs8NGGodV0LjOyn6s5lo+oAJ/9KtTURUZtpRxr60=
+knative.dev/pkg v0.0.0-20210909165259-d4505c660535/go.mod h1:jMSqkNMsrzuy+XR4Yr/BMy7SDVbUOl3KKB6+5MR+ZU8=
 pgregory.net/rapid v0.3.3 h1:jCjBsY4ln4Atz78QoBWxUEvAHaFyNDQg9+WU62aCn1U=
 pgregory.net/rapid v0.3.3/go.mod h1:UYpPVyjFHzYBGHIxLFoupi8vwk6rXNzRY9OMvVxFIOU=
 rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
diff --git a/pkg/client/injection/reconciler/autoscaling/v1alpha1/metric/reconciler.go b/pkg/client/injection/reconciler/autoscaling/v1alpha1/metric/reconciler.go
index f7ed0efec..e2f113287 100644
--- a/pkg/client/injection/reconciler/autoscaling/v1alpha1/metric/reconciler.go
+++ b/pkg/client/injection/reconciler/autoscaling/v1alpha1/metric/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go b/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go
index f0fc1ff3d..be15f07ba 100644
--- a/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go
+++ b/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go
@@ -262,6 +262,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1/configuration/reconciler.go b/pkg/client/injection/reconciler/serving/v1/configuration/reconciler.go
index db9565d25..ed47e6eaf 100644
--- a/pkg/client/injection/reconciler/serving/v1/configuration/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1/configuration/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go b/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go
index e432ba3de..9b7c9e955 100644
--- a/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1/route/reconciler.go b/pkg/client/injection/reconciler/serving/v1/route/reconciler.go
index 3d510d567..bf3bac9f3 100644
--- a/pkg/client/injection/reconciler/serving/v1/route/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1/route/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1/service/reconciler.go b/pkg/client/injection/reconciler/serving/v1/service/reconciler.go
index 84fc51d4e..a14342e12 100644
--- a/pkg/client/injection/reconciler/serving/v1/service/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1/service/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1alpha1/domainmapping/reconciler.go b/pkg/client/injection/reconciler/serving/v1alpha1/domainmapping/reconciler.go
index dc6ae8167..6fca16803 100644
--- a/pkg/client/injection/reconciler/serving/v1alpha1/domainmapping/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1alpha1/domainmapping/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/client/injection/reconciler/serving/v1beta1/domainmapping/reconciler.go b/pkg/client/injection/reconciler/serving/v1beta1/domainmapping/reconciler.go
index 507976edf..cb256e549 100644
--- a/pkg/client/injection/reconciler/serving/v1beta1/domainmapping/reconciler.go
+++ b/pkg/client/injection/reconciler/serving/v1beta1/domainmapping/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/pkg/queue/concurrency_state.go b/pkg/queue/concurrency_state.go
index 9220c2924..53a47fc64 100644
--- a/pkg/queue/concurrency_state.go
+++ b/pkg/queue/concurrency_state.go
@@ -18,7 +18,9 @@ package queue
 
 import (
 	"net/http"
+	"sync"
 
+	"go.uber.org/atomic"
 	"go.uber.org/zap"
 )
 
@@ -38,50 +40,48 @@ func ConcurrencyStateHandler(logger *zap.SugaredLogger, h http.Handler, pause, r
 		resume = func() {}
 	}
 
-	type req struct {
-		w http.ResponseWriter
-		r *http.Request
+	var (
+		inFlight = atomic.NewInt64(0)
+		paused   = true
+		mux      sync.RWMutex
+	)
 
-		done chan struct{}
-	}
-
-	reqCh := make(chan req)
-	doneCh := make(chan struct{})
-	go func() {
-		inFlight := 0
-
-		// This loop is entirely synchronous, so there's no cleverness needed in
-		// ensuring open and close dont run at the same time etc. Only the
-		// delegated ServeHTTP is done in a goroutine.
-		for {
-			select {
-			case <-doneCh:
-				inFlight--
-				if inFlight == 0 {
-					logger.Info("Requests dropped to zero ...")
+	return func(w http.ResponseWriter, r *http.Request) {
+		defer func() {
+			if inFlight.Dec() == 0 {
+				mux.Lock()
+				defer mux.Unlock()
+				// We need to doublecheck this since another request can have reached the
+				// handler meanwhile. We don't want to do anything in that case.
+				if !paused && inFlight.Load() == 0 {
 					pause()
+					paused = true
 				}
+			}
+		}()
 
-			case r := <-reqCh:
-				inFlight++
-				if inFlight == 1 {
-					logger.Info("Requests increased from zero ...")
-					resume()
-				}
+		inFlight.Inc()
 
-				go func(r req) {
-					h.ServeHTTP(r.w, r.r)
-					close(r.done) // Return from ServeHTTP
-					doneCh <- struct{}{}
-				}(r)
-			}
+		mux.RLock()
+		if !paused {
+			// General stable-state case.
+			defer mux.RUnlock()
+			h.ServeHTTP(w, r)
+			return
+		}
+		mux.RUnlock()
+		mux.Lock()
+		if !paused { // doubly-checked locking
+			// Another request raced us and resumed already.
+			defer mux.Unlock()
+			h.ServeHTTP(w, r)
+			return
 		}
-	}()
 
-	return func(w http.ResponseWriter, r *http.Request) {
-		done := make(chan struct{})
-		reqCh <- req{w, r, done}
-		// Block till we've processed the request
-		<-done
+		resume()
+		paused = false
+		mux.Unlock()
+
+		h.ServeHTTP(w, r)
 	}
 }
diff --git a/pkg/reconciler/revision/background.go b/pkg/reconciler/revision/background.go
index 470d8ba57..af5d2ff7c 100644
--- a/pkg/reconciler/revision/background.go
+++ b/pkg/reconciler/revision/background.go
@@ -144,7 +144,7 @@ func (r *backgroundResolver) Start(stop <-chan struct{}, maxInFlight int) (done
 // If this method returns `nil, nil` this implies a resolve was triggered or is
 // already in progress, so the reconciler should exit and wait for the revision
 // to be re-enqueued when the result is ready.
-func (r *backgroundResolver) Resolve(rev *v1.Revision, opt k8schain.Options, registriesToSkip sets.String, timeout time.Duration) ([]v1.ContainerStatus, error) {
+func (r *backgroundResolver) Resolve(logger *zap.SugaredLogger, rev *v1.Revision, opt k8schain.Options, registriesToSkip sets.String, timeout time.Duration) ([]v1.ContainerStatus, error) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
 
@@ -155,16 +155,19 @@ func (r *backgroundResolver) Resolve(rev *v1.Revision, opt k8schain.Options, reg
 
 	result, inFlight := r.results[name]
 	if !inFlight {
+		logger.Debugf("Adding Resolve request to queue (depth: %d)", r.queue.Len())
 		r.addWorkItems(rev, name, opt, registriesToSkip, timeout)
 		return nil, nil
 	}
 
 	if !result.ready() {
+		logger.Debug("Resolve request in flight, returning nil, nil")
 		return nil, nil
 	}
 
 	ret := r.results[name]
 	if ret.err != nil {
+		logger.Debugf("Resolve returned the resolved error: %v", ret.err)
 		return nil, ret.err
 	}
 
@@ -173,7 +176,8 @@ func (r *backgroundResolver) Resolve(rev *v1.Revision, opt k8schain.Options, reg
 		array[k] = v
 	}
 
-	return array, ret.err
+	logger.Debugf("Resolve returned %d resolved images for revision", len(array))
+	return array, nil
 }
 
 // addWorkItems adds a digest resolve item to the queue for each container in the revision.
@@ -209,6 +213,7 @@ func (r *backgroundResolver) addWorkItems(rev *v1.Revision, name types.Namespace
 // completionCallback is called.
 func (r *backgroundResolver) processWorkItem(item workItem) {
 	defer r.queue.Done(item)
+	r.logger.Debugf("Processing image %q from revision %q (depth: %d)", item.image, item.revision, r.queue.Len())
 
 	// We need to acquire the result under lock since it's theoretically possible
 	// for a Clear to race with this and try to delete the result from the map.
@@ -223,7 +228,9 @@ func (r *backgroundResolver) processWorkItem(item workItem) {
 	ctx, cancel := context.WithTimeout(context.Background(), item.timeout)
 	defer cancel()
 
+	r.logger.Debugf("Resolving image %q from revision %q to digest", item.image, item.revision)
 	resolvedDigest, resolveErr := r.resolver.Resolve(ctx, item.image, result.opt, result.registriesToSkip)
+	r.logger.Debugf("Resolved image %q from revision %q to digest %q, %v", item.image, item.revision, resolvedDigest, resolveErr)
 
 	// lock after the resolve because we don't want to block parallel resolves,
 	// just storing the result.
diff --git a/pkg/reconciler/revision/revision.go b/pkg/reconciler/revision/revision.go
index 3decda21d..fafd5c73a 100644
--- a/pkg/reconciler/revision/revision.go
+++ b/pkg/reconciler/revision/revision.go
@@ -23,6 +23,7 @@ import (
 
 	"github.com/davecgh/go-spew/spew"
 	"github.com/google/go-containerregistry/pkg/authn/k8schain"
+	"go.uber.org/zap"
 	"go.uber.org/zap/zapcore"
 
 	corev1 "k8s.io/api/core/v1"
@@ -44,7 +45,7 @@ import (
 )
 
 type resolver interface {
-	Resolve(*v1.Revision, k8schain.Options, sets.String, time.Duration) ([]v1.ContainerStatus, error)
+	Resolve(*zap.SugaredLogger, *v1.Revision, k8schain.Options, sets.String, time.Duration) ([]v1.ContainerStatus, error)
 	Clear(types.NamespacedName)
 	Forget(types.NamespacedName)
 }
@@ -102,7 +103,8 @@ func (c *Reconciler) reconcileDigest(ctx context.Context, rev *v1.Revision) (boo
 		ImagePullSecrets:   imagePullSecrets,
 	}
 
-	statuses, err := c.resolver.Resolve(rev, opt, cfgs.Deployment.RegistriesSkippingTagResolving, cfgs.Deployment.DigestResolutionTimeout)
+	logger := logging.FromContext(ctx)
+	statuses, err := c.resolver.Resolve(logger, rev, opt, cfgs.Deployment.RegistriesSkippingTagResolving, cfgs.Deployment.DigestResolutionTimeout)
 	if err != nil {
 		// Clear the resolver so we can retry the digest resolution rather than
 		// being stuck with this error.
diff --git a/pkg/reconciler/route/route.go b/pkg/reconciler/route/route.go
index edf5675de..f522e5b49 100644
--- a/pkg/reconciler/route/route.go
+++ b/pkg/reconciler/route/route.go
@@ -24,6 +24,8 @@ import (
 	"strings"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	kubelabels "k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/util/clock"
@@ -38,9 +40,11 @@ import (
 	networkinglisters "knative.dev/networking/pkg/client/listers/networking/v1alpha1"
 	"knative.dev/pkg/apis"
 	duckv1 "knative.dev/pkg/apis/duck/v1"
+	"knative.dev/pkg/controller"
 	"knative.dev/pkg/logging"
 	pkgreconciler "knative.dev/pkg/reconciler"
 	"knative.dev/pkg/tracker"
+	"knative.dev/serving/pkg/apis/serving"
 	v1 "knative.dev/serving/pkg/apis/serving/v1"
 	clientset "knative.dev/serving/pkg/client/clientset/versioned"
 	routereconciler "knative.dev/serving/pkg/client/injection/reconciler/serving/v1/route"
@@ -259,9 +263,55 @@ func (c *Reconciler) tls(ctx context.Context, host string, r *v1.Route, traffic
 	sort.Slice(acmeChallenges, func(i, j int) bool {
 		return acmeChallenges[i].URL.String() < acmeChallenges[j].URL.String()
 	})
+
+	orphanCerts, err := c.getOrphanRouteCerts(r, domainToTagMap)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	recorder := controller.GetEventRecorder(ctx)
+	for _, cert := range orphanCerts {
+		err = c.GetNetworkingClient().NetworkingV1alpha1().Certificates(cert.Namespace).Delete(ctx, cert.Name, metav1.DeleteOptions{})
+		if err != nil {
+			recorder.Eventf(cert, corev1.EventTypeNormal, "DeleteFailed",
+				"Failed to delete orphaned Knative Certificate %s/%s: %v", cert.Namespace, cert.Name, err)
+		} else {
+			recorder.Eventf(cert, corev1.EventTypeNormal, "Deleted",
+				"Deleted orphaned Knative Certificate %s/%s", cert.Namespace, cert.Name)
+		}
+	}
+
 	return tls, acmeChallenges, nil
 }
 
+// Returns a slice of certificates that used to belong route's old tags and are currently not in use.
+func (c *Reconciler) getOrphanRouteCerts(r *v1.Route, domainToTagMap map[string]string) ([]*netv1alpha1.Certificate, error) {
+	labelSelector := kubelabels.SelectorFromSet(kubelabels.Set{
+		serving.RouteLabelKey: r.Name,
+	})
+
+	certs, err := c.certificateLister.Certificates(r.Namespace).List(labelSelector)
+	if err != nil {
+		return nil, err
+	}
+
+	var unusedCerts []*netv1alpha1.Certificate
+	for _, cert := range certs {
+		var shouldKeepCert bool
+		for _, dn := range cert.Spec.DNSNames {
+			if _, used := domainToTagMap[dn]; used {
+				shouldKeepCert = true
+			}
+		}
+
+		if !shouldKeepCert {
+			unusedCerts = append(unusedCerts, cert)
+		}
+	}
+
+	return unusedCerts, nil
+}
+
 // configureTraffic attempts to configure traffic based on the RouteSpec.  If there are missing
 // targets (e.g. Configurations without a Ready Revision, or Revision that isn't Ready or Inactive),
 // no traffic will be configured.
diff --git a/vendor/knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/serverlessservice/reconciler.go b/vendor/knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/serverlessservice/reconciler.go
index 34e46dde5..a90410f7b 100644
--- a/vendor/knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/serverlessservice/reconciler.go
+++ b/vendor/knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/serverlessservice/reconciler.go
@@ -251,6 +251,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/vendor/knative.dev/pkg/apis/duck/ABOUT.md b/vendor/knative.dev/pkg/apis/duck/ABOUT.md
index b223412a0..989769220 100644
--- a/vendor/knative.dev/pkg/apis/duck/ABOUT.md
+++ b/vendor/knative.dev/pkg/apis/duck/ABOUT.md
@@ -302,7 +302,11 @@ is used with duck types:
                 ...
         }
         logger := c.Logger.Named("Revisions")
-        impl := controller.NewImpl(c, logger, "Revisions")
+        impl := controller.NewContext(
+          ctx,
+          c,
+          controller.ControllerOptions{WorkQueueName: "Revisions", Logger: logger},
+        )
 
         // Calls to Track create a 30 minute lease before they must be renewed.
         // Coordinate this value with controller resync periods.
diff --git a/vendor/knative.dev/pkg/apis/duck/v1/source_types.go b/vendor/knative.dev/pkg/apis/duck/v1/source_types.go
index 2f1afe246..00e699156 100644
--- a/vendor/knative.dev/pkg/apis/duck/v1/source_types.go
+++ b/vendor/knative.dev/pkg/apis/duck/v1/source_types.go
@@ -17,6 +17,7 @@ limitations under the License.
 package v1
 
 import (
+	"context"
 	"time"
 
 	corev1 "k8s.io/api/core/v1"
@@ -170,3 +171,45 @@ type SourceList struct {
 
 	Items []Source `json:"items"`
 }
+
+func (s *Source) Validate(ctx context.Context) *apis.FieldError {
+	if s == nil {
+		return nil
+	}
+	return s.Spec.Validate(ctx).ViaField("spec")
+}
+
+func (s *SourceSpec) Validate(ctx context.Context) *apis.FieldError {
+	return s.CloudEventOverrides.Validate(ctx).ViaField("ceOverrides")
+}
+
+func (ceOverrides *CloudEventOverrides) Validate(ctx context.Context) *apis.FieldError {
+	for key := range ceOverrides.Extensions {
+		if err := validateExtensionName(key); err != nil {
+			return err.ViaField("extensions")
+		}
+	}
+	return nil
+}
+
+func validateExtensionName(key string) *apis.FieldError {
+	if key == "" {
+		return apis.ErrInvalidKeyName(
+			key,
+			"",
+			"keys MUST NOT be empty",
+		)
+	}
+
+	for _, c := range key {
+		if !((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) {
+			return apis.ErrInvalidKeyName(
+				key,
+				"",
+				"keys are expected to be alphanumeric",
+			)
+		}
+	}
+
+	return nil
+}
diff --git a/vendor/knative.dev/pkg/client/injection/kube/reconciler/core/v1/namespace/reconciler.go b/vendor/knative.dev/pkg/client/injection/kube/reconciler/core/v1/namespace/reconciler.go
index 00747a699..d89572dc0 100644
--- a/vendor/knative.dev/pkg/client/injection/kube/reconciler/core/v1/namespace/reconciler.go
+++ b/vendor/knative.dev/pkg/client/injection/kube/reconciler/core/v1/namespace/reconciler.go
@@ -242,6 +242,10 @@ func (r *reconcilerImpl) Reconcile(ctx context.Context, key string) error {
 			return fmt.Errorf("failed to clear finalizers: %w", err)
 		}
 
+	case reconciler.DoObserveKind:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	// Synchronize the status.
diff --git a/vendor/knative.dev/pkg/codegen/cmd/injection-gen/generators/reconciler_reconciler.go b/vendor/knative.dev/pkg/codegen/cmd/injection-gen/generators/reconciler_reconciler.go
index 96e1edc25..e787096b3 100644
--- a/vendor/knative.dev/pkg/codegen/cmd/injection-gen/generators/reconciler_reconciler.go
+++ b/vendor/knative.dev/pkg/codegen/cmd/injection-gen/generators/reconciler_reconciler.go
@@ -465,6 +465,10 @@ func (r *reconcilerImpl) Reconcile(ctx {{.contextContext|raw}}, key string) erro
 			return {{.fmtErrorf|raw}}("failed to clear finalizers: %w", err)
 		}
 
+	case {{.doObserveKind|raw}}:
+		// Observe any changes to this resource, since we are not the leader.
+		reconcileEvent = do(ctx, resource)
+
 	}
 
 	{{if .hasStatus}}
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 30e03348f..34f634cf1 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -1017,7 +1017,7 @@ k8s.io/utils/buffer
 k8s.io/utils/integer
 k8s.io/utils/pointer
 k8s.io/utils/trace
-# knative.dev/caching v0.0.0-20210909065659-e5d224326175
+# knative.dev/caching v0.0.0-20210910063632-a51cf6760e00
 ## explicit
 knative.dev/caching/config
 knative.dev/caching/pkg/apis/caching
@@ -1042,7 +1042,7 @@ knative.dev/caching/pkg/client/listers/caching/v1alpha1
 ## explicit
 knative.dev/hack
 knative.dev/hack/shell
-# knative.dev/networking v0.0.0-20210909065659-060e46eeddbd
+# knative.dev/networking v0.0.0-20210910120730-eb37d69eb87f
 ## explicit
 knative.dev/networking/config
 knative.dev/networking/pkg
@@ -1072,7 +1072,7 @@ knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/serve
 knative.dev/networking/pkg/client/listers/networking/v1alpha1
 knative.dev/networking/pkg/ingress
 knative.dev/networking/pkg/prober
-# knative.dev/pkg v0.0.0-20210909102158-d569db39a812
+# knative.dev/pkg v0.0.0-20210909165259-d4505c660535
 ## explicit
 knative.dev/pkg/apiextensions/storageversion
 knative.dev/pkg/apiextensions/storageversion/cmd/migrate


Test patch:
diff --git a/.github/workflows/kind-e2e.yaml b/.github/workflows/kind-e2e.yaml
index d10aa0977..92c7be87c 100644
--- a/.github/workflows/kind-e2e.yaml
+++ b/.github/workflows/kind-e2e.yaml
@@ -21,6 +21,7 @@ jobs:
         - v1.19.11
         - v1.20.7
         - v1.21.1
+        - v1.22.0
 
         test-suite:
         - ./test/conformance/runtime
@@ -47,6 +48,11 @@ jobs:
           kind-image-sha: sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
           kingress: kourier
           cluster-suffix: c${{ github.run_id }}.local
+        - k8s-version: v1.22.0
+          kind-version: v0.11.1
+          kind-image-sha: sha256:f97edf7f7ed53c57762b24f90a34fad101386c5bd4d93baeb45449557148c717
+          kingress: kourier
+          cluster-suffix: c${{ github.run_id }}.local
 
           # The tests in api/v1alpha1 require the --enable-alpha flag to be set,
           # and alpha components to be deployed.
diff --git a/pkg/queue/concurrency_state_test.go b/pkg/queue/concurrency_state_test.go
index 85a84f4db..e3d635fc0 100644
--- a/pkg/queue/concurrency_state_test.go
+++ b/pkg/queue/concurrency_state_test.go
@@ -23,7 +23,6 @@ import (
 	"time"
 
 	"go.uber.org/atomic"
-	"k8s.io/apimachinery/pkg/util/wait"
 	pkglogging "knative.dev/pkg/logging"
 	ltesting "knative.dev/pkg/logging/testing"
 
@@ -39,20 +38,20 @@ func TestConcurrencyStateHandler(t *testing.T) {
 	h := ConcurrencyStateHandler(logger, http.HandlerFunc(handler), func() { paused.Inc() }, func() { resumed.Inc() })
 
 	h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "http://target", nil))
-	if got, want := pollFor(paused, 1), int64(1); got != want {
+	if got, want := paused.Load(), int64(1); got != want {
 		t.Errorf("Pause was called %d times, want %d times", got, want)
 	}
 
-	if got, want := pollFor(resumed, 1), int64(1); got != want {
+	if got, want := resumed.Load(), int64(1); got != want {
 		t.Errorf("Resume was called %d times, want %d times", got, want)
 	}
 
 	h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "http://target", nil))
-	if got, want := pollFor(paused, 2), int64(2); got != want {
+	if got, want := paused.Load(), int64(2); got != want {
 		t.Errorf("Pause was called %d times, want %d times", got, want)
 	}
 
-	if got, want := pollFor(resumed, 2), int64(2); got != want {
+	if got, want := resumed.Load(), int64(2); got != want {
 		t.Errorf("Resume was called %d times, want %d times", got, want)
 	}
 }
@@ -84,11 +83,11 @@ func TestConcurrencyStateHandlerParallelSubsumed(t *testing.T) {
 	<-req1 // Allow req1 to pass.
 	<-req1 // Wait for req1 to finish.
 
-	if got, want := pollFor(paused, 1), int64(1); got != want {
+	if got, want := paused.Load(), int64(1); got != want {
 		t.Errorf("Pause was called %d times, want %d times", got, want)
 	}
 
-	if got, want := pollFor(resumed, 1), int64(1); got != want {
+	if got, want := resumed.Load(), int64(1); got != want {
 		t.Errorf("Resume was called %d times, want %d times", got, want)
 	}
 }
@@ -131,11 +130,11 @@ func TestConcurrencyStateHandlerParallelOverlapping(t *testing.T) {
 	<-req2 // Allow req2 to pass.
 	<-req2 // Wait for req2 to finish.
 
-	if got, want := pollFor(paused, 1), int64(1); got != want {
+	if got, want := paused.Load(), int64(1); got != want {
 		t.Errorf("Pause was called %d times, want %d times", got, want)
 	}
 
-	if got, want := pollFor(resumed, 1), int64(1); got != want {
+	if got, want := resumed.Load(), int64(1); got != want {
 		t.Errorf("Resume was called %d times, want %d times", got, want)
 	}
 }
@@ -205,12 +204,3 @@ func BenchmarkConcurrencyStateProxyHandler(b *testing.B) {
 		reportTicker.Stop()
 	}
 }
-
-func pollFor(val *atomic.Int64, want int64) int64 {
-	var lastVal int64
-	wait.PollImmediate(1*time.Millisecond, 1*time.Second, func() (bool, error) {
-		lastVal = val.Load()
-		return lastVal == want, nil
-	})
-	return lastVal
-}
diff --git a/pkg/reconciler/revision/background_test.go b/pkg/reconciler/revision/background_test.go
index 5a04f06b9..c584b3fc2 100644
--- a/pkg/reconciler/revision/background_test.go
+++ b/pkg/reconciler/revision/background_test.go
@@ -148,7 +148,8 @@ func TestResolveInBackground(t *testing.T) {
 
 			for i := 0; i < 2; i++ {
 				t.Run(fmt.Sprint("iteration", i), func(t *testing.T) {
-					statuses, err := subject.Resolve(fakeRevision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), timeout)
+					logger := logtesting.TestLogger(t)
+					statuses, err := subject.Resolve(logger, fakeRevision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), timeout)
 					if err != nil || statuses != nil {
 						// Initial result should be nil, nil since we have nothing in cache.
 						t.Errorf("Resolve() = %v, %v, wanted nil, nil", statuses, err)
@@ -163,7 +164,7 @@ func TestResolveInBackground(t *testing.T) {
 						t.Fatalf("Resolver did not report ready")
 					}
 
-					statuses, err = subject.Resolve(fakeRevision, k8schain.Options{}, nil, timeout)
+					statuses, err = subject.Resolve(logger, fakeRevision, k8schain.Options{}, nil, timeout)
 					if got, want := err, tt.wantError; !errors.Is(got, want) {
 						t.Errorf("Resolve() = _, %q, wanted %q", got, want)
 					}
@@ -202,7 +203,7 @@ func TestRateLimitGlobal(t *testing.T) {
 
 	for i := 0; i < 5; i++ {
 		name := fmt.Sprint("rev", i)
-		subject.Resolve(rev(name, name+"img1", name+"img2"), k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
+		subject.Resolve(logger, rev(name, name+"img1", name+"img2"), k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
 	}
 
 	// Rate limit base rate in this test is 1 second, so this should give time for at most 1 resolve.
@@ -244,14 +245,14 @@ func TestRateLimitPerItem(t *testing.T) {
 	for i := 0; i < 3; i++ {
 		subject.Clear(types.NamespacedName{Name: revision.Name, Namespace: revision.Namespace})
 		start := time.Now()
-		resolution, err := subject.Resolve(revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
+		resolution, err := subject.Resolve(logger, revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
 		if err != nil || resolution != nil {
 			t.Fatalf("Expected Resolve to be nil, nil but got %v, %v", resolution, err)
 		}
 
 		<-enqueue
 
-		_, err = subject.Resolve(revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
+		_, err = subject.Resolve(logger, revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
 		if err == nil {
 			t.Fatalf("Expected Resolve to fail")
 		}
@@ -265,7 +266,7 @@ func TestRateLimitPerItem(t *testing.T) {
 
 	t.Run("Does not affect other revisions", func(t *testing.T) {
 		start := time.Now()
-		resolution, err := subject.Resolve(rev("another-revision", "img1", "img2"), k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
+		resolution, err := subject.Resolve(logger, rev("another-revision", "img1", "img2"), k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
 		if err != nil || resolution != nil {
 			t.Fatalf("Expected Resolve to be nil, nil but got %v, %v", resolution, err)
 		}
@@ -280,7 +281,7 @@ func TestRateLimitPerItem(t *testing.T) {
 		subject.Forget(types.NamespacedName{Name: revision.Name, Namespace: revision.Namespace})
 
 		start := time.Now()
-		resolution, err := subject.Resolve(revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
+		resolution, err := subject.Resolve(logger, revision, k8schain.Options{ServiceAccountName: "san"}, sets.NewString("skip"), 0)
 		if err != nil || resolution != nil {
 			t.Fatalf("Expected Resolve to be nil, nil but got %v, %v", resolution, err)
 		}
diff --git a/pkg/reconciler/revision/revision_test.go b/pkg/reconciler/revision/revision_test.go
index 1e60f5bdb..57e737031 100644
--- a/pkg/reconciler/revision/revision_test.go
+++ b/pkg/reconciler/revision/revision_test.go
@@ -25,6 +25,7 @@ import (
 	"time"
 
 	// Inject the fakes for informers this controller relies on.
+	"go.uber.org/zap"
 	fakecachingclient "knative.dev/caching/pkg/client/injection/client/fake"
 	fakeimageinformer "knative.dev/caching/pkg/client/injection/informers/caching/v1alpha1/image/fake"
 	fakekubeclient "knative.dev/pkg/client/injection/kube/client/fake"
@@ -228,7 +229,7 @@ func addResourcesToInformers(t *testing.T, ctx context.Context, rev *v1.Revision
 
 type nopResolver struct{}
 
-func (r *nopResolver) Resolve(rev *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
+func (r *nopResolver) Resolve(_ *zap.SugaredLogger, rev *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
 	return []v1.ContainerStatus{{
 		Name: rev.Spec.Containers[0].Name,
 	}}, nil
@@ -325,7 +326,7 @@ func testDefaultsCM() *corev1.ConfigMap {
 
 type notResolvedYetResolver struct{}
 
-func (r *notResolvedYetResolver) Resolve(_ *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
+func (r *notResolvedYetResolver) Resolve(_ *zap.SugaredLogger, _ *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
 	return nil, nil
 }
 
@@ -337,7 +338,7 @@ type errorResolver struct {
 	cleared bool
 }
 
-func (r *errorResolver) Resolve(_ *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
+func (r *errorResolver) Resolve(_ *zap.SugaredLogger, _ *v1.Revision, _ k8schain.Options, _ sets.String, _ time.Duration) ([]v1.ContainerStatus, error) {
 	return nil, r.err
 }
 
diff --git a/pkg/reconciler/route/table_test.go b/pkg/reconciler/route/table_test.go
index 7b88f0873..44b8756da 100644
--- a/pkg/reconciler/route/table_test.go
+++ b/pkg/reconciler/route/table_test.go
@@ -2583,8 +2583,145 @@ func TestReconcileEnableAutoTLS(t *testing.T) {
 		WantEvents: []string{
 			Eventf(corev1.EventTypeNormal, "Created", "Created placeholder service %q", "becomes-ready"),
 			Eventf(corev1.EventTypeNormal, "Updated", "Updated Spec for Certificate %s/%s", "default", "route-12-34"),
+			Eventf(corev1.EventTypeNormal, "Deleted", "Deleted orphaned Knative Certificate %s/%s", "default", "route-12-34"),
 			Eventf(corev1.EventTypeNormal, "Created", "Created Ingress %q", "becomes-ready"),
 		},
+		WantDeletes: []clientgotesting.DeleteActionImpl{{
+			// This certificate's DNS name is not the host name needed by the input Route.
+			ActionImpl: clientgotesting.ActionImpl{
+				Namespace: "default",
+				Verb:      "delete",
+				Resource:  netv1alpha1.SchemeGroupVersion.WithResource("certificates"),
+			},
+			Name: "route-12-34",
+		}},
+		Key: "default/becomes-ready",
+	}, {
+		Name: "check that unused certificate removal works as intended",
+		Objects: []runtime.Object{
+			Route("default", "becomes-ready", WithConfigTarget("config"), WithRouteUID("12-34")),
+			cfg("default", "config",
+				WithConfigGeneration(1), WithLatestCreated("config-00001"), WithLatestReady("config-00001")),
+			rev("default", "config", 1, MarkRevisionReady, WithRevName("config-00001")),
+			&netv1alpha1.Certificate{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "route-12-34",
+					Namespace: "default",
+					OwnerReferences: []metav1.OwnerReference{*kmeta.NewControllerRef(
+						Route("default", "becomes-ready", WithConfigTarget("config"), WithRouteUID("12-34")))},
+					Labels: map[string]string{
+						serving.RouteLabelKey: "becomes-ready",
+					},
+					Annotations: map[string]string{
+						networking.CertificateClassAnnotationKey: network.CertManagerCertificateClassName,
+					},
+				},
+				Spec: netv1alpha1.CertificateSpec{
+					DNSNames: []string{"abc.test.example.com", "becomes-ready.default.example.com"},
+				},
+				Status: readyCertStatus(),
+			},
+			// This certificate is not required; the route will use the certificate named "route-12-34".
+			// This certificate has a different DNSNames order to test the unused certificate deletion.
+			&netv1alpha1.Certificate{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "route-12-34-keep",
+					Namespace: "default",
+					OwnerReferences: []metav1.OwnerReference{*kmeta.NewControllerRef(
+						Route("default", "becomes-ready", WithConfigTarget("config"), WithRouteUID("12-34")))},
+					Labels: map[string]string{
+						serving.RouteLabelKey: "becomes-ready",
+					},
+					Annotations: map[string]string{
+						networking.CertificateClassAnnotationKey: network.CertManagerCertificateClassName,
+					},
+				},
+				Spec: netv1alpha1.CertificateSpec{
+					DNSNames: []string{"becomes-ready.default.example.com", "abc.test.example.com"},
+				},
+				Status: readyCertStatus(),
+			},
+			// MakeCertificates will create a certificate with DNS name "abc.test.example.com" which is not the host name
+			// needed by the input Route.
+			&netv1alpha1.Certificate{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "route-12-34-unused",
+					Namespace: "default",
+					OwnerReferences: []metav1.OwnerReference{*kmeta.NewControllerRef(
+						Route("default", "becomes-ready", WithConfigTarget("config"), WithRouteUID("12-34")))},
+					Labels: map[string]string{
+						serving.RouteLabelKey: "becomes-ready",
+					},
+					Annotations: map[string]string{
+						networking.CertificateClassAnnotationKey: network.CertManagerCertificateClassName,
+					},
+				},
+				Spec: netv1alpha1.CertificateSpec{
+					DNSNames: []string{"abc.test.example.com"},
+				},
+				Status: readyCertStatus(),
+			},
+		},
+		WantCreates: []runtime.Object{
+			ingressWithTLS(
+				Route("default", "becomes-ready", WithConfigTarget("config"), WithURL,
+					WithRouteUID("12-34")),
+				&traffic.Config{
+					Targets: map[string]traffic.RevisionTargets{
+						traffic.DefaultTarget: {{
+							TrafficTarget: v1.TrafficTarget{
+								ConfigurationName: "config",
+								LatestRevision:    ptr.Bool(true),
+								RevisionName:      "config-00001",
+								Percent:           ptr.Int64(100),
+							},
+						}},
+					},
+				},
+				[]netv1alpha1.IngressTLS{{
+					Hosts:           []string{"becomes-ready.default.example.com"},
+					SecretName:      "route-12-34",
+					SecretNamespace: "default",
+				}},
+				nil,
+			),
+			simpleK8sService(
+				Route("default", "becomes-ready", WithConfigTarget("config"), WithRouteUID("12-34")),
+				WithExternalName("becomes-ready.default.example.com"),
+			),
+		},
+		WantUpdates: []clientgotesting.UpdateActionImpl{{
+			Object: certificateWithStatus(resources.MakeCertificates(Route("default", "becomes-ready", WithConfigTarget("config"), WithURL, WithRouteUID("12-34")),
+				map[string]string{"becomes-ready.default.example.com": ""}, network.CertManagerCertificateClassName)[0], readyCertStatus()),
+		}},
+		WantStatusUpdates: []clientgotesting.UpdateActionImpl{{
+			Object: Route("default", "becomes-ready", WithConfigTarget("config"),
+				WithRouteUID("12-34"),
+				WithAddress, WithInitRouteConditions,
+				MarkTrafficAssigned, MarkIngressNotConfigured, WithStatusTraffic(
+					v1.TrafficTarget{
+						RevisionName:   "config-00001",
+						Percent:        ptr.Int64(100),
+						LatestRevision: ptr.Bool(true),
+					}), MarkCertificateReady,
+				// The certificate is ready. So we want to have HTTPS URL.
+				WithHTTPSDomain),
+		}},
+		WantEvents: []string{
+			Eventf(corev1.EventTypeNormal, "Created", "Created placeholder service %q", "becomes-ready"),
+			Eventf(corev1.EventTypeNormal, "Updated", "Updated Spec for Certificate %s/%s", "default", "route-12-34"),
+			Eventf(corev1.EventTypeNormal, "Deleted", "Deleted orphaned Knative Certificate %s/%s", "default", "route-12-34-unused"),
+			Eventf(corev1.EventTypeNormal, "Created", "Created Ingress %q", "becomes-ready"),
+		},
+		WantDeletes: []clientgotesting.DeleteActionImpl{{
+			// This certificate's DNS name is not the host name needed by the input Route.
+			ActionImpl: clientgotesting.ActionImpl{
+				Namespace: "default",
+				Verb:      "delete",
+				Resource:  netv1alpha1.SchemeGroupVersion.WithResource("certificates"),
+			},
+			Name: "route-12-34-unused",
+		}},
 		Key: "default/becomes-ready",
 	}, {
 		Name: "verify ingress rules created for http01 challenges",
@@ -2796,8 +2933,18 @@ func TestReconcileEnableAutoTLS(t *testing.T) {
 		WantEvents: []string{
 			Eventf(corev1.EventTypeNormal, "Created", "Created placeholder service %q", "becomes-ready"),
 			Eventf(corev1.EventTypeNormal, "Updated", "Updated Spec for Certificate %s/%s", "default", "route-12-34"),
+			Eventf(corev1.EventTypeNormal, "Deleted", "Deleted orphaned Knative Certificate %s/%s", "default", "route-12-34"),
 			Eventf(corev1.EventTypeNormal, "Created", "Created Ingress %q", "becomes-ready"),
 		},
+		WantDeletes: []clientgotesting.DeleteActionImpl{{
+			// This certificate's DNS name is not the host name needed by the input Route.
+			ActionImpl: clientgotesting.ActionImpl{
+				Namespace: "default",
+				Verb:      "delete",
+				Resource:  netv1alpha1.SchemeGroupVersion.WithResource("certificates"),
+			},
+			Name: "route-12-34",
+		}},
 		Key: "default/becomes-ready",
 	}, {
 		// This test is a same with "public becomes cluster local" above, but confirm it does not create certs with autoTLS for cluster-local.
diff --git a/test/conformance/api/v1/resources_test.go b/test/conformance/api/v1/resources_test.go
index 7f35686c1..023730202 100644
--- a/test/conformance/api/v1/resources_test.go
+++ b/test/conformance/api/v1/resources_test.go
@@ -114,7 +114,12 @@ func TestCustomResourcesLimits(t *testing.T) {
 		t.Fatalf("Didn't get a response from bloating cow with %d MBs of Memory: %v", 200, err)
 	}
 
-	if err := pokeCowForMB(500); err == nil {
-		t.Fatalf("We shouldn't have got a response from bloating cow with %d MBs of Memory: %v", 500, err)
+	// ExceedingMemoryLimitSize defaults to 500.
+	// Allows override the memory usage to get a non-200 resposne because the serverless platform
+	// MAY automatically adjust the resource limits.
+	// See https://github.com/knative/specs/blob/main/specs/serving/runtime-contract.md#memory-and-cpu-limits
+	exceedingMemory := test.ServingFlags.ExceedingMemoryLimitSize
+	if err := pokeCowForMB(exceedingMemory); err == nil {
+		t.Fatalf("We shouldn't have got a response from bloating cow with %d MBs of Memory: %v", exceedingMemory, err)
 	}
 }
diff --git a/test/e2e_flags.go b/test/e2e_flags.go
index 69154d6ca..73214d7d8 100644
--- a/test/e2e_flags.go
+++ b/test/e2e_flags.go
@@ -31,18 +31,19 @@ var ServingFlags = initializeServingFlags()
 
 // ServingEnvironmentFlags holds the e2e flags needed only by the serving repo.
 type ServingEnvironmentFlags struct {
-	ResolvableDomain    bool   // Resolve Route controller's `domainSuffix`
-	CustomDomain        string // Indicates the `domainSuffix` for custom domain test.
-	HTTPS               bool   // Indicates where the test service will be created with https
-	Buckets             int    // The number of reconciler buckets configured.
-	Replicas            int    // The number of controlplane replicas being run.
-	EnableAlphaFeatures bool   // Indicates whether we run tests for alpha features
-	EnableBetaFeatures  bool   // Indicates whether we run tests for beta features
-	DisableLogStream    bool   // Indicates whether log streaming is disabled
-	DisableOptionalAPI  bool   // Indicates whether to skip conformance tests against optional API
-	TestNamespace       string // Default namespace for Serving E2E/Conformance tests
-	AltTestNamespace    string // Alternative namespace for running cross-namespace tests in
-	TLSTestNamespace    string // Namespace for Serving TLS tests
+	ResolvableDomain         bool   // Resolve Route controller's `domainSuffix`
+	CustomDomain             string // Indicates the `domainSuffix` for custom domain test.
+	HTTPS                    bool   // Indicates where the test service will be created with https
+	Buckets                  int    // The number of reconciler buckets configured.
+	Replicas                 int    // The number of controlplane replicas being run.
+	EnableAlphaFeatures      bool   // Indicates whether we run tests for alpha features
+	EnableBetaFeatures       bool   // Indicates whether we run tests for beta features
+	DisableLogStream         bool   // Indicates whether log streaming is disabled
+	DisableOptionalAPI       bool   // Indicates whether to skip conformance tests against optional API
+	TestNamespace            string // Default namespace for Serving E2E/Conformance tests
+	AltTestNamespace         string // Alternative namespace for running cross-namespace tests in
+	TLSTestNamespace         string // Namespace for Serving TLS tests
+	ExceedingMemoryLimitSize int    // Memory size used to trigger a non-200 response when the service is set with 300MB memory limit.
 }
 
 func initializeServingFlags() *ServingEnvironmentFlags {
@@ -84,5 +85,9 @@ func initializeServingFlags() *ServingEnvironmentFlags {
 	flag.StringVar(&f.TLSTestNamespace, "tls-test-namespace", "tls",
 		"Set this flag to change the namespace for running TLS tests.")
 
+	flag.IntVar(&f.ExceedingMemoryLimitSize, "exceeding-memory-limit-size", 500,
+		"Set this flag to the MB of memory consumed by your service in resource limit tests. "+
+			"You service is set with 300 MB memory limit and shoud return a non-200 response when consuming such amount of memory.")
+
 	return &f
 }
diff --git a/test/v1/service.go b/test/v1/service.go
index 65a761f57..2e615e408 100644
--- a/test/v1/service.go
+++ b/test/v1/service.go
@@ -224,9 +224,7 @@ func WaitForServiceLatestRevision(clients *test.Clients, names test.ResourceName
 	if err := WaitForServiceState(clients.ServingClient, names.Service, func(s *v1.Service) (bool, error) {
 		if s.Status.LatestCreatedRevisionName != names.Revision {
 			revisionName = s.Status.LatestCreatedRevisionName
-			// Without this it might happen that the latest created revision is later overridden by a newer one
-			// and the following check for LatestReadyRevisionName would fail.
-			return CheckRevisionState(clients.ServingClient, revisionName, IsRevisionRoutingActive) == nil, nil
+			return true, nil
 		}
 		return false, nil
 	}, "ServiceUpdatedWithRevision"); err != nil {
diff --git a/third_party/cert-manager-latest/net-certmanager.yaml b/third_party/cert-manager-latest/net-certmanager.yaml
index 330118649..2e1eb28b1 100644
--- a/third_party/cert-manager-latest/net-certmanager.yaml
+++ b/third_party/cert-manager-latest/net-certmanager.yaml
@@ -18,7 +18,7 @@ metadata:
   # These are the permissions needed by the `cert-manager` `Certificate` implementation.
   name: knative-serving-certmanager
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     serving.knative.dev/controller: "true"
     networking.knative.dev/certificate-provider: cert-manager
 rules:
@@ -49,7 +49,7 @@ kind: ValidatingWebhookConfiguration
 metadata:
   name: config.webhook.net-certmanager.networking.internal.knative.dev
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 webhooks:
   - admissionReviewVersions:
@@ -88,7 +88,7 @@ metadata:
   name: net-certmanager-webhook-certs
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 
 ---
@@ -112,7 +112,7 @@ metadata:
   name: config-certmanager
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 data:
   _example: |
@@ -159,7 +159,7 @@ metadata:
   name: net-certmanager-controller
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 spec:
   selector:
@@ -171,14 +171,14 @@ spec:
         cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
       labels:
         app: net-certmanager-controller
-        serving.knative.dev/release: "v20210908-dd2ff643"
+        serving.knative.dev/release: "v20210910-58d9cad4"
     spec:
       serviceAccountName: controller
       containers:
         - name: controller
           # This is the Go import path for the binary that is containerized
           # and substituted here.
-          image: gcr.io/knative-nightly/knative.dev/net-certmanager/cmd/controller@sha256:af23e53aa6f0cc54e9ec78d751c6a35cdb23855287840d3fa287c15619727ee7
+          image: gcr.io/knative-nightly/knative.dev/net-certmanager/cmd/controller@sha256:f127be0e080bf9b95b0c37ae10ea4b685a10ab87e98ee8d2abff7108b1f74f6b
           resources:
             requests:
               cpu: 30m
@@ -216,7 +216,7 @@ kind: Service
 metadata:
   labels:
     app: net-certmanager-controller
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
   name: net-certmanager-controller
   namespace: knative-serving
@@ -253,7 +253,7 @@ metadata:
   name: net-certmanager-webhook
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 spec:
   selector:
@@ -267,14 +267,14 @@ spec:
       labels:
         app: net-certmanager-webhook
         role: net-certmanager-webhook
-        serving.knative.dev/release: "v20210908-dd2ff643"
+        serving.knative.dev/release: "v20210910-58d9cad4"
     spec:
       serviceAccountName: controller
       containers:
         - name: webhook
           # This is the Go import path for the binary that is containerized
           # and substituted here.
-          image: gcr.io/knative-nightly/knative.dev/net-certmanager/cmd/webhook@sha256:61883f82fe743dd7a2f044550640d6ac31da696da9e3f08a371b55b7d61d0928
+          image: gcr.io/knative-nightly/knative.dev/net-certmanager/cmd/webhook@sha256:97ce5cb8a8ecc800ed4135adf062c986298ae82afd34ed329a17fe8f526d88ce
           resources:
             requests:
               cpu: 20m
@@ -333,7 +333,7 @@ metadata:
   namespace: knative-serving
   labels:
     role: net-certmanager-webhook
-    serving.knative.dev/release: "v20210908-dd2ff643"
+    serving.knative.dev/release: "v20210910-58d9cad4"
     networking.knative.dev/certificate-provider: cert-manager
 spec:
   ports:
diff --git a/third_party/contour-latest/contour.yaml b/third_party/contour-latest/contour.yaml
index e2cc30313..bc2b9b283 100644
--- a/third_party/contour-latest/contour.yaml
+++ b/third_party/contour-latest/contour.yaml
@@ -1904,7 +1904,7 @@ spec:
     spec:
       containers:
         - name: contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           command:
             - contour
@@ -2155,7 +2155,7 @@ spec:
             - --config-path=/config/contour.yaml
           command:
             - contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           name: contour
           ports:
@@ -2244,7 +2244,7 @@ spec:
           args:
             - envoy
             - shutdown-manager
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           lifecycle:
             preStop:
@@ -2326,7 +2326,7 @@ spec:
             - --envoy-key-file=/certs/tls.key
           command:
             - contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           name: envoy-initconfig
           volumeMounts:
@@ -2605,7 +2605,7 @@ spec:
     spec:
       containers:
         - name: contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           command:
             - contour
@@ -2717,7 +2717,7 @@ spec:
             - --config-path=/config/contour.yaml
           command:
             - contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           name: contour
           ports:
@@ -2806,7 +2806,7 @@ spec:
           args:
             - envoy
             - shutdown-manager
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           lifecycle:
             preStop:
@@ -2888,7 +2888,7 @@ spec:
             - --envoy-key-file=/certs/tls.key
           command:
             - contour
-          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:4421d04bd44212495295c0ab744b6b912f9beb985b5bee1ffc56741cadee1af0
+          image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:cfa94afa242cde3247bc2f5be0371e3043b6c06b1419799a3685b83fa696232b
           imagePullPolicy: IfNotPresent
           name: envoy-initconfig
           volumeMounts:
diff --git a/third_party/contour-latest/net-contour.yaml b/third_party/contour-latest/net-contour.yaml
index 7b553522e..bcc3abbee 100644
--- a/third_party/contour-latest/net-contour.yaml
+++ b/third_party/contour-latest/net-contour.yaml
@@ -33,7 +33,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: contour
-    serving.knative.dev/release: "v20210908-2aa41854"
+    serving.knative.dev/release: "v20210910-5b0e8112"
 data:
   _example: |
     ################################
@@ -103,7 +103,7 @@ spec:
         - name: controller
           # This is the Go import path for the binary that is containerized
           # and substituted here.
-          image: gcr.io/knative-nightly/knative.dev/net-contour/cmd/controller@sha256:edf9301ee0be34bc422b492faf3ba66b7e066b4230bf2249fd0c331f203d9820
+          image: gcr.io/knative-nightly/knative.dev/net-contour/cmd/controller@sha256:55d192281ccfe727d19ea2b172364ca6b1e9cb0fce0145f93753a2989477da3c
           resources:
             requests:
               cpu: 40m
diff --git a/third_party/istio-latest/net-istio.yaml b/third_party/istio-latest/net-istio.yaml
index 5d7c4b18f..806406225 100644
--- a/third_party/istio-latest/net-istio.yaml
+++ b/third_party/istio-latest/net-istio.yaml
@@ -1,4 +1,4 @@
-# Generated when HEAD was 6793a7353f4c20c3d912a86280d9eef7b80932a4
+# Generated when HEAD was 2b9152729cc8cd2392e78f9a7f3a2b479f089a54
 #
 # Copyright 2019 The Knative Authors
 #
@@ -20,7 +20,7 @@ metadata:
   # These are the permissions needed by the Istio Ingress implementation.
   name: knative-serving-istio
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     serving.knative.dev/controller: "true"
     networking.knative.dev/ingress-provider: istio
 rules:
@@ -50,7 +50,7 @@ metadata:
   name: knative-ingress-gateway
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -87,7 +87,7 @@ metadata:
   name: knative-local-gateway
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -106,7 +106,7 @@ metadata:
   name: knative-local-gateway
   namespace: istio-system
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
     experimental.istio.io/disable-gateway-port-translation: "true"
 spec:
@@ -139,7 +139,7 @@ metadata:
   name: config-istio
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 data:
   _example: |
@@ -200,7 +200,7 @@ metadata:
   name: "webhook"
   namespace: "knative-serving"
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -216,7 +216,7 @@ metadata:
   name: "domainmapping-webhook"
   namespace: "knative-serving"
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -232,7 +232,7 @@ metadata:
   name: "net-istio-webhook"
   namespace: "knative-serving"
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -263,7 +263,7 @@ metadata:
   name: net-istio-controller
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -279,14 +279,14 @@ spec:
         sidecar.istio.io/inject: "false"
       labels:
         app: net-istio-controller
-        serving.knative.dev/release: "v20210908-6793a735"
+        serving.knative.dev/release: "v20210910-2b915272"
     spec:
       serviceAccountName: controller
       containers:
         - name: controller
           # This is the Go import path for the binary that is containerized
           # and substituted here.
-          image: gcr.io/knative-nightly/knative.dev/net-istio/cmd/controller@sha256:2f398fac34d00c0db1ff7212fd226cd7b315e726def887436600516dc7dc41a2
+          image: gcr.io/knative-nightly/knative.dev/net-istio/cmd/controller@sha256:94cf24beb5aeafd3b9b691e94315b80a536623a10df631ce768e48a63e761e76
           resources:
             requests:
               cpu: 30m
@@ -343,7 +343,7 @@ metadata:
   name: net-istio-webhook
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   selector:
@@ -357,14 +357,14 @@ spec:
       labels:
         app: net-istio-webhook
         role: net-istio-webhook
-        serving.knative.dev/release: "v20210908-6793a735"
+        serving.knative.dev/release: "v20210910-2b915272"
     spec:
       serviceAccountName: controller
       containers:
         - name: webhook
           # This is the Go import path for the binary that is containerized
           # and substituted here.
-          image: gcr.io/knative-nightly/knative.dev/net-istio/cmd/webhook@sha256:63bd5e0bdff6df2e05c5d04d2638fc2b56bb229246be8c569bc7edb9a2466e74
+          image: gcr.io/knative-nightly/knative.dev/net-istio/cmd/webhook@sha256:c67d637dfa826600c99e8f455be189dc09ccc7789815d7763f215b8108496e0b
           resources:
             requests:
               cpu: 20m
@@ -417,7 +417,7 @@ metadata:
   name: net-istio-webhook-certs
   namespace: knative-serving
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 
 ---
@@ -442,7 +442,7 @@ metadata:
   namespace: knative-serving
   labels:
     role: net-istio-webhook
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 spec:
   ports:
@@ -479,7 +479,7 @@ kind: MutatingWebhookConfiguration
 metadata:
   name: webhook.istio.networking.internal.knative.dev
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 webhooks:
   - admissionReviewVersions:
@@ -516,7 +516,7 @@ kind: ValidatingWebhookConfiguration
 metadata:
   name: config.webhook.istio.networking.internal.knative.dev
   labels:
-    serving.knative.dev/release: "v20210908-6793a735"
+    serving.knative.dev/release: "v20210910-2b915272"
     networking.knative.dev/ingress-provider: istio
 webhooks:
   - admissionReviewVersions:
diff --git a/third_party/kourier-latest/kourier.yaml b/third_party/kourier-latest/kourier.yaml
index ace6fbaff..0b60ad908 100644
--- a/third_party/kourier-latest/kourier.yaml
+++ b/third_party/kourier-latest/kourier.yaml
@@ -18,7 +18,7 @@ metadata:
   name: kourier-system
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 
 ---
 # Copyright 2020 The Knative Authors
@@ -42,7 +42,7 @@ metadata:
   namespace: kourier-system
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 data:
   envoy-bootstrap.yaml: |
     dynamic_resources:
@@ -145,7 +145,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 data:
   _example: |
     ################################
@@ -190,7 +190,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
@@ -199,7 +199,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 rules:
   - apiGroups: [""]
     resources: ["events"]
@@ -226,7 +226,7 @@ metadata:
   name: net-kourier
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
@@ -258,7 +258,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 spec:
   replicas: 1
   selector:
@@ -270,7 +270,7 @@ spec:
         app: net-kourier-controller
     spec:
       containers:
-        - image: gcr.io/knative-nightly/knative.dev/net-kourier/cmd/kourier@sha256:4939ab1b75c42ad179f835ed89ea33808c4bec07687361134c35c21212a8b700
+        - image: gcr.io/knative-nightly/knative.dev/net-kourier/cmd/kourier@sha256:8484c1473a43fa91308506d265dd883cbe1a63785036e4c466aa301f2e4c83cf
           name: controller
           env:
             - name: CERTS_SECRET_NAMESPACE
@@ -309,7 +309,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 spec:
   ports:
     - name: grpc-xds
@@ -342,7 +342,7 @@ metadata:
   namespace: kourier-system
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 spec:
   strategy:
     type: RollingUpdate
@@ -417,7 +417,7 @@ metadata:
   namespace: kourier-system
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 spec:
   ports:
     - name: http2
@@ -439,7 +439,7 @@ metadata:
   namespace: kourier-system
   labels:
     networking.knative.dev/ingress-provider: kourier
-    serving.knative.dev/release: "v20210908-1f4223a8"
+    serving.knative.dev/release: "v20210909-5c40d3d0"
 spec:
   ports:
     - name: http2