Repository: knative/serving
Pull request #9027 (2020-08-12 09:23:57)

Allow configuring activator max idle connections
Inject the transport into activatorHandler and allow configuring maxIdleConns using a flag. As a bonus, this means the tests for activatorHandler no longer need to do casts and set the private transport field \o/.

Using a flag rather than a configmap here because maxIdleConns can't be easily changed dynamically, so we really need the value on startup.

<!--
Request Prow to automatically lint any go code in this PR:

/lint
-->

/assign @markusthoemmes @vagababov 

knative-prow-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/serving/pull/9027#" title="Author self-approved">julz</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **jrbancel**
You can assign the PR to them by writing `/assign @jrbancel` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

<details open>
Needs approval from an approver in each of these files:

- **[cmd/activator/OWNERS](https://github.com/knative/serving/blob/master/cmd/activator/OWNERS)**
- ~~[pkg/activator/OWNERS](https://github.com/knative/serving/blob/master/pkg/activator/OWNERS)~~ [julz]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jrbancel"]} -->
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-serving-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/activator/handler/handler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-go-coverage/1293478451973459971/artifacts/line-cov.html#file0) | 96.9% | 96.8% | -0.1

julz: (looks like cmd/activator isn't owned by same owners as pkg/activator, seems wrong! Opened https://github.com/knative/serving/pull/9028).
knative-prow-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/serving/pull/9027#" title="Author self-approved">julz</a>*, *<a href="https://github.com/knative/serving/pull/9027#pullrequestreview-465752020" title="Approved">markusthoemmes</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[cmd/activator/OWNERS](https://github.com/knative/serving/blob/master/cmd/activator/OWNERS)~~ [markusthoemmes]
- ~~[pkg/activator/OWNERS](https://github.com/knative/serving/blob/master/pkg/activator/OWNERS)~~ [julz,markusthoemmes]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
knative-prow-robot: @julz: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-autotls-tests | 7b86df7fbcff40374f216d800aa33cdcb5e10c76 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-autotls-tests/1293478451973459973) | `/test pull-knative-serving-autotls-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/9027). [Your PR dashboard](https://gubernator.knative.dev/pr/julz).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-test-reporter-robot: <!--[AUTOMATED-FLAKY-RETRYER]7b86df7fbcff40374f216d800aa33cdcb5e10c76[AUTOMATED-FLAKY-RETRYER]-->
The following jobs failed:

Test name | Triggers | Retries
--- | --- | ---
pull-knative-serving-autotls-tests |  | 0/3

Failed non-flaky tests preventing automatic retry of pull-knative-serving-autotls-tests:

```
test/e2e/autotls.TestTls/http01
test/e2e/autotls.TestTls
```
knative-prow-robot: @julz: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-autotls-tests | 7b86df7fbcff40374f216d800aa33cdcb5e10c76 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-autotls-tests/1293478451973459973) | `/test pull-knative-serving-autotls-tests`
pull-knative-serving-integration-tests | 7b86df7fbcff40374f216d800aa33cdcb5e10c76 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-integration-tests/1293478451973459968) | `/test pull-knative-serving-integration-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/9027). [Your PR dashboard](https://gubernator.knative.dev/pr/julz).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-test-reporter-robot: <!--[AUTOMATED-FLAKY-RETRYER]7b86df7fbcff40374f216d800aa33cdcb5e10c76[AUTOMATED-FLAKY-RETRYER]-->
The following jobs failed:

Test name | Triggers | Retries
--- | --- | ---
pull-knative-serving-autotls-tests |  | 0/3
pull-knative-serving-integration-tests | [2020-08-12 10:16:29.521 +0000 UTC](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-integration-tests/1293478451973459968) | 1/3

Automatically retrying due to test flakiness...
/test pull-knative-serving-integration-tests
knative-prow-robot: @julz: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-autotls-tests | 7b86df7fbcff40374f216d800aa33cdcb5e10c76 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-autotls-tests/1293478451973459973) | `/test pull-knative-serving-autotls-tests`
pull-knative-serving-integration-tests | 7b86df7fbcff40374f216d800aa33cdcb5e10c76 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-integration-tests/1293491915341172736) | `/test pull-knative-serving-integration-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/9027). [Your PR dashboard](https://gubernator.knative.dev/pr/julz).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-test-reporter-robot: <!--[AUTOMATED-FLAKY-RETRYER]7b86df7fbcff40374f216d800aa33cdcb5e10c76[AUTOMATED-FLAKY-RETRYER]-->
The following jobs failed:

Test name | Triggers | Retries
--- | --- | ---
pull-knative-serving-autotls-tests |  | 0/3
pull-knative-serving-integration-tests | [2020-08-12 10:16:29.521 +0000 UTC](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-integration-tests/1293478451973459968)<br>[2020-08-12 11:05:29.411 +0000 UTC](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/9027/pull-knative-serving-integration-tests/1293491915341172736) | 2/3

Automatically retrying due to test flakiness...
/test pull-knative-serving-integration-tests

Patch:
diff --git a/cmd/activator/OWNERS b/cmd/activator/OWNERS
index 8787cec26..4a5322417 100644
--- a/cmd/activator/OWNERS
+++ b/cmd/activator/OWNERS
@@ -1,10 +1,13 @@
 # The OWNERS file is used by prow to automatically merge approved PRs.
 
 approvers:
+- autoscaling-approvers
 - networking-approvers
 
 reviewers:
+- autoscaling-reviewers
 - networking-reviewers
 
 labels:
+- area/autoscale
 - area/networking
diff --git a/cmd/activator/main.go b/cmd/activator/main.go
index 40f84a49c..0b2447cfd 100644
--- a/cmd/activator/main.go
+++ b/cmd/activator/main.go
@@ -195,9 +195,17 @@ func main() {
 	concurrencyReporter := activatorhandler.NewConcurrencyReporter(ctx, env.PodName, statCh)
 	go concurrencyReporter.Run(ctx.Done())
 
+	// This is here to allow configuring higher values of keep-alive for larger environments.
+	// TODO: run loadtests using these flags to determine optimal default values.
+	maxIdleProxyConns := intFromEnv(logger, "MAX_IDLE_PROXY_CONNS", 1000)
+	maxIdleProxyConnsPerHost := intFromEnv(logger, "MAX_IDLE_PROXY_CONNS_PER_HOST", 100)
+	logger.Debugf("MaxIdleProxyConns: %d, MaxIdleProxyConnsPerHost: %d", maxIdleProxyConns, maxIdleProxyConnsPerHost)
+
+	proxyTransport := pkgnet.NewAutoTransport(maxIdleProxyConns, maxIdleProxyConnsPerHost)
+
 	// Create activation handler chain
 	// Note: innermost handlers are specified first, ie. the last handler in the chain will be executed first
-	var ah http.Handler = activatorhandler.New(ctx, throttler)
+	var ah http.Handler = activatorhandler.New(ctx, throttler, proxyTransport)
 	ah = concurrencyReporter.Handler(ah)
 	ah = tracing.HTTPSpanMiddleware(ah)
 	ah = configStore.HTTPMiddleware(ah)
@@ -300,3 +308,18 @@ func flush(logger *zap.SugaredLogger) {
 	os.Stderr.Sync()
 	metrics.FlushExporter()
 }
+
+func intFromEnv(logger *zap.SugaredLogger, envName string, defaultValue int) int {
+	env := os.Getenv(envName)
+	if env == "" {
+		return defaultValue
+	}
+
+	parsed, err := strconv.Atoi(env)
+	if err != nil {
+		logger.Warnf("parse %q env var as int: %v", envName, err)
+		return defaultValue
+	}
+
+	return parsed
+}
diff --git a/cmd/queue/main.go b/cmd/queue/main.go
index 2cd95d4e6..5b0d156a7 100644
--- a/cmd/queue/main.go
+++ b/cmd/queue/main.go
@@ -325,8 +325,13 @@ func buildServer(env config, healthState *health.State, rp *readiness.Probe, sta
 		Host:   net.JoinHostPort("127.0.0.1", strconv.Itoa(env.UserPort)),
 	}
 
+	maxIdleConns := 1000 // TODO: somewhat arbitrary value for CC=0, needs experimental validation.
+	if env.ContainerConcurrency > 0 {
+		maxIdleConns = env.ContainerConcurrency
+	}
+
 	httpProxy := httputil.NewSingleHostReverseProxy(target)
-	httpProxy.Transport = buildTransport(env, logger)
+	httpProxy.Transport = buildTransport(env, logger, maxIdleConns)
 	httpProxy.ErrorHandler = pkgnet.ErrorHandler(logger)
 	httpProxy.BufferPool = network.NewBufferPool()
 	httpProxy.FlushInterval = network.FlushInterval
@@ -361,9 +366,12 @@ func buildServer(env config, healthState *health.State, rp *readiness.Probe, sta
 	return pkgnet.NewServer(":"+strconv.Itoa(env.QueueServingPort), composedHandler)
 }
 
-func buildTransport(env config, logger *zap.SugaredLogger) http.RoundTripper {
+func buildTransport(env config, logger *zap.SugaredLogger, maxConns int) http.RoundTripper {
+	// set max-idle and max-idle-per-host to same value since we're always proxying to the same host.
+	transport := pkgnet.NewAutoTransport(maxConns /* max-idle */, maxConns /* max-idle-per-host */)
+
 	if env.TracingConfigBackend == tracingconfig.None {
-		return pkgnet.AutoTransport
+		return transport
 	}
 
 	oct := tracing.NewOpenCensusTracer(tracing.WithExporterFull(env.ServingPod, env.ServingPodIP, logger))
@@ -376,7 +384,7 @@ func buildTransport(env config, logger *zap.SugaredLogger) http.RoundTripper {
 	})
 
 	return &ochttp.Transport{
-		Base:        pkgnet.AutoTransport,
+		Base:        transport,
 		Propagation: tracecontextb3.B3Egress,
 	}
 }
diff --git a/pkg/activator/handler/handler.go b/pkg/activator/handler/handler.go
index 95f6a057c..0e3eeee1a 100644
--- a/pkg/activator/handler/handler.go
+++ b/pkg/activator/handler/handler.go
@@ -52,14 +52,11 @@ type activationHandler struct {
 }
 
 // New constructs a new http.Handler that deals with revision activation.
-func New(ctx context.Context, t Throttler) http.Handler {
-	// TODO: run loadtests to determine the optimal values here.
-	defaultTransport := pkgnet.NewAutoTransport(1000, /*maxidleconnes*/
-		100 /*maxidleperhos*/)
+func New(ctx context.Context, t Throttler, transport http.RoundTripper) http.Handler {
 	return &activationHandler{
-		transport: defaultTransport,
+		transport: transport,
 		tracingTransport: &ochttp.Transport{
-			Base:        defaultTransport,
+			Base:        transport,
 			Propagation: tracecontextb3.B3Egress,
 		},
 		throttler:  t,
diff --git a/pkg/apis/config/store.go b/pkg/apis/config/store.go
index 29e2afab1..172a113eb 100644
--- a/pkg/apis/config/store.go
+++ b/pkg/apis/config/store.go
@@ -101,9 +101,15 @@ func (s *Store) ToContext(ctx context.Context) context.Context {
 
 // Load creates a Config from the current config state of the Store.
 func (s *Store) Load() *Config {
-	return &Config{
-		Defaults:   s.UntypedLoad(DefaultsConfigName).(*Defaults).DeepCopy(),
-		Features:   s.UntypedLoad(FeaturesConfigName).(*Features).DeepCopy(),
-		Autoscaler: s.UntypedLoad(autoscalerconfig.ConfigName).(*autoscalerconfig.Config).DeepCopy(),
+	cfg := &Config{}
+	if def, ok := s.UntypedLoad(DefaultsConfigName).(*Defaults); ok {
+		cfg.Defaults = def.DeepCopy()
 	}
+	if feat, ok := s.UntypedLoad(FeaturesConfigName).(*Features); ok {
+		cfg.Features = feat.DeepCopy()
+	}
+	if as, ok := s.UntypedLoad(autoscalerconfig.ConfigName).(*autoscalerconfig.Config); ok {
+		cfg.Autoscaler = as.DeepCopy()
+	}
+	return cfg
 }
diff --git a/pkg/reconciler/revision/config/store.go b/pkg/reconciler/revision/config/store.go
index 5d29c21ac..def510fd2 100644
--- a/pkg/reconciler/revision/config/store.go
+++ b/pkg/reconciler/revision/config/store.go
@@ -19,13 +19,12 @@ package config
 import (
 	"context"
 
-	"knative.dev/serving/pkg/apis/config"
-
 	network "knative.dev/networking/pkg"
 	"knative.dev/pkg/configmap"
 	"knative.dev/pkg/logging"
 	"knative.dev/pkg/metrics"
 	pkgtracing "knative.dev/pkg/tracing/config"
+	"knative.dev/serving/pkg/apis/config"
 	autoscalerconfig "knative.dev/serving/pkg/autoscaler/config"
 	"knative.dev/serving/pkg/deployment"
 )
@@ -57,6 +56,7 @@ func ToContext(ctx context.Context, c *Config) context.Context {
 // +k8s:deepcopy-gen=false
 type Store struct {
 	*configmap.UntypedStore
+	apiStore *config.Store
 }
 
 // NewStore creates a new store of Configs and optionally calls functions when ConfigMaps are updated for Revisions
@@ -76,24 +76,46 @@ func NewStore(logger configmap.Logger, onAfterStore ...func(name string, value i
 			},
 			onAfterStore...,
 		),
+		apiStore: config.NewStore(logger),
 	}
 	return store
 }
 
+func (s *Store) WatchConfigs(cmw configmap.Watcher) {
+	s.UntypedStore.WatchConfigs(cmw)
+	s.apiStore.WatchConfigs(cmw)
+}
+
 // ToContext persists the config on the context.
 func (s *Store) ToContext(ctx context.Context) context.Context {
-	return ToContext(ctx, s.Load())
+	return s.apiStore.ToContext(ToContext(ctx, s.Load()))
 }
 
 // Load returns the config from the store.
 func (s *Store) Load() *Config {
-	return &Config{
-		Defaults:      s.UntypedLoad(config.DefaultsConfigName).(*config.Defaults).DeepCopy(),
-		Deployment:    s.UntypedLoad(deployment.ConfigName).(*deployment.Config).DeepCopy(),
-		Logging:       s.UntypedLoad((logging.ConfigMapName())).(*logging.Config).DeepCopy(),
-		Network:       s.UntypedLoad(network.ConfigName).(*network.Config).DeepCopy(),
-		Observability: s.UntypedLoad(metrics.ConfigMapName()).(*metrics.ObservabilityConfig).DeepCopy(),
-		Tracing:       s.UntypedLoad(pkgtracing.ConfigName).(*pkgtracing.Config).DeepCopy(),
-		Autoscaler:    s.UntypedLoad(autoscalerconfig.ConfigName).(*autoscalerconfig.Config).DeepCopy(),
+	cfg := &Config{}
+
+	if def, ok := s.UntypedLoad(config.DefaultsConfigName).(*config.Defaults); ok {
+		cfg.Defaults = def.DeepCopy()
+	}
+	if dep, ok := s.UntypedLoad(deployment.ConfigName).(*deployment.Config); ok {
+		cfg.Deployment = dep.DeepCopy()
 	}
+	if log, ok := s.UntypedLoad((logging.ConfigMapName())).(*logging.Config); ok {
+		cfg.Logging = log.DeepCopy()
+	}
+	if net, ok := s.UntypedLoad(network.ConfigName).(*network.Config); ok {
+		cfg.Network = net.DeepCopy()
+	}
+	if obs, ok := s.UntypedLoad(metrics.ConfigMapName()).(*metrics.ObservabilityConfig); ok {
+		cfg.Observability = obs.DeepCopy()
+	}
+	if tr, ok := s.UntypedLoad(pkgtracing.ConfigName).(*pkgtracing.Config); ok {
+		cfg.Tracing = tr.DeepCopy()
+	}
+	if as, ok := s.UntypedLoad(autoscalerconfig.ConfigName).(*autoscalerconfig.Config); ok {
+		cfg.Autoscaler = as.DeepCopy()
+	}
+
+	return cfg
 }


Test patch:
diff --git a/pkg/activator/handler/handler_test.go b/pkg/activator/handler/handler_test.go
index f58b8b927..bf635ce7f 100644
--- a/pkg/activator/handler/handler_test.go
+++ b/pkg/activator/handler/handler_test.go
@@ -28,7 +28,6 @@ import (
 	"testing"
 	"time"
 
-	"go.opencensus.io/plugin/ochttp"
 	"go.uber.org/atomic"
 	"go.uber.org/zap"
 	corev1 "k8s.io/api/core/v1"
@@ -38,7 +37,6 @@ import (
 	rtesting "knative.dev/pkg/reconciler/testing"
 	"knative.dev/pkg/tracing"
 	tracingconfig "knative.dev/pkg/tracing/config"
-	"knative.dev/pkg/tracing/propagation/tracecontextb3"
 	tracetesting "knative.dev/pkg/tracing/testing"
 	"knative.dev/serving/pkg/activator"
 	activatorconfig "knative.dev/serving/pkg/activator/config"
@@ -131,10 +129,7 @@ func TestActivationHandler(t *testing.T) {
 
 			ctx, cancel, _ := rtesting.SetupFakeContextWithCancel(t)
 			defer cancel()
-			handler := (New(ctx, test.throttler)).(*activationHandler)
-
-			// Setup transports.
-			handler.transport = rt
+			handler := New(ctx, test.throttler, rt)
 
 			resp := httptest.NewRecorder()
 			req := httptest.NewRequest(http.MethodPost, "http://example.com", nil)
@@ -173,8 +168,7 @@ func TestActivationHandlerProxyHeader(t *testing.T) {
 	ctx, cancel, _ := rtesting.SetupFakeContextWithCancel(t)
 	defer cancel()
 
-	handler := (New(ctx, fakeThrottler{})).(*activationHandler)
-	handler.transport = rt
+	handler := New(ctx, fakeThrottler{}, rt)
 
 	writer := httptest.NewRecorder()
 	req := httptest.NewRequest(http.MethodPost, "http://example.com", nil)
@@ -252,12 +246,7 @@ func TestActivationHandlerTraceSpans(t *testing.T) {
 				oct.Finish()
 			}()
 
-			handler := (New(ctx, fakeThrottler{})).(*activationHandler)
-			handler.transport = rt
-			handler.tracingTransport = &ochttp.Transport{
-				Base:        rt,
-				Propagation: tracecontextb3.B3Egress,
-			}
+			handler := New(ctx, fakeThrottler{}, rt)
 
 			// Set up config store to populate context.
 			configStore := setupConfigStore(t, logging.FromContext(ctx))
@@ -280,7 +269,7 @@ func TestActivationHandlerTraceSpans(t *testing.T) {
 	}
 }
 
-func sendRequest(namespace, revName string, handler *activationHandler, store *activatorconfig.Store) *httptest.ResponseRecorder {
+func sendRequest(namespace, revName string, handler http.Handler, store *activatorconfig.Store) *httptest.ResponseRecorder {
 	resp := httptest.NewRecorder()
 	req := httptest.NewRequest(http.MethodPost, "http://example.com", nil)
 	ctx := store.ToContext(req.Context())
@@ -328,8 +317,7 @@ func BenchmarkHandler(b *testing.B) {
 			}, nil
 		})
 
-		handler := (New(ctx, fakeThrottler{})).(*activationHandler)
-		handler.transport = rt
+		handler := New(ctx, fakeThrottler{}, rt)
 
 		request := func() *http.Request {
 			req := httptest.NewRequest(http.MethodGet, "http://example.com", nil)
diff --git a/pkg/reconciler/revision/queueing_test.go b/pkg/reconciler/revision/queueing_test.go
index b64019de2..5799a1f6c 100644
--- a/pkg/reconciler/revision/queueing_test.go
+++ b/pkg/reconciler/revision/queueing_test.go
@@ -171,6 +171,12 @@ func newTestController(t *testing.T, opts ...reconcilerOption) (
 				Namespace: system.Namespace(),
 				Name:      network.ConfigName,
 			}}, {
+			ObjectMeta: metav1.ObjectMeta{
+				Namespace: system.Namespace(),
+				Name:      config.FeaturesConfigName,
+			},
+			Data: map[string]string{},
+		}, {
 			ObjectMeta: metav1.ObjectMeta{
 				Namespace: system.Namespace(),
 				Name:      logging.ConfigMapName(),
diff --git a/pkg/reconciler/revision/table_test.go b/pkg/reconciler/revision/table_test.go
index 0a046a108..d9d340498 100644
--- a/pkg/reconciler/revision/table_test.go
+++ b/pkg/reconciler/revision/table_test.go
@@ -624,7 +624,11 @@ func TestReconcile(t *testing.T) {
 
 		return revisionreconciler.NewReconciler(ctx, logging.FromContext(ctx), servingclient.Get(ctx),
 			listers.GetRevisionLister(), controller.GetEventRecorder(ctx), r,
-			controller.Options{ConfigStore: &testConfigStore{config: ReconcilerTestConfig()}})
+			controller.Options{
+				ConfigStore: &testConfigStore{
+					config: ReconcilerTestConfig(),
+				},
+			})
 	}))
 }
 
diff --git a/test/conformance/api/v1/util.go b/test/conformance/api/v1/util.go
index fc9dc1334..0e32bfcfa 100644
--- a/test/conformance/api/v1/util.go
+++ b/test/conformance/api/v1/util.go
@@ -18,6 +18,7 @@ package v1
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"math"
 	"net/http"
@@ -29,6 +30,7 @@ import (
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
+
 	pkgTest "knative.dev/pkg/test"
 	"knative.dev/pkg/test/spoof"
 	"knative.dev/serving/pkg/apis/serving"
@@ -37,9 +39,7 @@ import (
 	v1test "knative.dev/serving/test/v1"
 )
 
-const (
-	scaleToZeroGracePeriod = 30 * time.Second
-)
+const scaleToZeroGracePeriod = 30 * time.Second
 
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
@@ -56,11 +56,11 @@ func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.
 
 func validateDomains(t pkgTest.TLegacy, clients *test.Clients, baseDomain *url.URL,
 	baseExpected, trafficTargets, targetsExpected []string) error {
-	subdomains := make([]*url.URL, 0, len(trafficTargets))
-	for _, target := range trafficTargets {
+	subdomains := make([]*url.URL, len(trafficTargets))
+	for i, target := range trafficTargets {
 		subdomain, _ := url.Parse(baseDomain.String())
 		subdomain.Host = target + "-" + baseDomain.Host
-		subdomains = append(subdomains, subdomain)
+		subdomains[i] = subdomain
 	}
 
 	g, _ := errgroup.WithContext(context.Background())
@@ -124,10 +124,19 @@ func checkDistribution(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, n
 	return checkResponses(t, num, min, url.Hostname(), expectedResponses, actualResponses)
 }
 
+func substrInList(key string, targets []string) string {
+	for _, t := range targets {
+		if strings.Contains(key, t) {
+			return t
+		}
+	}
+	return ""
+}
+
 // checkResponses verifies that each "expectedResponse" is present in "actualResponses" at least "min" times.
-func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expectedResponses []string, actualResponses []string) error {
+func checkResponses(t pkgTest.TLegacy, num, min int, domain string, expectedResponses, actualResponses []string) error {
 	// counts maps the expected response body to the number of matching requests we saw.
-	counts := make(map[string]int)
+	counts := make(map[string]int, len(expectedResponses))
 	// badCounts maps the unexpected response body to the number of matching requests we saw.
 	badCounts := make(map[string]int)
 
@@ -138,39 +147,43 @@ func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expected
 	//   GROUP BY body
 	// )
 	for _, ar := range actualResponses {
-		expected := false
-		for _, er := range expectedResponses {
-			if strings.Contains(ar, er) {
-				counts[er]++
-				expected = true
-			}
-		}
-		if !expected {
+		if er := substrInList(ar, expectedResponses); er != "" {
+			counts[er]++
+		} else {
 			badCounts[ar]++
 		}
 	}
 
+	// Verify that the total expected responses match the number of requests made.
+	for badResponse, count := range badCounts {
+		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
+	}
+
 	// Verify that we saw each entry in "expectedResponses" at least "min" times.
 	// check(SELECT body FROM $counts WHERE total < $min)
 	totalMatches := 0
+	errMsg := []string{}
 	for _, er := range expectedResponses {
 		count := counts[er]
 		if count < min {
-			return fmt.Errorf("domain %s failed: want at least %d, got %d for response %q", domain, min, count, er)
+			errMsg = append(errMsg,
+				fmt.Sprintf("domain %s failed: want at least %d, got %d for response %q",
+					domain, min, count, er))
 		}
 
-		t.Logf("For domain %s: wanted at least %d, got %d requests.", domain, min, count)
+		t.Logf("For domain %s: wanted at least %d, got %d requests for response %q",
+			domain, min, count, er)
 		totalMatches += count
 	}
-	// Verify that the total expected responses match the number of requests made.
-	for badResponse, count := range badCounts {
-		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
-	}
 	if totalMatches < num {
-		return fmt.Errorf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num)
+		errMsg = append(errMsg,
+			fmt.Sprintf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num))
 	}
-	// If we made it here, the implementation conforms. Congratulations!
-	return nil
+	if len(errMsg) == 0 {
+		// If we made it here, the implementation conforms. Congratulations!
+		return nil
+	}
+	return errors.New(strings.Join(errMsg, ","))
 }
 
 // sendRequests sends "num" requests to "url", returning a string for each spoof.Response.Body.
diff --git a/test/conformance/api/v1alpha1/util.go b/test/conformance/api/v1alpha1/util.go
index d63bc34b6..b488f3ae9 100644
--- a/test/conformance/api/v1alpha1/util.go
+++ b/test/conformance/api/v1alpha1/util.go
@@ -18,6 +18,7 @@ package v1alpha1
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"math"
 	"net/http"
@@ -29,6 +30,7 @@ import (
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
+
 	pkgTest "knative.dev/pkg/test"
 	"knative.dev/pkg/test/spoof"
 	"knative.dev/serving/pkg/apis/serving"
@@ -37,9 +39,7 @@ import (
 	v1a1test "knative.dev/serving/test/v1alpha1"
 )
 
-const (
-	scaleToZeroGracePeriod = 30 * time.Second
-)
+const scaleToZeroGracePeriod = 30 * time.Second
 
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
@@ -56,11 +56,11 @@ func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.
 
 func validateDomains(t pkgTest.TLegacy, clients *test.Clients, baseDomain *url.URL,
 	baseExpected, trafficTargets, targetsExpected []string) error {
-	subdomains := make([]*url.URL, 0, len(trafficTargets))
-	for _, target := range trafficTargets {
+	subdomains := make([]*url.URL, len(trafficTargets))
+	for i, target := range trafficTargets {
 		subdomain, _ := url.Parse(baseDomain.String())
 		subdomain.Host = target + "-" + baseDomain.Host
-		subdomains = append(subdomains, subdomain)
+		subdomains[i] = subdomain
 	}
 
 	g, _ := errgroup.WithContext(context.Background())
@@ -124,10 +124,19 @@ func checkDistribution(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, n
 	return checkResponses(t, num, min, url.Hostname(), expectedResponses, actualResponses)
 }
 
+func substrInList(key string, targets []string) string {
+	for _, t := range targets {
+		if strings.Contains(key, t) {
+			return t
+		}
+	}
+	return ""
+}
+
 // checkResponses verifies that each "expectedResponse" is present in "actualResponses" at least "min" times.
-func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expectedResponses []string, actualResponses []string) error {
+func checkResponses(t pkgTest.TLegacy, num, min int, domain string, expectedResponses, actualResponses []string) error {
 	// counts maps the expected response body to the number of matching requests we saw.
-	counts := make(map[string]int)
+	counts := make(map[string]int, len(expectedResponses))
 	// badCounts maps the unexpected response body to the number of matching requests we saw.
 	badCounts := make(map[string]int)
 
@@ -138,39 +147,42 @@ func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expected
 	//   GROUP BY body
 	// )
 	for _, ar := range actualResponses {
-		expected := false
-		for _, er := range expectedResponses {
-			if strings.Contains(ar, er) {
-				counts[er]++
-				expected = true
-			}
-		}
-		if !expected {
+		if er := substrInList(ar, expectedResponses); er != "" {
+			counts[er]++
+		} else {
 			badCounts[ar]++
 		}
 	}
 
+	// Verify that the total expected responses match the number of requests made.
+	for badResponse, count := range badCounts {
+		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
+	}
+
 	// Verify that we saw each entry in "expectedResponses" at least "min" times.
 	// check(SELECT body FROM $counts WHERE total < $min)
 	totalMatches := 0
+	errMsg := []string{}
 	for _, er := range expectedResponses {
 		count := counts[er]
 		if count < min {
-			return fmt.Errorf("domain %s failed: want at least %d, got %d for response %q", domain, min, count, er)
+			errMsg = append(errMsg,
+				fmt.Sprintf("domain %s failed: want at least %d, got %d for response %q",
+					domain, min, count, er))
 		}
 
 		t.Logf("For domain %s: wanted at least %d, got %d requests.", domain, min, count)
 		totalMatches += count
 	}
-	// Verify that the total expected responses match the number of requests made.
-	for badResponse, count := range badCounts {
-		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
-	}
 	if totalMatches < num {
-		return fmt.Errorf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num)
+		errMsg = append(errMsg,
+			fmt.Sprintf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num))
 	}
-	// If we made it here, the implementation conforms. Congratulations!
-	return nil
+	if len(errMsg) == 0 {
+		// If we made it here, the implementation conforms. Congratulations!
+		return nil
+	}
+	return errors.New(strings.Join(errMsg, ","))
 }
 
 // sendRequests sends "num" requests to "url", returning a string for each spoof.Response.Body.
diff --git a/test/conformance/api/v1beta1/util.go b/test/conformance/api/v1beta1/util.go
index 8e0d85f68..7d8088ce3 100644
--- a/test/conformance/api/v1beta1/util.go
+++ b/test/conformance/api/v1beta1/util.go
@@ -18,6 +18,7 @@ package v1beta1
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"math"
 	"net/http"
@@ -29,6 +30,7 @@ import (
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
+
 	pkgTest "knative.dev/pkg/test"
 	"knative.dev/pkg/test/spoof"
 	"knative.dev/serving/pkg/apis/serving"
@@ -37,9 +39,7 @@ import (
 	v1b1test "knative.dev/serving/test/v1beta1"
 )
 
-const (
-	scaleToZeroGracePeriod = 30 * time.Second
-)
+const scaleToZeroGracePeriod = 30 * time.Second
 
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
@@ -124,10 +124,19 @@ func checkDistribution(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, n
 	return checkResponses(t, num, min, url.Hostname(), expectedResponses, actualResponses)
 }
 
+func substrInList(key string, targets []string) string {
+	for _, t := range targets {
+		if strings.Contains(key, t) {
+			return t
+		}
+	}
+	return ""
+}
+
 // checkResponses verifies that each "expectedResponse" is present in "actualResponses" at least "min" times.
-func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expectedResponses []string, actualResponses []string) error {
+func checkResponses(t pkgTest.TLegacy, num, min int, domain string, expectedResponses, actualResponses []string) error {
 	// counts maps the expected response body to the number of matching requests we saw.
-	counts := make(map[string]int)
+	counts := make(map[string]int, len(expectedResponses))
 	// badCounts maps the unexpected response body to the number of matching requests we saw.
 	badCounts := make(map[string]int)
 
@@ -138,39 +147,42 @@ func checkResponses(t pkgTest.TLegacy, num int, min int, domain string, expected
 	//   GROUP BY body
 	// )
 	for _, ar := range actualResponses {
-		expected := false
-		for _, er := range expectedResponses {
-			if strings.Contains(ar, er) {
-				counts[er]++
-				expected = true
-			}
-		}
-		if !expected {
+		if er := substrInList(ar, expectedResponses); er != "" {
+			counts[er]++
+		} else {
 			badCounts[ar]++
 		}
 	}
 
+	// Verify that the total expected responses match the number of requests made.
+	for badResponse, count := range badCounts {
+		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
+	}
+
 	// Verify that we saw each entry in "expectedResponses" at least "min" times.
 	// check(SELECT body FROM $counts WHERE total < $min)
 	totalMatches := 0
+	errMsg := []string{}
 	for _, er := range expectedResponses {
 		count := counts[er]
 		if count < min {
-			return fmt.Errorf("domain %s failed: want at least %d, got %d for response %q", domain, min, count, er)
+			errMsg = append(errMsg,
+				fmt.Sprintf("domain %s failed: want at least %d, got %d for response %q",
+					domain, min, count, er))
 		}
 
 		t.Logf("For domain %s: wanted at least %d, got %d requests.", domain, min, count)
 		totalMatches += count
 	}
-	// Verify that the total expected responses match the number of requests made.
-	for badResponse, count := range badCounts {
-		t.Logf("Saw unexpected response %q %d times.", badResponse, count)
-	}
 	if totalMatches < num {
-		return fmt.Errorf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num)
+		errMsg = append(errMsg,
+			fmt.Sprintf("domain %s: saw expected responses %d times, wanted %d", domain, totalMatches, num))
 	}
-	// If we made it here, the implementation conforms. Congratulations!
-	return nil
+	if len(errMsg) == 0 {
+		// If we made it here, the implementation conforms. Congratulations!
+		return nil
+	}
+	return errors.New(strings.Join(errMsg, ","))
 }
 
 // sendRequests sends "num" requests to "url", returning a string for each spoof.Response.Body.
diff --git a/test/e2e/logging_test.go b/test/e2e/logging_test.go
index 59c8270a5..2c3353996 100644
--- a/test/e2e/logging_test.go
+++ b/test/e2e/logging_test.go
@@ -22,9 +22,11 @@ import (
 	"bufio"
 	"encoding/json"
 	"fmt"
+	"strings"
 	"testing"
 	"time"
 
+	"github.com/google/go-cmp/cmp"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
@@ -39,10 +41,21 @@ import (
 	v1test "knative.dev/serving/test/v1"
 )
 
+const template = `{"httpRequest": {"requestMethod": "{{.Request.Method}}", "requestUrl": "{{js .Request.RequestURI}}", "requestSize": "{{.Request.ContentLength}}", "status": {{.Response.Code}}, "responseSize": "{{.Response.Size}}", "userAgent": "{{js .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}", "serverIp": "{{.Revision.PodIP}}", "referer": "{{js .Request.Referer}}", "latency": "{{.Response.Latency}}s", "protocol": "{{.Request.Proto}}"}, "traceId": "{{index .Request.Header "X-B3-Traceid"}}"}`
+
 func TestRequestLogs(t *testing.T) {
 	t.Parallel()
 	clients := Setup(t)
 
+	cm, err := clients.KubeClient.GetConfigMap(system.Namespace()).Get("config-observability", metav1.GetOptions{})
+	if err != nil {
+		t.Fatalf("Fail to get ConfigMap config-observability: %v", err)
+	}
+
+	if got, want := cm.Data["logging.request-log-template"], template; got != want {
+		t.Skipf("Skipping verifing request logs because the template doesn't match:\n%s", cmp.Diff(want, got))
+	}
+
 	names := test.ResourceNames{
 		Service: test.ObjectNameForTest(t),
 		Image:   "helloworld",
@@ -78,6 +91,7 @@ func TestRequestLogs(t *testing.T) {
 		t.Fatalf("Fail to fetch the pod: %v", err)
 	}
 
+	// TODO: add logging.enable-request-log check once it doesn't depends on the template.
 	// A request was sent to / in WaitForEndpointState.
 	if err := waitForLog(t, clients, pod.Namespace, pod.Name, "queue-proxy", func(log logLine) bool {
 		return log.HTTPRequest.RequestURL == "/" &&
@@ -86,12 +100,17 @@ func TestRequestLogs(t *testing.T) {
 		t.Fatalf("Got error waiting for normal request logs: %v", err)
 	}
 
-	// Health check requests are sent to / with a specific userAgent value periodically.
-	if err := waitForLog(t, clients, pod.Namespace, pod.Name, "queue-proxy", func(log logLine) bool {
-		return log.HTTPRequest.RequestURL == "/" &&
-			log.HTTPRequest.UserAgent == network.QueueProxyUserAgent
-	}); err != nil {
-		t.Fatalf("Got error waiting for health check log: %v", err)
+	// Only check probe request logs if the feature is enabled in config-observability.
+	if strings.EqualFold(cm.Data["logging.enable-probe-request-log"], "true") {
+		// Health check requests are sent to / with a specific userAgent value periodically.
+		if err := waitForLog(t, clients, pod.Namespace, pod.Name, "queue-proxy", func(log logLine) bool {
+			return log.HTTPRequest.RequestURL == "/" &&
+				log.HTTPRequest.UserAgent == network.QueueProxyUserAgent
+		}); err != nil {
+			t.Fatalf("Got error waiting for health check log: %v", err)
+		}
+	} else {
+		t.Log("Skipping verifing probe request logs because they are not enabled")
 	}
 }
 
diff --git a/test/e2e/minscale_readiness_test.go b/test/e2e/minscale_readiness_test.go
index d6095751a..bd04db302 100644
--- a/test/e2e/minscale_readiness_test.go
+++ b/test/e2e/minscale_readiness_test.go
@@ -66,8 +66,8 @@ func TestMinScale(t *testing.T) {
 	serviceName := privateServiceName(t, clients, revName)
 
 	t.Log("Waiting for revision to scale to minScale before becoming ready")
-	if err := waitForDesiredScale(clients, serviceName, gte(minScale)); err != nil {
-		t.Fatalf("The revision %q did not scale >= %d before becoming ready: %v", revName, minScale, err)
+	if lr, err := waitForDesiredScale(clients, serviceName, gte(minScale)); err != nil {
+		t.Fatalf("The revision %q scaled to %d < %d before becoming ready: %v", revName, lr, minScale, err)
 	}
 
 	t.Log("Waiting for revision to become ready")
@@ -78,8 +78,8 @@ func TestMinScale(t *testing.T) {
 	}
 
 	t.Log("Holding revision at minScale after becoming ready")
-	if !ensureDesiredScale(clients, serviceName, gte(minScale)) {
-		t.Fatalf("The revision %q did not stay at scale >= %d after becoming ready", revName, minScale)
+	if lr, ok := ensureDesiredScale(clients, t, serviceName, gte(minScale)); !ok {
+		t.Fatalf("The revision %q observed scale %d < %d after becoming ready", revName, lr, minScale)
 	}
 
 	t.Log("Updating configuration")
@@ -91,8 +91,8 @@ func TestMinScale(t *testing.T) {
 	newServiceName := privateServiceName(t, clients, newRevName)
 
 	t.Log("Waiting for new revision to scale to minScale after update")
-	if err := waitForDesiredScale(clients, newServiceName, gte(minScale)); err != nil {
-		t.Fatalf("The revision %q did not scale >= %d after creating route: %v", newRevName, minScale, err)
+	if lr, err := waitForDesiredScale(clients, newServiceName, gte(minScale)); err != nil {
+		t.Fatalf("The revision %q scaled to %d < %d after creating route: %v", newRevName, lr, minScale, err)
 	}
 
 	t.Log("Waiting for new revision to become ready")
@@ -103,23 +103,23 @@ func TestMinScale(t *testing.T) {
 	}
 
 	t.Log("Holding new revision at minScale after becoming ready")
-	if !ensureDesiredScale(clients, newServiceName, gte(minScale)) {
-		t.Fatalf("The new revision %q did not stay at scale >= %d after becoming ready", newRevName, minScale)
+	if lr, ok := ensureDesiredScale(clients, t, newServiceName, gte(minScale)); !ok {
+		t.Fatalf("The revision %q observed scale %d < %d after becoming ready", newRevName, lr, minScale)
 	}
 
 	t.Log("Waiting for old revision to scale below minScale after being replaced")
-	if err := waitForDesiredScale(clients, serviceName, lt(minScale)); err != nil {
-		t.Fatalf("The revision %q did not scale < minScale after being replaced: %v", revName, err)
+	if lr, err := waitForDesiredScale(clients, serviceName, lt(minScale)); err != nil {
+		t.Fatalf("The revision %q scaled to %d > %d after not being routable anymore: %v", revName, lr, minScale, err)
 	}
 
-	t.Log("Deleting route")
+	t.Log("Deleting route", names.Route)
 	if err := clients.ServingClient.Routes.Delete(names.Route, &metav1.DeleteOptions{}); err != nil {
 		t.Fatalf("Failed to delete route %q: %v", names.Route, err)
 	}
 
 	t.Log("Waiting for new revision to scale below minScale when there is no route")
-	if err := waitForDesiredScale(clients, newServiceName, lt(minScale)); err != nil {
-		t.Fatalf("The revision %q did not scale < minScale after being replaced: %v", newRevName, err)
+	if lr, err := waitForDesiredScale(clients, newServiceName, lt(minScale)); err != nil {
+		t.Fatalf("The revision %q scaled to %d > %d after not being routable anymore: %v", newRevName, lr, minScale, err)
 	}
 }
 
@@ -180,31 +180,39 @@ func privateServiceName(t *testing.T, clients *test.Clients, revisionName string
 	return privateServiceName
 }
 
-func waitForDesiredScale(clients *test.Clients, serviceName string, cond func(int) bool) error {
+// waitForDesiredScale retuns the last observed number of pods and/or error if the cond
+// callback is never satisfied.
+func waitForDesiredScale(clients *test.Clients, serviceName string, cond func(int) bool) (latestReady int, err error) {
 	endpoints := clients.KubeClient.Kube.CoreV1().Endpoints(test.ServingNamespace)
 
-	return wait.PollImmediate(250*time.Millisecond, 1*time.Minute, func() (bool, error) {
+	return latestReady, wait.PollImmediate(250*time.Millisecond, time.Minute, func() (bool, error) {
 		endpoint, err := endpoints.Get(serviceName, metav1.GetOptions{})
 		if err != nil {
 			return false, nil
 		}
-		return cond(resources.ReadyAddressCount(endpoint)), nil
+		latestReady = resources.ReadyAddressCount(endpoint)
+		return cond(latestReady), nil
 	})
 }
 
-func ensureDesiredScale(clients *test.Clients, serviceName string, cond func(int) bool) bool {
+func ensureDesiredScale(clients *test.Clients, t *testing.T, serviceName string, cond func(int) bool) (latestReady int, observed bool) {
 	endpoints := clients.KubeClient.Kube.CoreV1().Endpoints(test.ServingNamespace)
 
-	return wait.PollImmediate(250*time.Millisecond, 5*time.Second, func() (bool, error) {
+	err := wait.PollImmediate(250*time.Millisecond, 10*time.Second, func() (bool, error) {
 		endpoint, err := endpoints.Get(serviceName, metav1.GetOptions{})
 		if err != nil {
 			return false, nil
 		}
 
-		if scale := resources.ReadyAddressCount(endpoint); !cond(scale) {
-			return false, fmt.Errorf("scale %d didn't meet condition", scale)
+		if latestReady = resources.ReadyAddressCount(endpoint); !cond(latestReady) {
+			return false, fmt.Errorf("scale %d didn't meet condition", latestReady)
 		}
 
 		return false, nil
-	}) == wait.ErrWaitTimeout
+	})
+	if err != wait.ErrWaitTimeout {
+		t.Log("PollError =", err)
+	}
+
+	return latestReady, err == wait.ErrWaitTimeout
 }
diff --git a/third_party/contour-latest/contour.yaml b/third_party/contour-latest/contour.yaml
index 5229deb17..4c8f2c0fb 100644
--- a/third_party/contour-latest/contour.yaml
+++ b/third_party/contour-latest/contour.yaml
@@ -1432,7 +1432,7 @@ spec:
         # if we change this on each version, you can no longer upgrade
         # just by applying the deployment YAML.
         # See #2423, #2395, #2150, and #2030 for earlier questions about this.
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         command:
         - contour
@@ -1672,7 +1672,7 @@ spec:
         - --contour-key-file=/certs/tls.key
         - --config-path=/config/contour.yaml
         command: ["contour"]
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         name: contour
         ports:
@@ -1764,7 +1764,7 @@ spec:
         args:
         - envoy
         - shutdown-manager
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         lifecycle:
           preStop:
@@ -1840,7 +1840,7 @@ spec:
         - --envoy-key-file=/certs/tls.key
         command:
         - contour
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         name: envoy-initconfig
         volumeMounts:
@@ -3305,7 +3305,7 @@ spec:
         # if we change this on each version, you can no longer upgrade
         # just by applying the deployment YAML.
         # See #2423, #2395, #2150, and #2030 for earlier questions about this.
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         command:
         - contour
@@ -3545,7 +3545,7 @@ spec:
         - --contour-key-file=/certs/tls.key
         - --config-path=/config/contour.yaml
         command: ["contour"]
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         name: contour
         ports:
@@ -3637,7 +3637,7 @@ spec:
         args:
         - envoy
         - shutdown-manager
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         lifecycle:
           preStop:
@@ -3713,7 +3713,7 @@ spec:
         - --envoy-key-file=/certs/tls.key
         command:
         - contour
-        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:0c16f2b09105576f5cd0a5688029e19227c0dcd13854ed5a0d63a028c3f495eb
+        image: gcr.io/knative-nightly/github.com/projectcontour/contour/cmd/contour@sha256:a293ec55809f4dd4bb401151b65b8819cdf026323d776207ebcf2a0a2ddefd09
         imagePullPolicy: Always
         name: envoy-initconfig
         volumeMounts:
diff --git a/third_party/contour-latest/net-contour.yaml b/third_party/contour-latest/net-contour.yaml
index ee7f2e4f6..309a9a7b2 100644
--- a/third_party/contour-latest/net-contour.yaml
+++ b/third_party/contour-latest/net-contour.yaml
@@ -35,7 +35,7 @@ metadata:
   namespace: knative-serving
   labels:
     networking.knative.dev/ingress-provider: contour
-    serving.knative.dev/release: "v20200810-4cc4b348"
+    serving.knative.dev/release: "v20200812-0b01e9b5"
 data:
   _example: |
     ################################
@@ -44,6 +44,12 @@ data:
     #                              #
     ################################
 
+    # If auto-TLS is disabled fallback to the following certificate
+    #
+    # An operator is required to setup a TLSCertificateDelegation
+    # for this secret to be used
+    default-tls-secret: "some-namespace/some-secret"
+
     # visibility contains the configuration for how to expose services
     # of assorted visibilities.  Each entry is keyed by the visibility
     # and contains two keys:
@@ -94,7 +100,7 @@ spec:
       - name: controller
         # This is the Go import path for the binary that is containerized
         # and substituted here.
-        image: gcr.io/knative-nightly/knative.dev/net-contour/cmd/controller@sha256:d893736c59eaac36a49729891906ccaaa1b63433a5db81b124801c1a019662cc
+        image: gcr.io/knative-nightly/knative.dev/net-contour/cmd/controller@sha256:281de2b6ee0ae67d276dbd4948010d20e0e501e5d90a1031530d5e8fbc9a73a9
         resources:
           requests:
             cpu: 40m
diff --git a/third_party/kourier-latest/kourier.yaml b/third_party/kourier-latest/kourier.yaml
index 7a9097b5c..620878dcf 100644
--- a/third_party/kourier-latest/kourier.yaml
+++ b/third_party/kourier-latest/kourier.yaml
@@ -117,7 +117,7 @@ spec:
         - /tmp/config/envoy-bootstrap.yaml
         command:
         - /usr/local/bin/envoy
-        image: docker.io/maistra/proxyv2-ubi8:1.1.3
+        image: docker.io/maistra/proxyv2-ubi8:1.1.5
         imagePullPolicy: Always
         name: kourier-gateway
         ports:
@@ -172,7 +172,7 @@ spec:
         app: 3scale-kourier-control
     spec:
       containers:
-      - image: gcr.io/knative-nightly/knative.dev/net-kourier/cmd/kourier@sha256:826c51ebdaa6a277f85f89a9fbe837ac404af65246e558432675d1df4f84b3b2
+      - image: gcr.io/knative-nightly/knative.dev/net-kourier/cmd/kourier@sha256:6c5fe5d62246b07ac59cfd603763d1da5f32ee37c923a89345535dd0a82e2de7
         imagePullPolicy: Always
         name: kourier-control
         env: