Repository: knative/serving
Pull request #8568 (2020-07-07 10:44:54)

Change transport of direct client for customized MaxIdleConns
Currently we noticed the CPU usage for autoscaler is high when there are hundreds of pods running in cluster. From the CPU profling we can tell that it is spending on dialing for connections.
```
      flat  flat%   sum%        cum   cum%
         0     0%     0%      0.81s 19.61%  net/http.(*Transport).dialConn
         0     0%     0%      0.81s 19.61%  net/http.(*Transport).dialConnFor
         0     0%     0%      0.75s 18.16%  runtime.mcall
         0     0%     0%      0.74s 17.92%  net.(*Dialer).DialContext
         0     0%     0%      0.74s 17.92%  net/http.(*Transport).dial
     0.01s  0.24%  0.24%      0.74s 17.92%  runtime.schedule
         0     0%  0.24%      0.72s 17.43%  golang.org/x/sync/errgroup.(*Group).Go.func1
     0.70s 16.95% 17.19%      0.72s 17.43%  syscall.Syscall
         0     0% 17.19%      0.70s 16.95%  knative.dev/serving/pkg/autoscaler/metrics.(*serviceScraper).scrapePods.func1
         0     0% 17.19%      0.68s 16.46%  knative.dev/serving/pkg/autoscaler/metrics.(*httpScrapeClient).Scrape
```

The optimization in https://github.com/knative/serving/pull/8367 of @julz works but the default value 100 of `MaxIdleConns` is too small for pod scraping. The customized transport config from pkg/network with 1000 of `MaxIdleConns` should works better. After applying this change  I see a significant CPU drop ( 54% -> 36%) with 300 pods running in cluster.

![image](https://user-images.githubusercontent.com/62631896/86768456-31ad2880-c080-11ea-837f-36d4cb3c56d8.png)


**Release Note**

```release-note
None
```


knative-prow-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/serving/pull/8568#" title="Author self-approved">lanceliuu</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **mdemirhan**
You can assign the PR to them by writing `/assign @mdemirhan` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

<details open>
Needs approval from an approver in each of these files:

- **[pkg/autoscaler/OWNERS](https://github.com/knative/serving/blob/master/pkg/autoscaler/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["mdemirhan"]} -->
knative-prow-robot: Hi @lanceliuu. Thanks for your PR.

I'm waiting for a [knative](https://github.com/orgs/knative/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/knative/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

knative-prow-robot: Welcome @lanceliuu! It looks like this is your first PR to knative/serving 🎉
markusthoemmes: /ok-to-test

/hold
until after the release. Caution and stuff.

Patch:
diff --git a/pkg/autoscaler/metrics/stats_scraper.go b/pkg/autoscaler/metrics/stats_scraper.go
index 0a1fbc1b9..9bcfee711 100644
--- a/pkg/autoscaler/metrics/stats_scraper.go
+++ b/pkg/autoscaler/metrics/stats_scraper.go
@@ -118,6 +118,10 @@ var noKeepaliveClient = &http.Client{
 // between scrapes of the same pod.
 var client = &http.Client{
 	Timeout: httpClientTimeout,
+	Transport: &http.Transport{
+		MaxIdleConns:    1000,
+		IdleConnTimeout: 90 * time.Second,
+	},
 }
 
 // serviceScraper scrapes Revision metrics via a K8S service by sampling. Which
diff --git a/pkg/metrics/key.go b/pkg/metrics/key.go
index 93fd9e5c2..a4d17cb59 100644
--- a/pkg/metrics/key.go
+++ b/pkg/metrics/key.go
@@ -35,6 +35,7 @@ var (
 	ContainerTagKey      = tag.MustNewKey(metricskey.ContainerName)
 	ResponseCodeKey      = tag.MustNewKey(metricskey.LabelResponseCode)
 	ResponseCodeClassKey = tag.MustNewKey(metricskey.LabelResponseCodeClass)
+	RouteTagKey          = tag.MustNewKey("tag")
 
 	CommonRevisionKeys = []tag.Key{NamespaceTagKey, ServiceTagKey, ConfigTagKey, RevisionTagKey}
 )
diff --git a/pkg/metrics/tags.go b/pkg/metrics/tags.go
index 2bc69633a..f798522e3 100644
--- a/pkg/metrics/tags.go
+++ b/pkg/metrics/tags.go
@@ -135,6 +135,16 @@ func AugmentWithResponse(baseCtx context.Context, responseCode int) context.Cont
 	return ctx
 }
 
+// AugmentWithResponseAndRouteTag augments the given context with response-code and route-tag specific tags.
+func AugmentWithResponseAndRouteTag(baseCtx context.Context, responseCode int, routeTag string) context.Context {
+	ctx, _ := tag.New(
+		baseCtx,
+		tag.Upsert(ResponseCodeKey, strconv.Itoa(responseCode)),
+		tag.Upsert(ResponseCodeClassKey, responseCodeClass(responseCode)),
+		tag.Upsert(RouteTagKey, routeTag))
+	return ctx
+}
+
 // responseCodeClass converts response code to a string of response code class.
 // e.g. The response code class is "5xx" for response code 503.
 func responseCodeClass(responseCode int) string {
diff --git a/pkg/queue/request_metric.go b/pkg/queue/request_metric.go
index dc1a802bb..94c8cd80f 100644
--- a/pkg/queue/request_metric.go
+++ b/pkg/queue/request_metric.go
@@ -60,6 +60,12 @@ var (
 		stats.UnitDimensionless)
 )
 
+const (
+	defaultTagName   = "DEFAULT"
+	undefinedTagName = "UNDEFINED"
+	disabledTagName  = "DISABLED"
+)
+
 type requestMetricsHandler struct {
 	next     http.Handler
 	statsCtx context.Context
@@ -75,7 +81,7 @@ type appRequestMetricsHandler struct {
 func NewRequestMetricsHandler(next http.Handler,
 	ns, service, config, rev, pod string) (http.Handler, error) {
 	keys := append(metrics.CommonRevisionKeys, metrics.PodTagKey,
-		metrics.ContainerTagKey, metrics.ResponseCodeKey, metrics.ResponseCodeClassKey)
+		metrics.ContainerTagKey, metrics.ResponseCodeKey, metrics.ResponseCodeClassKey, metrics.RouteTagKey)
 	if err := pkgmetrics.RegisterResourceView(
 		&view.View{
 			Description: "The number of requests that are routed to queue-proxy",
@@ -117,13 +123,16 @@ func (h *requestMetricsHandler) ServeHTTP(w http.ResponseWriter, r *http.Request
 		// If ServeHTTP panics, recover, record the failure and panic again.
 		err := recover()
 		latency := time.Since(startTime)
+		routeTag := GetRouteTagNameFromRequest(r)
 		if err != nil {
-			ctx := metrics.AugmentWithResponse(h.statsCtx, http.StatusInternalServerError)
+			ctx := metrics.AugmentWithResponseAndRouteTag(h.statsCtx,
+				http.StatusInternalServerError, routeTag)
 			pkgmetrics.RecordBatch(ctx, requestCountM.M(1),
 				responseTimeInMsecM.M(float64(latency.Milliseconds())))
 			panic(err)
 		}
-		ctx := metrics.AugmentWithResponse(h.statsCtx, rr.ResponseCode)
+		ctx := metrics.AugmentWithResponseAndRouteTag(h.statsCtx,
+			rr.ResponseCode, routeTag)
 		pkgmetrics.RecordBatch(ctx, requestCountM.M(1),
 			responseTimeInMsecM.M(float64(latency.Milliseconds())))
 	}()
@@ -189,9 +198,32 @@ func (h *appRequestMetricsHandler) ServeHTTP(w http.ResponseWriter, r *http.Requ
 				appResponseTimeInMsecM.M(float64(latency.Milliseconds())))
 			panic(err)
 		}
+
 		ctx := metrics.AugmentWithResponse(h.statsCtx, rr.ResponseCode)
 		pkgmetrics.RecordBatch(ctx, appRequestCountM.M(1),
 			appResponseTimeInMsecM.M(float64(latency.Milliseconds())))
 	}()
 	h.next.ServeHTTP(rr, r)
 }
+
+// GetRouteTagNameFromRequest extracts the value of the tag header from http.Request
+func GetRouteTagNameFromRequest(r *http.Request) string {
+	name := r.Header.Get(network.TagHeaderName)
+	isDefaultRoute := r.Header.Get(network.DefaultRouteHeaderName)
+
+	if name == "" {
+		if isDefaultRoute == "" {
+			// If there are no tag header and no `Knative-Serving-Default-Route` header,
+			// it means that the tag header based routing is disabled, so the tag value is set to `disabled`.
+			return disabledTagName
+		}
+		// If there is no tag header, just returns "default".
+		return defaultTagName
+	} else if isDefaultRoute == "true" {
+		// If there is a tag header with not-empty string and the request is routed via the default route,
+		// returns "undefined".
+		return undefinedTagName
+	}
+	// Otherwise, returns the value of the tag header.
+	return name
+}
diff --git a/pkg/reconciler/labeler/accessors.go b/pkg/reconciler/labeler/accessors.go
deleted file mode 100644
index be168640f..000000000
--- a/pkg/reconciler/labeler/accessors.go
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
-Copyright 2020 The Knative Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    https://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package labeler
-
-import (
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/apimachinery/pkg/types"
-	"knative.dev/pkg/kmeta"
-
-	"knative.dev/serving/pkg/apis/serving"
-)
-
-// accessor defines an abstraction for manipulating labeled entity
-// (Configuration, Revision) with shared logic.
-type accessor interface {
-	get(ns, name string) (kmeta.Accessor, error)
-	list(ns, name string) ([]kmeta.Accessor, error)
-	patch(ns, name string, pt types.PatchType, p []byte) error
-}
-
-// revision is an implementation of accessor for Revisions
-type revision struct {
-	r *Reconciler
-}
-
-// revision implements accessor
-var _ accessor = (*revision)(nil)
-
-// get implements accessor
-func (r *revision) get(ns, name string) (kmeta.Accessor, error) {
-	return r.r.revisionLister.Revisions(ns).Get(name)
-}
-
-// list implements accessor
-func (r *revision) list(ns, name string) ([]kmeta.Accessor, error) {
-	rl, err := r.r.revisionLister.Revisions(ns).List(labels.SelectorFromSet(labels.Set{
-		serving.RouteLabelKey: name,
-	}))
-	if err != nil {
-		return nil, err
-	}
-	// Need a copy to change types in Go
-	kl := make([]kmeta.Accessor, 0, len(rl))
-	for _, r := range rl {
-		kl = append(kl, r)
-	}
-	return kl, err
-}
-
-// patch implements accessor
-func (r *revision) patch(ns, name string, pt types.PatchType, p []byte) error {
-	_, err := r.r.client.ServingV1().Revisions(ns).Patch(name, pt, p)
-	return err
-}
-
-// configuration is an implementation of accessor for Configurations
-type configuration struct {
-	r *Reconciler
-}
-
-// configuration implements accessor
-var _ accessor = (*configuration)(nil)
-
-// get implements accessor
-func (c *configuration) get(ns, name string) (kmeta.Accessor, error) {
-	return c.r.configurationLister.Configurations(ns).Get(name)
-}
-
-// list implements accessor
-func (c *configuration) list(ns, name string) ([]kmeta.Accessor, error) {
-	rl, err := c.r.configurationLister.Configurations(ns).List(labels.SelectorFromSet(labels.Set{
-		serving.RouteLabelKey: name,
-	}))
-	if err != nil {
-		return nil, err
-	}
-	// Need a copy to change types in Go
-	kl := make([]kmeta.Accessor, 0, len(rl))
-	for _, r := range rl {
-		kl = append(kl, r)
-	}
-	return kl, err
-}
-
-// patch implements accessor
-func (c *configuration) patch(ns, name string, pt types.PatchType, p []byte) error {
-	_, err := c.r.client.ServingV1().Configurations(ns).Patch(name, pt, p)
-	return err
-}
diff --git a/pkg/reconciler/labeler/controller.go b/pkg/reconciler/labeler/controller.go
index 29a4812a0..eefc439ed 100644
--- a/pkg/reconciler/labeler/controller.go
+++ b/pkg/reconciler/labeler/controller.go
@@ -21,7 +21,7 @@ import (
 
 	"k8s.io/client-go/tools/cache"
 
-	"knative.dev/serving/pkg/apis/serving"
+	v1 "knative.dev/serving/pkg/apis/serving/v1"
 	servingclient "knative.dev/serving/pkg/client/injection/client"
 	configurationinformer "knative.dev/serving/pkg/client/injection/informers/serving/v1/configuration"
 	revisioninformer "knative.dev/serving/pkg/client/injection/informers/serving/v1/revision"
@@ -32,7 +32,7 @@ import (
 	"knative.dev/pkg/configmap"
 	"knative.dev/pkg/controller"
 	"knative.dev/pkg/logging"
-	pkgreconciler "knative.dev/pkg/reconciler"
+	"knative.dev/pkg/tracker"
 )
 
 const controllerAgentName = "labeler-controller"
@@ -65,10 +65,26 @@ func NewController(
 	logger.Info("Setting up event handlers")
 	routeInformer.Informer().AddEventHandler(controller.HandleAll(impl.Enqueue))
 
-	configInformer.Informer().AddEventHandler(cache.FilteringResourceEventHandler{
-		FilterFunc: pkgreconciler.LabelExistsFilterFunc(serving.RouteLabelKey),
-		Handler:    controller.HandleAll(impl.EnqueueLabelOfNamespaceScopedResource("", serving.RouteLabelKey)),
+	c.tracker = tracker.New(impl.EnqueueKey, controller.GetTrackerLease(ctx))
+
+	// Make sure trackers are deleted once the observers are removed.
+	routeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
+		DeleteFunc: c.tracker.OnDeletedObserver,
 	})
 
+	configInformer.Informer().AddEventHandler(controller.HandleAll(
+		controller.EnsureTypeMeta(
+			c.tracker.OnChanged,
+			v1.SchemeGroupVersion.WithKind("Configuration"),
+		),
+	))
+
+	revisionInformer.Informer().AddEventHandler(controller.HandleAll(
+		controller.EnsureTypeMeta(
+			c.tracker.OnChanged,
+			v1.SchemeGroupVersion.WithKind("Revision"),
+		),
+	))
+
 	return impl
 }
diff --git a/pkg/reconciler/labeler/labeler.go b/pkg/reconciler/labeler/labeler.go
index e45d5b6a9..f0aa71d3c 100644
--- a/pkg/reconciler/labeler/labeler.go
+++ b/pkg/reconciler/labeler/labeler.go
@@ -20,16 +20,21 @@ import (
 	"context"
 
 	pkgreconciler "knative.dev/pkg/reconciler"
+	"knative.dev/pkg/tracker"
 	v1 "knative.dev/serving/pkg/apis/serving/v1"
 	clientset "knative.dev/serving/pkg/client/clientset/versioned"
 	routereconciler "knative.dev/serving/pkg/client/injection/reconciler/serving/v1/route"
 	listers "knative.dev/serving/pkg/client/listers/serving/v1"
+	labelerv1 "knative.dev/serving/pkg/reconciler/labeler/v1"
 )
 
 // Reconciler implements controller.Reconciler for Route resources.
 type Reconciler struct {
 	client clientset.Interface
 
+	// tracker to track revisions and configurations
+	tracker tracker.Interface
+
 	// Listers index properties about resources
 	configurationLister listers.ConfigurationLister
 	revisionLister      listers.RevisionLister
@@ -40,9 +45,13 @@ var _ routereconciler.Interface = (*Reconciler)(nil)
 var _ routereconciler.Finalizer = (*Reconciler)(nil)
 
 func (c *Reconciler) FinalizeKind(ctx context.Context, r *v1.Route) pkgreconciler.Event {
-	return c.clearLabels(ctx, r.Namespace, r.Name)
+	cacc := labelerv1.NewConfigurationAccessor(c.client, c.tracker, c.configurationLister)
+	racc := labelerv1.NewRevisionAccessor(c.client, c.tracker, c.revisionLister)
+	return labelerv1.ClearLabels(r.Namespace, r.Name, cacc, racc)
 }
 
 func (c *Reconciler) ReconcileKind(ctx context.Context, r *v1.Route) pkgreconciler.Event {
-	return c.syncLabels(ctx, r)
+	cacc := labelerv1.NewConfigurationAccessor(c.client, c.tracker, c.configurationLister)
+	racc := labelerv1.NewRevisionAccessor(c.client, c.tracker, c.revisionLister)
+	return labelerv1.SyncLabels(r, cacc, racc)
 }
diff --git a/pkg/reconciler/labeler/v1/accessors.go b/pkg/reconciler/labeler/v1/accessors.go
new file mode 100644
index 000000000..f29975d20
--- /dev/null
+++ b/pkg/reconciler/labeler/v1/accessors.go
@@ -0,0 +1,134 @@
+/*
+Copyright 2020 The Knative Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1
+
+import (
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/types"
+	"knative.dev/pkg/kmeta"
+
+	"knative.dev/pkg/tracker"
+	"knative.dev/serving/pkg/apis/serving"
+	clientset "knative.dev/serving/pkg/client/clientset/versioned"
+	listers "knative.dev/serving/pkg/client/listers/serving/v1"
+)
+
+// Accessor defines an abstraction for manipulating labeled entity
+// (Configuration, Revision) with shared logic.
+type Accessor interface {
+	get(ns, name string) (kmeta.Accessor, error)
+	list(ns, name string) ([]kmeta.Accessor, error)
+	patch(ns, name string, pt types.PatchType, p []byte) error
+}
+
+// Revision is an implementation of Accessor for Revisions.
+type Revision struct {
+	client         clientset.Interface
+	tracker        tracker.Interface
+	revisionLister listers.RevisionLister
+}
+
+// Revision implements Accessor
+var _ Accessor = (*Revision)(nil)
+
+// NewRevisionAccessor is a factory function to make a new revision accessor.
+func NewRevisionAccessor(
+	client clientset.Interface,
+	tracker tracker.Interface,
+	lister listers.RevisionLister) *Revision {
+	return &Revision{
+		client:         client,
+		tracker:        tracker,
+		revisionLister: lister,
+	}
+}
+
+// get implements Accessor
+func (r *Revision) get(ns, name string) (kmeta.Accessor, error) {
+	return r.revisionLister.Revisions(ns).Get(name)
+}
+
+// list implements Accessor
+func (r *Revision) list(ns, name string) ([]kmeta.Accessor, error) {
+	rl, err := r.revisionLister.Revisions(ns).List(labels.SelectorFromSet(labels.Set{
+		serving.RouteLabelKey: name,
+	}))
+	if err != nil {
+		return nil, err
+	}
+	// Need a copy to change types in Go
+	kl := make([]kmeta.Accessor, 0, len(rl))
+	for _, r := range rl {
+		kl = append(kl, r)
+	}
+	return kl, err
+}
+
+// patch implements Accessor
+func (r *Revision) patch(ns, name string, pt types.PatchType, p []byte) error {
+	_, err := r.client.ServingV1().Revisions(ns).Patch(name, pt, p)
+	return err
+}
+
+// Configuration is an implementation of Accessor for Configurations.
+type Configuration struct {
+	client              clientset.Interface
+	tracker             tracker.Interface
+	configurationLister listers.ConfigurationLister
+}
+
+// Configuration implements Accessor
+var _ Accessor = (*Configuration)(nil)
+
+// NewConfigurationAccessor is a factory function to make a new configuration Accessor.
+func NewConfigurationAccessor(
+	client clientset.Interface,
+	tracker tracker.Interface,
+	lister listers.ConfigurationLister) *Configuration {
+	return &Configuration{
+		client:              client,
+		tracker:             tracker,
+		configurationLister: lister,
+	}
+}
+
+// get implements Accessor
+func (c *Configuration) get(ns, name string) (kmeta.Accessor, error) {
+	return c.configurationLister.Configurations(ns).Get(name)
+}
+
+// list implements Accessor
+func (c *Configuration) list(ns, name string) ([]kmeta.Accessor, error) {
+	rl, err := c.configurationLister.Configurations(ns).List(labels.SelectorFromSet(labels.Set{
+		serving.RouteLabelKey: name,
+	}))
+	if err != nil {
+		return nil, err
+	}
+	// Need a copy to change types in Go
+	kl := make([]kmeta.Accessor, 0, len(rl))
+	for _, r := range rl {
+		kl = append(kl, r)
+	}
+	return kl, err
+}
+
+// patch implements Accessor
+func (c *Configuration) patch(ns, name string, pt types.PatchType, p []byte) error {
+	_, err := c.client.ServingV1().Configurations(ns).Patch(name, pt, p)
+	return err
+}
diff --git a/pkg/reconciler/labeler/labels.go b/pkg/reconciler/labeler/v1/labels.go
similarity index 71%
rename from pkg/reconciler/labeler/labels.go
rename to pkg/reconciler/labeler/v1/labels.go
index 9a9566483..e6b16e9dc 100644
--- a/pkg/reconciler/labeler/labels.go
+++ b/pkg/reconciler/labeler/v1/labels.go
@@ -14,10 +14,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package labeler
+package v1
 
 import (
-	"context"
 	"encoding/json"
 	"fmt"
 
@@ -25,14 +24,15 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"knative.dev/pkg/kmeta"
+	"knative.dev/pkg/tracker"
 
 	"knative.dev/serving/pkg/apis/serving"
 	v1 "knative.dev/serving/pkg/apis/serving/v1"
 )
 
-// syncLabels makes sure that the revisions and configurations referenced from
+// SyncLabels makes sure that the revisions and configurations referenced from
 // a Route are labeled with route labels.
-func (c *Reconciler) syncLabels(ctx context.Context, r *v1.Route) error {
+func SyncLabels(r *v1.Route, cacc *Configuration, racc *Revision) error {
 	revisions := sets.NewString()
 	configs := sets.NewString()
 
@@ -43,11 +43,16 @@ func (c *Reconciler) syncLabels(ctx context.Context, r *v1.Route) error {
 		configName := tt.ConfigurationName
 
 		if revName != "" {
-			rev, err := c.revisionLister.Revisions(r.Namespace).Get(revName)
-			if err != nil {
+			if err := racc.tracker.TrackReference(ref(r.Namespace, revName, "Revision"), r); err != nil {
 				return err
 			}
 
+			rev, err := racc.get(r.Namespace, revName)
+			if err != nil {
+				// The revision might not exist (yet). The informers will notify if it gets created.
+				continue
+			}
+
 			revisions.Insert(revName)
 
 			// If the owner reference is a configuration, treat it like a configuration target
@@ -57,11 +62,16 @@ func (c *Reconciler) syncLabels(ctx context.Context, r *v1.Route) error {
 		}
 
 		if configName != "" {
-			config, err := c.configurationLister.Configurations(r.Namespace).Get(configName)
-			if err != nil {
+			if err := cacc.tracker.TrackReference(ref(r.Namespace, configName, "Configuration"), r); err != nil {
 				return err
 			}
 
+			config, err := cacc.configurationLister.Configurations(r.Namespace).Get(configName)
+			if err != nil {
+				// The config might not exist (yet). The informers will notify if it gets created.
+				continue
+			}
+
 			configs.Insert(configName)
 
 			// If the target is for the latest revision, add the latest created revision to the list
@@ -73,8 +83,7 @@ func (c *Reconciler) syncLabels(ctx context.Context, r *v1.Route) error {
 	}
 
 	// Use a revision accessor to manipulate the revisions.
-	racc := &revision{r: c}
-	if err := deleteLabelForNotListed(ctx, r.Namespace, r.Name, racc, revisions); err != nil {
+	if err := deleteLabelForNotListed(r.Namespace, r.Name, racc, revisions); err != nil {
 		return err
 	}
 	if err := setLabelForListed(r, racc, revisions); err != nil {
@@ -82,26 +91,25 @@ func (c *Reconciler) syncLabels(ctx context.Context, r *v1.Route) error {
 	}
 
 	// Use a config access to manipulate the configs.
-	cacc := &configuration{r: c}
-	if err := deleteLabelForNotListed(ctx, r.Namespace, r.Name, cacc, configs); err != nil {
+	if err := deleteLabelForNotListed(r.Namespace, r.Name, cacc, configs); err != nil {
 		return err
 	}
 	return setLabelForListed(r, cacc, configs)
 }
 
-// clearLabels removes any labels for a named route from configurations and revisions.
-func (c *Reconciler) clearLabels(ctx context.Context, ns, name string) error {
-	racc := &revision{r: c}
-	if err := deleteLabelForNotListed(ctx, ns, name, racc, sets.NewString()); err != nil {
-		return err
+// ClearLabels removes any labels for a named route from given accessors.
+func ClearLabels(ns, name string, accs ...Accessor) error {
+	for _, acc := range accs {
+		if err := deleteLabelForNotListed(ns, name, acc, nil /*none listed*/); err != nil {
+			return err
+		}
 	}
-	cacc := &configuration{r: c}
-	return deleteLabelForNotListed(ctx, ns, name, cacc, sets.NewString())
+	return nil
 }
 
 // setLabelForListed uses the accessor to attach the label for this route to every element
 // listed within "names" in the same namespace.
-func setLabelForListed(route *v1.Route, acc accessor, names sets.String) error {
+func setLabelForListed(route *v1.Route, acc Accessor, names sets.String) error {
 	for name := range names {
 		elt, err := acc.get(route.Namespace, name)
 		if err != nil {
@@ -127,7 +135,7 @@ func setLabelForListed(route *v1.Route, acc accessor, names sets.String) error {
 // deleteLabelForNotListed uses the accessor to delete the label from any listable entity that is
 // not named within our list.  Unlike setLabelForListed, this function takes ns/name instead of a
 // Route so that it can clean things up when a Route ceases to exist.
-func deleteLabelForNotListed(ctx context.Context, ns, name string, acc accessor, names sets.String) error {
+func deleteLabelForNotListed(ns, name string, acc Accessor, names sets.String) error {
 	oldList, err := acc.list(ns, name)
 	if err != nil {
 		return err
@@ -151,7 +159,7 @@ func deleteLabelForNotListed(ctx context.Context, ns, name string, acc accessor,
 // setRouteLabel toggles the route label on the specified element through the provided accessor.
 // a nil route name will cause the route label to be deleted, and a non-nil route will cause
 // that route name to be attached to the element.
-func setRouteLabel(acc accessor, elt kmeta.Accessor, routeName *string) error {
+func setRouteLabel(acc Accessor, elt kmeta.Accessor, routeName *string) error {
 	mergePatch := map[string]interface{}{
 		"metadata": map[string]interface{}{
 			"labels": map[string]interface{}{
@@ -167,3 +175,13 @@ func setRouteLabel(acc accessor, elt kmeta.Accessor, routeName *string) error {
 
 	return acc.patch(elt.GetNamespace(), elt.GetName(), types.MergePatchType, patch)
 }
+
+func ref(namespace, name, kind string) tracker.Reference {
+	apiVersion, kind := v1.SchemeGroupVersion.WithKind(kind).ToAPIVersionAndKind()
+	return tracker.Reference{
+		APIVersion: apiVersion,
+		Kind:       kind,
+		Name:       name,
+		Namespace:  namespace,
+	}
+}
diff --git a/pkg/reconciler/route/traffic/traffic.go b/pkg/reconciler/route/traffic/traffic.go
index 1c56b75e3..c4389dcdc 100644
--- a/pkg/reconciler/route/traffic/traffic.go
+++ b/pkg/reconciler/route/traffic/traffic.go
@@ -82,7 +82,10 @@ type Config struct {
 func BuildTrafficConfiguration(configLister listers.ConfigurationLister, revLister listers.RevisionLister,
 	r *v1.Route) (*Config, error) {
 	builder := newBuilder(configLister, revLister, r.Namespace, len(r.Spec.Traffic))
-	builder.applySpecTraffic(r.Spec.Traffic)
+	err := builder.applySpecTraffic(r.Spec.Traffic)
+	if err != nil {
+		return nil, err
+	}
 	return builder.build()
 }
 


Test patch:
diff --git a/pkg/queue/request_metric_test.go b/pkg/queue/request_metric_test.go
index 5c3db10df..81aaa1456 100644
--- a/pkg/queue/request_metric_test.go
+++ b/pkg/queue/request_metric_test.go
@@ -57,6 +57,7 @@ func TestRequestMetricsHandler(t *testing.T) {
 		metricskey.ContainerName:          "queue-proxy",
 		metricskey.LabelResponseCode:      "200",
 		metricskey.LabelResponseCodeClass: "2xx",
+		"tag":                             disabledTagName,
 	}
 
 	metricstest.CheckCountData(t, "request_count", wantTags, 1)
@@ -69,6 +70,60 @@ func TestRequestMetricsHandler(t *testing.T) {
 	metricstest.CheckDistributionCount(t, "request_latencies", wantTags, 1)
 }
 
+func TestRequestMetricsHandlerWithEnablingTagOnRequestMetrics(t *testing.T) {
+	defer reset()
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})
+	handler, err := NewRequestMetricsHandler(baseHandler, "ns", "svc", "cfg", "rev", "pod")
+	if err != nil {
+		t.Fatalf("Failed to create handler: %v", err)
+	}
+
+	resp := httptest.NewRecorder()
+	req := httptest.NewRequest(http.MethodPost, targetURI, bytes.NewBufferString("test"))
+	req.Header.Set(network.TagHeaderName, "test-tag")
+
+	handler.ServeHTTP(resp, req)
+
+	wantTags := map[string]string{
+		metricskey.LabelNamespaceName:     "ns",
+		metricskey.LabelRevisionName:      "rev",
+		metricskey.LabelServiceName:       "svc",
+		metricskey.LabelConfigurationName: "cfg",
+		metricskey.PodName:                "pod",
+		metricskey.ContainerName:          "queue-proxy",
+		metricskey.LabelResponseCode:      "200",
+		metricskey.LabelResponseCodeClass: "2xx",
+		"tag":                             "test-tag",
+	}
+
+	metricstest.CheckCountData(t, "request_count", wantTags, 1)
+
+	// Testing for default route
+	reset()
+	handler, err = NewRequestMetricsHandler(baseHandler, "ns", "svc", "cfg", "rev", "pod")
+	req.Header.Del(network.TagHeaderName)
+	req.Header.Set(network.DefaultRouteHeaderName, "true")
+	handler.ServeHTTP(resp, req)
+	wantTags["tag"] = defaultTagName
+	metricstest.CheckCountData(t, "request_count", wantTags, 1)
+
+	reset()
+	handler, err = NewRequestMetricsHandler(baseHandler, "ns", "svc", "cfg", "rev", "pod")
+	req.Header.Set(network.TagHeaderName, "test-tag")
+	req.Header.Set(network.DefaultRouteHeaderName, "true")
+	handler.ServeHTTP(resp, req)
+	wantTags["tag"] = undefinedTagName
+	metricstest.CheckCountData(t, "request_count", wantTags, 1)
+
+	reset()
+	handler, err = NewRequestMetricsHandler(baseHandler, "ns", "svc", "cfg", "rev", "pod")
+	req.Header.Set(network.TagHeaderName, "test-tag")
+	req.Header.Set(network.DefaultRouteHeaderName, "false")
+	handler.ServeHTTP(resp, req)
+	wantTags["tag"] = "test-tag"
+	metricstest.CheckCountData(t, "request_count", wantTags, 1)
+}
+
 func reset() {
 	metricstest.Unregister(
 		requestCountM.Name(), appRequestCountM.Name(),
@@ -101,6 +156,7 @@ func TestRequestMetricsHandlerPanickingHandler(t *testing.T) {
 			metricskey.ContainerName:          "queue-proxy",
 			metricskey.LabelResponseCode:      "500",
 			metricskey.LabelResponseCodeClass: "5xx",
+			"tag":                             disabledTagName,
 		}
 		metricstest.CheckCountData(t, "request_count", wantTags, 1)
 		metricstest.CheckDistributionCount(t, "request_latencies", wantTags, 1) // Dummy latency range.
diff --git a/pkg/reconciler/labeler/labeler_test.go b/pkg/reconciler/labeler/labeler_test.go
index 2166e78d6..02139fb70 100644
--- a/pkg/reconciler/labeler/labeler_test.go
+++ b/pkg/reconciler/labeler/labeler_test.go
@@ -308,6 +308,7 @@ func TestReconcile(t *testing.T) {
 			client:              servingclient.Get(ctx),
 			configurationLister: listers.GetConfigurationLister(),
 			revisionLister:      listers.GetRevisionLister(),
+			tracker:             &NullTracker{},
 		}
 
 		return routereconciler.NewReconciler(ctx, logging.FromContext(ctx), servingclient.Get(ctx),
diff --git a/pkg/reconciler/revision/resources/queue_test.go b/pkg/reconciler/revision/resources/queue_test.go
index 28f4a5d03..65a174172 100644
--- a/pkg/reconciler/revision/resources/queue_test.go
+++ b/pkg/reconciler/revision/resources/queue_test.go
@@ -80,6 +80,7 @@ func TestMakeQueueContainer(t *testing.T) {
 		name string
 		rev  *v1.Revision
 		lc   logging.Config
+		nc   network.Config
 		oc   metrics.ObservabilityConfig
 		dc   deployment.Config
 		want corev1.Container
diff --git a/pkg/reconciler/route/traffic/traffic_test.go b/pkg/reconciler/route/traffic/traffic_test.go
index f77b7e7ff..1788175ee 100644
--- a/pkg/reconciler/route/traffic/traffic_test.go
+++ b/pkg/reconciler/route/traffic/traffic_test.go
@@ -17,6 +17,8 @@ package traffic
 
 import (
 	"context"
+	"errors"
+	"log"
 	"os"
 	"testing"
 	"time"
@@ -24,6 +26,7 @@ import (
 	"github.com/google/go-cmp/cmp"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
 
 	net "knative.dev/networking/pkg/apis/networking"
@@ -80,6 +83,9 @@ var (
 	configLister listers.ConfigurationLister
 	revLister    listers.RevisionLister
 
+	// fake lister to return API error.
+	revErrorLister listers.RevisionLister
+
 	cmpOpts = []cmp.Option{cmp.AllowUnexported(Config{})}
 )
 
@@ -99,6 +105,8 @@ func setUp() {
 	revInformer := servingInformer.Serving().V1().Revisions()
 	revLister = revInformer.Lister()
 
+	revErrorLister = revFakeErrorLister{}
+
 	// Add these test objects to the informers.
 	objs := []runtime.Object{
 		unreadyConfig, unreadyRev,
@@ -897,6 +905,36 @@ func TestBuildTrafficConfiguration_MissingRevision(t *testing.T) {
 	}
 }
 
+var apiErr = errors.New("failed to connect API")
+
+type revFakeErrorLister struct {
+	listerErr error
+}
+
+func (l revFakeErrorLister) Get(name string) (*v1.Revision, error) {
+	return nil, apiErr
+}
+
+func (l revFakeErrorLister) List(selector labels.Selector) ([]*v1.Revision, error) {
+	log.Panic("not implemented")
+	return nil, nil
+}
+
+func (l revFakeErrorLister) Revisions(namespace string) listers.RevisionNamespaceLister {
+	return l
+}
+
+func TestBuildTrafficConfiguration_FailedGetRevision(t *testing.T) {
+	_, err := BuildTrafficConfiguration(configLister, revErrorLister, testRouteWithTrafficTargets(WithSpecTraffic(v1.TrafficTarget{
+		RevisionName: goodNewRev.Name,
+		Percent:      ptr.Int64(50)})))
+	if err != nil && err.Error() != apiErr.Error() {
+		t.Errorf("err: %s, want: %s", err.Error(), apiErr.Error())
+	} else if err == nil {
+		t.Errorf("err: %s, want: no error", apiErr.Error())
+	}
+}
+
 func TestRoundTripping(t *testing.T) {
 	expected := []v1.TrafficTarget{{
 		RevisionName:   goodOldRev.Name,
diff --git a/test/conformance/api/v1/revision_timeout_test.go b/test/conformance/api/v1/revision_timeout_test.go
index b2fe6c5d7..a05127f69 100644
--- a/test/conformance/api/v1/revision_timeout_test.go
+++ b/test/conformance/api/v1/revision_timeout_test.go
@@ -28,7 +28,6 @@ import (
 	pkgTest "knative.dev/pkg/test"
 	resourcenames "knative.dev/serving/pkg/reconciler/revision/resources/names"
 	"knative.dev/serving/test"
-	"knative.dev/serving/test/e2e"
 	v1test "knative.dev/serving/test/v1"
 
 	. "knative.dev/serving/pkg/testing/v1"
@@ -117,7 +116,7 @@ func TestRevisionTimeout(t *testing.T) {
 
 			if tc.shouldScaleTo0 {
 				t.Log("Waiting to scale down to 0")
-				if err := e2e.WaitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
+				if err := WaitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
 					t.Fatal("Could not scale to zero:", err)
 				}
 			} else {
diff --git a/test/conformance/api/v1/util.go b/test/conformance/api/v1/util.go
index 7b2a5d7af..fc9dc1334 100644
--- a/test/conformance/api/v1/util.go
+++ b/test/conformance/api/v1/util.go
@@ -23,6 +23,9 @@ import (
 	"net/http"
 	"net/url"
 	"strings"
+	"time"
+
+	appsv1 "k8s.io/api/apps/v1"
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
@@ -34,6 +37,10 @@ import (
 	v1test "knative.dev/serving/test/v1"
 )
 
+const (
+	scaleToZeroGracePeriod = 30 * time.Second
+)
+
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
 	if err != nil {
@@ -340,3 +347,21 @@ func validateImageDigest(imageName string, imageDigest string) (bool, error) {
 
 	return ref.Context().String() == digest.Context().String(), nil
 }
+
+// WaitForScaleToZero will wait for the specified deployment to scale to 0 replicas.
+// Will wait up to 6 times the scaleToZeroGracePeriod (30 seconds) before failing.
+func WaitForScaleToZero(t pkgTest.TLegacy, deploymentName string, clients *test.Clients) error {
+	t.Helper()
+	t.Logf("Waiting for %q to scale to zero", deploymentName)
+
+	return pkgTest.WaitForDeploymentState(
+		clients.KubeClient,
+		deploymentName,
+		func(d *appsv1.Deployment) (bool, error) {
+			return d.Status.ReadyReplicas == 0, nil
+		},
+		"DeploymentIsScaledDown",
+		test.ServingNamespace,
+		scaleToZeroGracePeriod*6,
+	)
+}
diff --git a/test/conformance/api/v1alpha1/revision_timeout_test.go b/test/conformance/api/v1alpha1/revision_timeout_test.go
index b2e829aed..2b72478c3 100644
--- a/test/conformance/api/v1alpha1/revision_timeout_test.go
+++ b/test/conformance/api/v1alpha1/revision_timeout_test.go
@@ -28,7 +28,6 @@ import (
 	pkgTest "knative.dev/pkg/test"
 	resourcenames "knative.dev/serving/pkg/reconciler/revision/resources/names"
 	"knative.dev/serving/test"
-	"knative.dev/serving/test/e2e"
 	v1a1test "knative.dev/serving/test/v1alpha1"
 
 	. "knative.dev/serving/pkg/testing/v1alpha1"
@@ -116,7 +115,7 @@ func TestRevisionTimeout(t *testing.T) {
 
 			if tc.shouldScaleTo0 {
 				t.Log("Waiting to scale down to 0")
-				if err := e2e.WaitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
+				if err := waitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
 					t.Fatal("Could not scale to zero:", err)
 				}
 			} else {
diff --git a/test/conformance/api/v1alpha1/util.go b/test/conformance/api/v1alpha1/util.go
index c31677d92..d63bc34b6 100644
--- a/test/conformance/api/v1alpha1/util.go
+++ b/test/conformance/api/v1alpha1/util.go
@@ -23,6 +23,9 @@ import (
 	"net/http"
 	"net/url"
 	"strings"
+	"time"
+
+	appsv1 "k8s.io/api/apps/v1"
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
@@ -34,6 +37,10 @@ import (
 	v1a1test "knative.dev/serving/test/v1alpha1"
 )
 
+const (
+	scaleToZeroGracePeriod = 30 * time.Second
+)
+
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
 	if err != nil {
@@ -340,3 +347,21 @@ func validateImageDigest(imageName string, imageDigest string) (bool, error) {
 
 	return ref.Context().String() == digest.Context().String(), nil
 }
+
+// waitForScaleToZero will wait for the specified deployment to scale to 0 replicas.
+// Will wait up to 6 times the scaleToZeroGracePeriod (30 seconds) before failing.
+func waitForScaleToZero(t pkgTest.TLegacy, deploymentName string, clients *test.Clients) error {
+	t.Helper()
+	t.Logf("Waiting for %q to scale to zero", deploymentName)
+
+	return pkgTest.WaitForDeploymentState(
+		clients.KubeClient,
+		deploymentName,
+		func(d *appsv1.Deployment) (bool, error) {
+			return d.Status.ReadyReplicas == 0, nil
+		},
+		"DeploymentIsScaledDown",
+		test.ServingNamespace,
+		scaleToZeroGracePeriod*6,
+	)
+}
diff --git a/test/conformance/api/v1beta1/revision_timeout_test.go b/test/conformance/api/v1beta1/revision_timeout_test.go
index 2d2909c46..a470a197c 100644
--- a/test/conformance/api/v1beta1/revision_timeout_test.go
+++ b/test/conformance/api/v1beta1/revision_timeout_test.go
@@ -28,7 +28,6 @@ import (
 	pkgTest "knative.dev/pkg/test"
 	resourcenames "knative.dev/serving/pkg/reconciler/revision/resources/names"
 	"knative.dev/serving/test"
-	"knative.dev/serving/test/e2e"
 	v1b1test "knative.dev/serving/test/v1beta1"
 
 	. "knative.dev/serving/pkg/testing/v1beta1"
@@ -116,7 +115,7 @@ func TestRevisionTimeout(t *testing.T) {
 
 			if tc.shouldScaleTo0 {
 				t.Log("Waiting to scale down to 0")
-				if err := e2e.WaitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
+				if err := waitForScaleToZero(t, resourcenames.Deployment(resources.Revision), clients); err != nil {
 					t.Fatal("Could not scale to zero:", err)
 				}
 			} else {
diff --git a/test/conformance/api/v1beta1/util.go b/test/conformance/api/v1beta1/util.go
index 44b383f02..8e0d85f68 100644
--- a/test/conformance/api/v1beta1/util.go
+++ b/test/conformance/api/v1beta1/util.go
@@ -23,6 +23,9 @@ import (
 	"net/http"
 	"net/url"
 	"strings"
+	"time"
+
+	appsv1 "k8s.io/api/apps/v1"
 
 	"github.com/google/go-containerregistry/pkg/name"
 	"golang.org/x/sync/errgroup"
@@ -34,6 +37,10 @@ import (
 	v1b1test "knative.dev/serving/test/v1beta1"
 )
 
+const (
+	scaleToZeroGracePeriod = 30 * time.Second
+)
+
 func waitForExpectedResponse(t pkgTest.TLegacy, clients *test.Clients, url *url.URL, expectedResponse string) error {
 	client, err := pkgTest.NewSpoofingClient(clients.KubeClient, t.Logf, url.Hostname(), test.ServingFlags.ResolvableDomain, test.AddRootCAtoTransport(t.Logf, clients, test.ServingFlags.Https))
 	if err != nil {
@@ -340,3 +347,21 @@ func validateImageDigest(imageName string, imageDigest string) (bool, error) {
 
 	return ref.Context().String() == digest.Context().String(), nil
 }
+
+// waitForScaleToZero will wait for the specified deployment to scale to 0 replicas.
+// Will wait up to 6 times the scaleToZeroGracePeriod (30 seconds) before failing.
+func waitForScaleToZero(t pkgTest.TLegacy, deploymentName string, clients *test.Clients) error {
+	t.Helper()
+	t.Logf("Waiting for %q to scale to zero", deploymentName)
+
+	return pkgTest.WaitForDeploymentState(
+		clients.KubeClient,
+		deploymentName,
+		func(d *appsv1.Deployment) (bool, error) {
+			return d.Status.ReadyReplicas == 0, nil
+		},
+		"DeploymentIsScaledDown",
+		test.ServingNamespace,
+		scaleToZeroGracePeriod*6,
+	)
+}
diff --git a/test/conformance/runtime/readiness_probe_test.go b/test/conformance/runtime/readiness_probe_test.go
index 6d48730db..479e668dd 100644
--- a/test/conformance/runtime/readiness_probe_test.go
+++ b/test/conformance/runtime/readiness_probe_test.go
@@ -26,7 +26,7 @@ import (
 	revisionresourcenames "knative.dev/serving/pkg/reconciler/revision/resources/names"
 	v1opts "knative.dev/serving/pkg/testing/v1"
 	"knative.dev/serving/test"
-	"knative.dev/serving/test/e2e"
+	v1conf "knative.dev/serving/test/conformance/api/v1"
 	v1test "knative.dev/serving/test/v1"
 )
 
@@ -85,7 +85,7 @@ func TestProbeRuntime(t *testing.T) {
 				t.Fatalf("Failed to create initial Service: %v: %v", names.Service, err)
 			}
 			// Check if scaling down works even if access from liveness probe exists.
-			if err := e2e.WaitForScaleToZero(t, revisionresourcenames.Deployment(resources.Revision), clients); err != nil {
+			if err := v1conf.WaitForScaleToZero(t, revisionresourcenames.Deployment(resources.Revision), clients); err != nil {
 				t.Fatal("Could not scale to zero:", err)
 			}
 		})
diff --git a/test/e2e/autoscale_test.go b/test/e2e/autoscale_test.go
index 35839617d..ce08f58ad 100644
--- a/test/e2e/autoscale_test.go
+++ b/test/e2e/autoscale_test.go
@@ -207,8 +207,8 @@ func TestFastScaleToZero(t *testing.T) {
 	epsN := epsL.Items[0].Name
 	t.Logf("Waiting for emptying of %q ", epsN)
 
-	// The first thing that happens when pods are starting to terminate,
-	// if that they stop being ready and endpoints controller removes them
+	// The first thing that happens when pods are starting to terminate
+	// is that they stop being ready and endpoints controller removes them
 	// from the ready set.
 	// While pod termination itself can last quite some time (our pod termination
 	// test allows for up to a minute). The 15s delay is based upon maximum