Repository: knative/serving
Pull request #8533 (2020-07-01 11:39:55)

Add StaticTimeoutFunc handler and speed up timeout handler tests.
<!--
Request Prow to automatically lint any go code in this PR:

/lint
-->

<!-- Please include the 'why' behind your changes if no issue exists -->
## Proposed Changes

- Add a `StaticTimeoutFunc` helper to easily return static timeouts.
- Clean up the tests to make them quicker (used to take 10s, now < 10ms).

**Release Note**

<!-- Enter your extended release note in the below block. If the PR requires
additional action from users switching to the new release, include the string
"action required". If no release note is required, write "NONE". -->

```release-note
NONE
```

/assign @JRBANCEL 

You've added the sleep in the test. Would you elaborate on why that's needed or can we ditch it?


knative-prow-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/serving/pull/8533#" title="Author self-approved">markusthoemmes</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fserving).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[cmd/queue/OWNERS](https://github.com/knative/serving/blob/master/cmd/queue/OWNERS)~~ [markusthoemmes]
- ~~[pkg/http/OWNERS](https://github.com/knative/serving/blob/master/pkg/http/OWNERS)~~ [markusthoemmes]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-serving-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/http/handler/timeout.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-go-coverage/1278292409318379522/artifacts/line-cov.html#file0) | 93.6% | 93.9% | 0.3

knative-prow-robot: @markusthoemmes: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-integration-tests | 002b30e8e42b895d9041eff5c95ea20822a83e3d | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-integration-tests/1278292409318379521) | `/test pull-knative-serving-integration-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/8533). [Your PR dashboard](https://gubernator.knative.dev/pr/markusthoemmes).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-test-reporter-robot: <!--[AUTOMATED-FLAKY-RETRYER]002b30e8e42b895d9041eff5c95ea20822a83e3d[AUTOMATED-FLAKY-RETRYER]-->
The following jobs failed:

Test name | Triggers | Retries
--- | --- | ---
pull-knative-serving-integration-tests | [pull-knative-serving-integration-tests](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-integration-tests/1278292409318379521) | 1/3

Automatically retrying due to test flakiness...
/test pull-knative-serving-integration-tests
knative-prow-robot: @markusthoemmes: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-integration-tests | 002b30e8e42b895d9041eff5c95ea20822a83e3d | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-integration-tests/1278302224908292096) | `/test pull-knative-serving-integration-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/8533). [Your PR dashboard](https://gubernator.knative.dev/pr/markusthoemmes).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-test-reporter-robot: <!--[AUTOMATED-FLAKY-RETRYER]002b30e8e42b895d9041eff5c95ea20822a83e3d[AUTOMATED-FLAKY-RETRYER]-->
The following jobs failed:

Test name | Triggers | Retries
--- | --- | ---
pull-knative-serving-integration-tests | [pull-knative-serving-integration-tests](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-integration-tests/1278292409318379521)<br>[pull-knative-serving-integration-tests](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-integration-tests/1278302224908292096) | 2/3

Automatically retrying due to test flakiness...
/test pull-knative-serving-integration-tests
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-serving-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[pkg/http/handler/timeout.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-go-coverage/1278355953309192194/artifacts/line-cov.html#file0) | 93.6% | 93.9% | 0.3

knative-prow-robot: @markusthoemmes: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-upgrade-tests | 9b3a2589ac1fc5fb0faa9f16733e862322ad76ca | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-upgrade-tests/1278355953309192192) | `/test pull-knative-serving-upgrade-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/8533). [Your PR dashboard](https://gubernator.knative.dev/pr/markusthoemmes).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
markusthoemmes: /retest

Unrelated.
knative-prow-robot: @markusthoemmes: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-serving-upgrade-tests | 9b3a2589ac1fc5fb0faa9f16733e862322ad76ca | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8533/pull-knative-serving-upgrade-tests/1278362496528289794) | `/test pull-knative-serving-upgrade-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_serving/8533). [Your PR dashboard](https://gubernator.knative.dev/pr/markusthoemmes).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/cmd/queue/main.go b/cmd/queue/main.go
index 217687234..f747f9527 100644
--- a/cmd/queue/main.go
+++ b/cmd/queue/main.go
@@ -340,9 +340,7 @@ func buildServer(env config, healthState *health.State, rp *readiness.Probe, sta
 	}
 	composedHandler = proxyHandler(breaker, stats, tracingEnabled, composedHandler)
 	composedHandler = queue.ForwardedShimHandler(composedHandler)
-	composedHandler = handler.NewTimeToFirstByteTimeoutHandler(composedHandler, "request timeout", func(*http.Request) time.Duration {
-		return timeout
-	})
+	composedHandler = handler.NewTimeToFirstByteTimeoutHandler(composedHandler, "request timeout", handler.StaticTimeoutFunc(timeout))
 
 	if metricsSupported {
 		composedHandler = requestMetricsHandler(composedHandler, env)
diff --git a/pkg/http/handler/timeout.go b/pkg/http/handler/timeout.go
index 1be9072f1..1c052199e 100644
--- a/pkg/http/handler/timeout.go
+++ b/pkg/http/handler/timeout.go
@@ -31,6 +31,13 @@ import (
 // TimeoutFunc returns the timeout duration to be used by the timeout handler.
 type TimeoutFunc func(req *http.Request) time.Duration
 
+// StaticTimeoutFunc returns a TimeoutFunc that always returns the same duration.
+func StaticTimeoutFunc(timeout time.Duration) TimeoutFunc {
+	return func(req *http.Request) time.Duration {
+		return timeout
+	}
+}
+
 type timeToFirstByteTimeoutHandler struct {
 	handler     http.Handler
 	timeoutFunc TimeoutFunc


Test patch:
diff --git a/pkg/http/handler/timeout_test.go b/pkg/http/handler/timeout_test.go
index 6a3c40ddf..9e53a0263 100644
--- a/pkg/http/handler/timeout_test.go
+++ b/pkg/http/handler/timeout_test.go
@@ -73,28 +73,23 @@ func TestTimeoutWriterErrorsWriteAfterTimeout(t *testing.T) {
 }
 
 func TestTimeToFirstByteTimeoutHandler(t *testing.T) {
-	const longTimeout = 10 * time.Second
-
-	longTimeoutFunc := func(*http.Request) time.Duration {
-		return longTimeout
-	}
-	failingTimeoutFunc := func(*http.Request) time.Duration {
-		return 0 * time.Millisecond
-	}
+	const (
+		immediateTimeout = 0 * time.Millisecond
+		longTimeout      = 1 * time.Minute // Super long, not supposed to hit this.
+	)
 
 	tests := []struct {
-		name               string
-		timeoutFunc        TimeoutFunc
-		handler            func(mux *sync.Mutex, writeErrors chan error) http.Handler
-		timeoutMessage     string
-		wantStatus         int
-		wantBody           string
-		wantWriteError     bool
-		wantPanic          bool
-		sleepBeforeExiting time.Duration
+		name           string
+		timeoutFunc    TimeoutFunc
+		handler        func(mux *sync.Mutex, writeErrors chan error) http.Handler
+		timeoutMessage string
+		wantStatus     int
+		wantBody       string
+		wantWriteError bool
+		wantPanic      bool
 	}{{
 		name:        "all good",
-		timeoutFunc: longTimeoutFunc,
+		timeoutFunc: StaticTimeoutFunc(longTimeout),
 		handler: func(*sync.Mutex, chan error) http.Handler {
 			return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 				w.Write([]byte("hi"))
@@ -104,7 +99,7 @@ func TestTimeToFirstByteTimeoutHandler(t *testing.T) {
 		wantBody:   "hi",
 	}, {
 		name:        "custom timeout message",
-		timeoutFunc: failingTimeoutFunc,
+		timeoutFunc: StaticTimeoutFunc(immediateTimeout),
 		handler: func(mux *sync.Mutex, writeErrors chan error) http.Handler {
 			return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 				mux.Lock()
@@ -119,7 +114,7 @@ func TestTimeToFirstByteTimeoutHandler(t *testing.T) {
 		wantWriteError: true,
 	}, {
 		name:        "propagate panic",
-		timeoutFunc: longTimeoutFunc,
+		timeoutFunc: StaticTimeoutFunc(longTimeout),
 		handler: func(*sync.Mutex, chan error) http.Handler {
 			return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 				panic(http.ErrAbortHandler)
@@ -130,18 +125,17 @@ func TestTimeToFirstByteTimeoutHandler(t *testing.T) {
 		wantPanic:  true,
 	}, {
 		name:        "timeout before panic",
-		timeoutFunc: failingTimeoutFunc,
+		timeoutFunc: StaticTimeoutFunc(immediateTimeout),
 		handler: func(*sync.Mutex, chan error) http.Handler {
 			return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 				time.Sleep(1 * time.Second)
 				panic(http.ErrAbortHandler)
 			})
 		},
-		timeoutMessage:     "request timeout",
-		wantStatus:         http.StatusGatewayTimeout,
-		wantBody:           "request timeout",
-		wantPanic:          false,
-		sleepBeforeExiting: longTimeout,
+		timeoutMessage: "request timeout",
+		wantStatus:     http.StatusGatewayTimeout,
+		wantBody:       "request timeout",
+		wantPanic:      false,
 	}}
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
@@ -180,8 +174,6 @@ func TestTimeToFirstByteTimeoutHandler(t *testing.T) {
 					t.Errorf("Expected a timeout error, got %v", err)
 				}
 			}
-
-			time.Sleep(test.sleepBeforeExiting)
 		})
 	}
 }
diff --git a/test/conformance/api/v1/errorcondition_test.go b/test/conformance/api/v1/errorcondition_test.go
index 96ad04b63..6a64da3c9 100644
--- a/test/conformance/api/v1/errorcondition_test.go
+++ b/test/conformance/api/v1/errorcondition_test.go
@@ -71,12 +71,12 @@ func TestContainerErrorMsg(t *testing.T) {
 	t.Log("When the imagepath is invalid, the Configuration should have error status.")
 
 	// Wait for ServiceState becomes NotReady. It also waits for the creation of Configuration.
-	if err := v1test.WaitForServiceState(clients.ServingClient, names.Service, v1test.IsServiceFailed, "ServiceIsFailed"); err != nil {
+	if err := v1test.WaitForServiceState(clients.ServingClient, names.Service, v1test.IsServiceAndChildrenFailed, "ServiceIsFailed"); err != nil {
 		t.Fatalf("The Service %s was unexpected state: %v", names.Service, err)
 	}
 
 	// Checking for "Container image not present in repository" scenario defined in error condition spec
-	err = v1test.WaitForConfigurationState(clients.ServingClient, names.Config, func(r *v1.Configuration) (bool, error) {
+	err = v1test.CheckConfigurationState(clients.ServingClient, names.Config, func(r *v1.Configuration) (bool, error) {
 		cond := r.Status.GetCondition(v1.ConfigurationConditionReady)
 		if cond != nil && !cond.IsUnknown() {
 			if strings.Contains(cond.Message, manifestUnknown) && cond.IsFalse() {
@@ -87,7 +87,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				names.Config, containerMissing, manifestUnknown, "False", cond.Reason, cond.Message, cond.Status)
 		}
 		return false, nil
-	}, "ContainerImageNotPresent")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate configuration state:", err)
@@ -99,7 +99,7 @@ func TestContainerErrorMsg(t *testing.T) {
 	}
 
 	t.Log("When the imagepath is invalid, the revision should have error status.")
-	err = v1test.WaitForRevisionState(clients.ServingClient, revisionName, func(r *v1.Revision) (bool, error) {
+	err = v1test.CheckRevisionState(clients.ServingClient, revisionName, func(r *v1.Revision) (bool, error) {
 		cond := r.Status.GetCondition(v1.RevisionConditionReady)
 		if cond != nil {
 			if cond.Reason == containerMissing && strings.Contains(cond.Message, manifestUnknown) {
@@ -109,7 +109,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				revisionName, containerMissing, manifestUnknown, cond.Reason, cond.Message)
 		}
 		return false, nil
-	}, "ImagePathInvalid")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate revision state:", err)
diff --git a/test/conformance/api/v1alpha1/errorcondition_test.go b/test/conformance/api/v1alpha1/errorcondition_test.go
index f34fc9924..0247eeda6 100644
--- a/test/conformance/api/v1alpha1/errorcondition_test.go
+++ b/test/conformance/api/v1alpha1/errorcondition_test.go
@@ -69,12 +69,12 @@ func TestContainerErrorMsg(t *testing.T) {
 	t.Log("When the imagepath is invalid, the Configuration should have error status.")
 
 	// Wait for ServiceState becomes NotReady. It also waits for the creation of Configuration.
-	if err := v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceNotReady, "ServiceIsNotReady"); err != nil {
+	if err := v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceAndChildrenFailed, "ServiceIsNotReady"); err != nil {
 		t.Fatalf("The Service %s was unexpected state: %v", names.Service, err)
 	}
 
 	// Checking for "Container image not present in repository" scenario defined in error condition spec
-	err = v1a1test.WaitForConfigurationState(clients.ServingAlphaClient, names.Config, func(r *v1alpha1.Configuration) (bool, error) {
+	err = v1a1test.CheckConfigurationState(clients.ServingAlphaClient, names.Config, func(r *v1alpha1.Configuration) (bool, error) {
 		cond := r.Status.GetCondition(v1alpha1.ConfigurationConditionReady)
 		if cond != nil && !cond.IsUnknown() {
 			if strings.Contains(cond.Message, manifestUnknown) && cond.IsFalse() {
@@ -85,7 +85,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				names.Config, containerMissing, manifestUnknown, "False", cond.Reason, cond.Message, cond.Status)
 		}
 		return false, nil
-	}, "ContainerImageNotPresent")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate configuration state:", err)
@@ -97,7 +97,7 @@ func TestContainerErrorMsg(t *testing.T) {
 	}
 
 	t.Log("When the imagepath is invalid, the revision should have error status.")
-	err = v1a1test.WaitForRevisionState(clients.ServingAlphaClient, revisionName, func(r *v1alpha1.Revision) (bool, error) {
+	err = v1a1test.CheckRevisionState(clients.ServingAlphaClient, revisionName, func(r *v1alpha1.Revision) (bool, error) {
 		cond := r.Status.GetCondition(v1alpha1.RevisionConditionReady)
 		if cond != nil {
 			if cond.Reason == containerMissing && strings.Contains(cond.Message, manifestUnknown) {
@@ -107,7 +107,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				revisionName, containerMissing, manifestUnknown, cond.Reason, cond.Message)
 		}
 		return false, nil
-	}, "ImagePathInvalid")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate revision state:", err)
diff --git a/test/conformance/api/v1beta1/errorcondition_test.go b/test/conformance/api/v1beta1/errorcondition_test.go
index ad4d1f605..c52878475 100644
--- a/test/conformance/api/v1beta1/errorcondition_test.go
+++ b/test/conformance/api/v1beta1/errorcondition_test.go
@@ -71,12 +71,12 @@ func TestContainerErrorMsg(t *testing.T) {
 	t.Log("When the imagepath is invalid, the Configuration should have error status.")
 
 	// Wait for ServiceState becomes NotReady. It also waits for the creation of Configuration.
-	if err := v1b1test.WaitForServiceState(clients.ServingBetaClient, names.Service, v1b1test.IsServiceFailed, "ServiceIsNotReady"); err != nil {
+	if err := v1b1test.WaitForServiceState(clients.ServingBetaClient, names.Service, v1b1test.IsServiceAndChildrenFailed, "ServiceIsNotReady"); err != nil {
 		t.Fatalf("The Service %s was unexpected state: %v", names.Service, err)
 	}
 
 	// Checking for "Container image not present in repository" scenario defined in error condition spec
-	err = v1b1test.WaitForConfigurationState(clients.ServingBetaClient, names.Config, func(r *v1beta1.Configuration) (bool, error) {
+	err = v1b1test.CheckConfigurationState(clients.ServingBetaClient, names.Config, func(r *v1beta1.Configuration) (bool, error) {
 		cond := r.Status.GetCondition(v1beta1.ConfigurationConditionReady)
 		if cond != nil && !cond.IsUnknown() {
 			if strings.Contains(cond.Message, manifestUnknown) && cond.IsFalse() {
@@ -87,7 +87,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				names.Config, containerMissing, manifestUnknown, "False", cond.Reason, cond.Message, cond.Status)
 		}
 		return false, nil
-	}, "ContainerImageNotPresent")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate configuration state:", err)
@@ -99,7 +99,7 @@ func TestContainerErrorMsg(t *testing.T) {
 	}
 
 	t.Log("When the imagepath is invalid, the revision should have error status.")
-	err = v1b1test.WaitForRevisionState(clients.ServingBetaClient, revisionName, func(r *v1beta1.Revision) (bool, error) {
+	err = v1b1test.CheckRevisionState(clients.ServingBetaClient, revisionName, func(r *v1beta1.Revision) (bool, error) {
 		cond := r.Status.GetCondition(v1beta1.RevisionConditionReady)
 		if cond != nil {
 			if cond.Reason == containerMissing && strings.Contains(cond.Message, manifestUnknown) {
@@ -109,7 +109,7 @@ func TestContainerErrorMsg(t *testing.T) {
 				revisionName, containerMissing, manifestUnknown, cond.Reason, cond.Message)
 		}
 		return false, nil
-	}, "ImagePathInvalid")
+	})
 
 	if err != nil {
 		t.Fatal("Failed to validate revision state:", err)
diff --git a/test/v1/service.go b/test/v1/service.go
index 500b1be42..6124d913c 100644
--- a/test/v1/service.go
+++ b/test/v1/service.go
@@ -279,6 +279,30 @@ func IsServiceFailed(s *v1.Service) (bool, error) {
 	return s.IsFailed(), nil
 }
 
+// IsServiceAndChildrenFailed will check the readiness, route and config conditions of the service
+// and return true if they are all failed.
+func IsServiceAndChildrenFailed(s *v1.Service) (bool, error) {
+	if s.Generation != s.Status.ObservedGeneration {
+		return false, nil
+	}
+
+	if failed := s.IsFailed(); !failed {
+		return false, nil
+	}
+
+	routeCond := s.Status.GetCondition(v1.ServiceConditionRoutesReady)
+	if routeCond == nil || routeCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	configCond := s.Status.GetCondition(v1.ServiceConditionConfigurationsReady)
+	if configCond == nil || configCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	return true, nil
+}
+
 // IsServiceRoutesNotReady checks the RoutesReady status of the service and returns true only if RoutesReady is set to False.
 func IsServiceRoutesNotReady(s *v1.Service) (bool, error) {
 	result := s.Status.GetCondition(v1.ServiceConditionRoutesReady)
diff --git a/test/v1alpha1/service.go b/test/v1alpha1/service.go
index 87013713b..14ced52d9 100644
--- a/test/v1alpha1/service.go
+++ b/test/v1alpha1/service.go
@@ -348,6 +348,31 @@ func IsServiceNotReady(s *v1alpha1.Service) (bool, error) {
 	return s.Generation == s.Status.ObservedGeneration && result != nil && result.Status == corev1.ConditionFalse, nil
 }
 
+// IsServiceAndChildrenFailed will check the readiness, route and config conditions of the service
+// and return true if they are all failed.
+func IsServiceAndChildrenFailed(s *v1alpha1.Service) (bool, error) {
+	if s.Generation != s.Status.ObservedGeneration {
+		return false, nil
+	}
+
+	readyCond := s.Status.GetCondition(v1alpha1.ServiceConditionReady)
+	if readyCond == nil || readyCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	routeCond := s.Status.GetCondition(v1alpha1.ServiceConditionRoutesReady)
+	if routeCond == nil || routeCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	configCond := s.Status.GetCondition(v1alpha1.ServiceConditionConfigurationsReady)
+	if configCond == nil || configCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	return true, nil
+}
+
 // IsServiceRoutesNotReady checks the RoutesReady status of the service and returns true only if RoutesReady is set to False.
 func IsServiceRoutesNotReady(s *v1alpha1.Service) (bool, error) {
 	result := s.Status.GetCondition(v1alpha1.ServiceConditionRoutesReady)
diff --git a/test/v1beta1/service.go b/test/v1beta1/service.go
index 6a33cddfa..26e6b264f 100644
--- a/test/v1beta1/service.go
+++ b/test/v1beta1/service.go
@@ -22,6 +22,7 @@ import (
 	"fmt"
 
 	"github.com/mattbaird/jsonpatch"
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/wait"
@@ -255,3 +256,27 @@ func IsServiceReady(s *v1beta1.Service) (bool, error) {
 func IsServiceFailed(s *v1beta1.Service) (bool, error) {
 	return s.IsFailed(), nil
 }
+
+// IsServiceAndChildrenFailed will check the readiness, route and config conditions of the service
+// and return true if they are all failed.
+func IsServiceAndChildrenFailed(s *v1beta1.Service) (bool, error) {
+	if s.Generation != s.Status.ObservedGeneration {
+		return false, nil
+	}
+
+	if failed := s.IsFailed(); !failed {
+		return false, nil
+	}
+
+	routeCond := s.Status.GetCondition(v1.ServiceConditionRoutesReady)
+	if routeCond == nil || routeCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	configCond := s.Status.GetCondition(v1.ServiceConditionConfigurationsReady)
+	if configCond == nil || configCond.Status != corev1.ConditionFalse {
+		return false, nil
+	}
+
+	return true, nil
+}