Repository: deis/deis

Issue:
Design Doc: HTTP Healthcheck on Deploy
refs the new proposal in #4019. All input/feedback on the doc is welcome.
refs #3813

Please note that this document is being maintained at https://gist.github.com/bacongobbler/d2f0ef2311d1e3e28997 to keep a living history of any changes made.

During a typical deployment of an application release, the controller performs a status check from Fleet after it has been deployed to ensure that it is running. However, Fleet is a glorified systemd wrapper, which means that it only checks systemd if the process is running (in this case, `docker run`). If a container's web process takes some time to boot, this may cause a false positive status check from the user's perspective. This proposal introduces a new concept into Deis known as a health check, which will make the controller check Deis' Publisher component for real-time stats on the app to ensure the container is running before recycling the old release.
## Goals

The main goals of this proposal is to
- ensure a zero-downtime deployment between deployments of new releases
- allow the user to configure the URL path, the health check delay, and the health check request timeout value.
## Code Changes

In order for this change to occur, there needs to be a way for users to configure both the expected health check URL as well as the initial delay and timeout value of the request. Since the `Config` object currently exists and the health checks are considered part of an application's configuration, the code changes in the controller is fairly minimal:
- `deis config:set HEALTHCHECK_URL=/` will configure a health check URL for the application
- the expected status code from the health check is a 200 OK. This is non-configurable.
- `deis config:set HEALTHCHECK_PORT=5432` will configure a health check port for the application (default: 5000)
- `deis config:set HEALTHCHECK_INITIAL_DELAY=10` will tell Deis' Publisher to wait 10 seconds before performing a health check (default: 0)
- `deis config:set HEALTHCHECK_TIMEOUT=10` will tell Deis' Publisher that the request may take up to 10 seconds before the app is considered "down" (default: 5).

These keys will be exposed to Etcd at /Deis/config/appname/health check_URL=/ so deis's Publisher can retrieve these values to perform its job.

Once `HEALTHCHECK_URL` is set, the controller will do the typical application release workflow. However, once the controller is notified by Fleet that the new version of the application is "up", the controller will wait a minimum of x+y seconds, where x is equal to the value of HEALTHCHECK_INITIAL_DELAY, and y is equal to the value of HEALTHCHECK_TIMEOUT. By default this is 5 seconds. Once the waiting is over, the controller will then perform an API request against the Publisher to inspect the container's health (more on that below).

On the Publisher's side, before it publishes a container to Etcd, it will check Etcd for these keys. If health check_URL is set, then it will perform the health check. It will also expose an API which the controller can now ping. The API will be very simple:

Example request:

```
GET /up
```

Example Response:

```
[
    "foo_v2.web.1",
    "foo_v2.web.3",
    "bar_v3.cmd.1"
]
```

This response is a list of all containers which Publisher has exposed to Etcd and are considered "up". It will be the responsibility of the controller to parse this response to check if the containers deployed are up based on the container ID. If the health check passes, the rest of the deployment workflow will then continue, recycling old containers and returning a successful response to the client. If only a few are down (few being anything less than all of the new containers), a warning will be raised, but the operation will be considered "passed". However, if health check failed, the controller will respond with an appropriate error and roll back to the old version of the application.
## Issues to Note

With this approach, We have some legacy code in the Publisher which will only publish the latest version of the container at any given time (see #2413 for a history lesson). This was put in place to prevent two existing versions of the same app to be published to the router. We'll have to be extra careful not to break this functionality, but we also have to make sure that we're still publishing the older version of the controller to the load balancer (Deis-router) before the new container passes the health check. For example, if the user has set an initial delay that is higher than the TTL of the Etcd keys, this could cause the old version of the containers to come down, causing downtime. This can be mitigated with proper unit tests within Publisher to ensure that an older version of the container is still getting published at this time.
## Testing

In order for this proposal to work, we need to ensure a zero-downtime deployment, as well as proper errors to be returned when an application fails to respond in time or with an incorrect status code. Since the controller has a great suite for unit tests, most of the workflow for the client will be covered by unit tests to ensure that the proper errors are propagated when certain cases occur. However, the zero-downtime approach will be covered by a combination of unit tests in Deis-Publisher as well as some comprehensive integration tests. How this will be accomplished is by invoking a goroutine that will continuously ping the app URL. If a 404 response shows up (due to #3774), then the deployment failed as a zero-downtime deployment.



Discussion:
A few thoughts:
1. Let's take a closer look at GoogleCloudPlatform/kubernetes#365 and make sure our interface is compatible (right now it's not)
2. I understand publisher has proved a problematic place for this code, but I really don't like the idea of the controller implementing continuous anything.  What about the router?  Aren't there Nginx directives we could use to avoid the blips?
3. It's great that we're taking a test-driven approach to this.  Let's make sure we test for the exact error conditions we see today on a slow starting application (502's, 404's, etc.)

Alright, so, this is a feature I am really looking forward to :+1: 

So far I have one comment:

> This health check will continuously perform an HTTP request every second for 60 seconds.

If possible, make that configurable...us people running Java applications which do classpath scanning, dependency tree building, dependency injection, database initialization and so on so forth during boot, can often take a bit more then 60 seconds to start up (especially on older monolithic applications). I would prefer a default of perhaps 2 or 5 minutes. Are there any good reasons why it should be limited to 1 minute? The routing layer won't timeout for 20 minutes.

> Are there any good reasons why it should be limited to 1 minute?

Simply put, precedence set by Heroku. We can easily make this configurable though, no doubt :)

Perfect. I was also a bit sure about this part:

>  If the health check passes, the rest of the deployment workflow will continue, recycling old containers and returning a successful response to the client. 

My original thought was that this might cause problems if an application doing database migration or something is slow (or, alternatively, connecting to database takes a few seconds too much) (and in that case, maybe allow for a configurable number of health check failures). _But_ I just did a test with a Spring Boot application I have (fyi, Spring Boot ships with a health endpoint solution), and it seemed to handle database migrations fine, and report the correct status all the way (since it didn't serve requests while doing the database migrations).

@gabrtv to address your comments:

> Let's take a closer look at GoogleCloudPlatform/kubernetes#365 and make sure our interface is compatible (right now it's not)

I just finished talking with @carmstrong about this and he said the same thing, so needless to say I'll look into it. :)

> I understand publisher has proved a problematic place for this code, but I really don't like the idea of the controller implementing continuous anything. What about the router? Aren't there Nginx directives we could use to avoid the blips?

Sadly that's an NGINX+ feature, so it's unavailable on the open-source side.

> This health check will continuously perform an HTTP request every second for 60 seconds.

This also made me stop and think. Perhaps that should be the default, but the time should be configurable? And generally speaking I'm also wary of putting functionality in the controller if we can think of another way to do it.

Deis currently does a full A->B deploy, meaning we bring up X containers to replace the X currently running, which is naive and problematic if you don't have basically double the resources needed to run your app containers. It would be useful if that strategy were also configurable, so a user could ask to roll the new release out in groups of 3, for example. But I assume making the release promotion strategy configurable is out of scope for this particular change...

We have a manual "zero downtime" test script that would be a good starting point for the integration tests.

Normally your service discovery agent would be health checking the system and removing services from the load balancer when they go down. 

It would also be used to add them to the load balancer when new containers come online. Is it time to start talking about implementing some type of service discovery agent to fulfill this need?

> Is it time to start talking about implementing some type of service discovery agent to fulfill this need?

That would be [publisher](http://docs.deis.io/en/latest/understanding_deis/components/#publisher). :)

Thank you for your comments. Let me come up with a proper response to answer both of your comments. I need to chew on this for a sec. :tea: 

> This also made me stop and think. Perhaps that should be the default, but the time should be configurable? And generally speaking I'm also wary of putting functionality in the controller if we can think of another way to do it.

Yep! I made the same comment [earlier in this thread](https://github.com/deis/deis/issues/4045#issuecomment-121767735).

> But I assume making the release promotion strategy configurable is out of scope for this particular change...

Yes. Those are nice changes, but it is out of scope.

> Normally your service discovery agent would be health checking the system and removing services from the load balancer when they go down.

I completely agree. However, I have left out the service discovery agent from performing health checks in this design for two reasons:

1) The controller would have no idea if the container is truly "up" or "down", making it difficult to give the app owner a good response about the app's health as well as a true zero-downtime deployment (the controller handles the app's lifecycle).
2) publisher's workflow conflicts with [the controller's workflow](https://github.com/deis/deis/pull/3982#issuecomment-120112608) when we tried it out in an earlier iteration.

However, we could expose an API endpoint from publisher to publish an app's health information. That could be a possible replacement to the entire `_do_healthcheck()` workflow in the controller. The problem with that workflow is yet another coupling between microservices, which may or may not be a bad thing since the controller is meant to be "the mainframe" for the entire platform (IOW as long as it's not the other way around it should be fine). I'll keep chewing on this one.

@jchauncey how would you imagine this would work, from your perspective?

i think in an ideal world i would want this extracted into some external service that is responsible for the overall health of the cluster. It would perform both http and cmd health checks and then take some action  based on that health (like adding/removing them from the load balancer and alerting). I think for the sake of time and knowing that the controller already performs several of the tasks you need I think having it there is probably the best compromise for now. 

I'm a little late to the discussion on this, but... that's never stopped me from weighing in before. ;-)

Kubernetes provides a good pattern for how we want to do health checks.

Containers can expose a healthcheck. There are two probe types that a container provides:
- Liveness: Whether the container is alive (failure = restart). Liveness is used to monitor health of existing pods.
- Readiness: Whether the container is ready to serve responses. Rediness is used to determine whether a pod's containers are all ready to handle requests.

Both probe types have the same data structure. K8s can do three types of probe:
- exec a command in the Docker contaner
- send an HTTP request and expect a 200 response
- make a TCP connection and expect the connection to be successfully established

I think we're mainly interested in HTTP probes right now, though I think exec probes will be useful for workers. An HTTP probe is pretty simple. Checks have the following major attributes:
- initialDelaySeconds: How long to wait before the first liveness probe
- timeOut: How long to wait for a response before marking unhealthy
- scheme, domain, port, path: To construct the URL to query

And that's about it. Here's an example usage from the User Guide:

``` yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - args:
    - /server
    image: gcr.io/google_containers/liveness
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      timeoutSeconds: 1
    name: liveness
```

All HTTP probes are GETs (not configurable) and all expect a 200 status code (not configurable). I personally think this is fine. 

In Kubernetes, health checks are considered part of scheduling. Kublets execute health checks, and when a container fails a healthcheck, a replacement is spun up to take its place.

My opinion is that we should stick pretty close to this definition. While a methodology match would have us doing this as part of Publisher, I could see shunting this to another part of the platform purely for pragmatic reasons.

Workflow-wise, what we'd need is to handle two cases: Standard health polling and zero-downtime upgrades. Health polling is a relatively easy case. But what are the minimum number of changes we'd need to make to implement this style of health check?

## References:
- Pod health: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/pod-states.md
- Description of Liveness Probes (exec and http): https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/user-guide/liveness
- Probe type: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1/types.go#L625
- Liveliness example: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/user-guide/liveness/http-liveness.yaml

Awesome! So to take in your feedback:
- Remove HEALTHCHECK_STATUS_CODE from being configurable, enforce 200 OK
- Keep HEALTHCHECK_URL to match `path`
- Add HEALTHCHECK_INITIAL_DELAY (default = 0), which will have an initial delay of x seconds before performing the health check
- Add HEALTHCHECK_TIMEOUT (default = 60), which will determine how long we will wait until we determine that the endpoint is unhealthy.

I'm not too sure how we can handle the `port` option with our current architecture, since the router listens on port 80 for all apps. Maybe we can set it to 80 by default and expose it as HEALTHCHECK_PORT? This would allow users who have https enabled in their routers to be able to hit their apps as well. How does that sound?

Shouldnt you hit the app directly including the port they expose (not 80). 

like 
`http://<ip of machine here>:5000/healthcheck`

I guess we could, but the controller at the present time does not have knowledge from the scheduler as to which host each instance of the app is running. However, we could try to extrapolate that information from the scheduler and then hit the app instance directly. In fleet's case, that would be the machine's private IP address. For kubernetes, that would likely be the DNS name.

For buildpack apps the user is not supposed to care which port is exposed, but I guess for this workflow we could just document that they should set HEALTHCHECK_PORT to 5000 and be done with it.

So was the thought to healthcheck through the router then?

On Tue, Jul 21, 2015 at 12:13 PM, Matthew Fisher notifications@github.com
wrote:

> I guess we could, but the controller at the present time does not have
> knowledge from the scheduler as to which host each instance of the app is
> running. However, we could try to extrapolate that information from the
> scheduler and then hit the app instance directly. In fleet's case, that
> would be the machine's private IP address. For kubernetes, that would
> likely be the DNS name.
> 
> For buildpack apps the user is not supposed to care which port is exposed,
> but I guess for this workflow we could just document that they should set
> HEALTHCHECK_PORT to 5000 and be done with it.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/deis/deis/issues/4045#issuecomment-123420919.

I think that initially we should not expose configuration for healthcheck port _to users as a config option_. If there's significant demand, then we can re-think that as an addition. This gives us the opportunity to enjoy a little extra flexibility in how we implement this first version.

Also, one thing I didn't make too clear in my earlier post: I think that liveness and readiness checks can also be conflated for the first release(s). I don't think we need to give separate options for both. I think @bacongobbler is assuming that already.

You cant health check an app through the router because you need to know which container is down/up when performing actions against the load balancer. This means you need to know which port the application is running on. So you can either determine that programmatically or you can have them specify it as part of some config detail when creating an application. 

Yeah, I don't think we should rely on the user to set the port.

> So was the thought to healthcheck through the router then?

Yes, the initial thought was to do the healthcheck through the router. If we want to hit each instance directly, how would the readiness checks in the controller look? Would we pick one of the new instances at random and do a health check, or do a health check on each new instance?

> Also, one thing I didn't make too clear in my earlier post: I think that liveness and readiness checks can also be conflated for the first release(s). I don't think we need to give separate options for both. I think @bacongobbler is assuming that already.

Yes, I am assuming that at the moment since we don't have something that acts as a Liveness checker. This discussion is mostly wrapped around emulating a Readiness checker :)

> This means you need to know which port the application is running on. So you can either determine that programmatically or you can have them specify it as part of some config detail when creating an application.
> 
> Yeah, I don't think we should rely on the user to set the port.

OTOH, the user would know which port the app is expecting to expose because it's [right in the Docker image](http://docs.docker.com/reference/builder/#expose). For users who use the Heroku workflow, we can document that this should be port 5000 as specified [here](https://github.com/deis/deis/blob/master/builder/rootfs/usr/local/src/slugrunner/Dockerfile#L12). Alternatively, we can use the Docker API to do this programatically, but then the user won't have a way to say which port to use if the app exposes more than one port.

As the controller launches new instances it should also check those instances first before they ever get added to the load balancer. if an application is flapping you cannot place that container in the rotation. 

The controller should just do a simple http check against the ip of the host and the port that it exposes as its web port (which i thought was not necessarily configurable but not always 5000). 

Like this - https://github.com/RallySoftware/armada/blob/master/lib/armada/connection/health_check.rb

@bacongobbler said:

> OTOH, the user would know which port the app is expecting to expose because it's typically right in the Docker image. For users who use the Heroku workflow, we can document that this should be port 5000 as specified here. Alternatively, we can use the Docker API to do this programatically, but then the user won't have a way to say which port to use if the app exposes more than one port.

Alright, I'm convinced. We can do a HEALTHCHECK_PORT.

@smothiki and @kmala -- anything you'd like to add to this discussion?

I have updated the OP to address changes discussed in this conversation. When you have time, PTAL, otherwise I'll consider this document as final at noon PDT tomorrow and start working on an implementation :)

Hmm, so, if I understand correctly, you will first do a healthcheck at `HEALTHCHECK_INITIAL_DELAY` instead of attempting for up to 60 (configurable) seconds. Now, if, after `HEALTHCHECK_INITIAL_DELAY` the healthcheck fails, then the publish process will be cancelled, right?

With this solution, a user would probably have to configure `HEALTHCHECK_INITIAL_DELAY` at somewhere between 150% and 200% of the actual expected time for the first healthcheck. E.g. if I expect my application to be up and running in 10 seconds, I would configure this probably at like 20 seconds. (In case of new code slowing it down, or database migrations, or other stuff taking extra time). 

This would cause every new deploy to take extra time (20 seconds instead of 10 seconds), which is not a lot, but over time it does add up (+ it's annoying). 

OR, do you mean that you will first wait `HEALTHCHECK_INITIAL_DELAY` and then keep trying for `HEALTHCHECK_TIMEOUT`? 

E.g. if I configure `HEALTHCHECK_INITIAL_DELAY` to 10 seconds (as I expect the application to start in around 10 seconds), and then leave `HEALTHCHECK_TIMEOUT` at it's default 60s, then it will try to establish a connection to `HEALTHCHECK_URL` with 1s connect/data timeout for 60s?

Your first assumption is correct. Take a look at @mbutcher's comment above for context :)

Wait so you will fail the publish if the first check fails?
On Jul 22, 2015 6:07 PM, "Matthew Fisher" notifications@github.com wrote:

> Your first assumption is correct. Take a look at @mbutcher
> https://github.com/mbutcher's comment above for context :)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/deis/deis/issues/4045#issuecomment-123912140.

Yes. The first request after the initial delay should give us the correct response (or a failboat)

I think thats the wrong way to do this. Ive never seen a health check
system that fails after the first try. All apps have a variable time in
starting up. Sometimes it could take 60 seconds the next 65, especially
complicated java apps that are loading a ton of spring beans.

I think you want to have a retry_delay (number of seconds between tries)
and then a max number of tries. Both should be configurable.

If we choose the proposed solution I know of a number of apps that would
fail every time they were deployed because of the variability in startup
times.

On Wed, Jul 22, 2015 at 8:36 PM, Matthew Fisher notifications@github.com
wrote:

> Yes. The first request after the initial delay should give us the correct
> response (or a failboat)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/deis/deis/issues/4045#issuecomment-123938036.

I second @jchauncey . It's an easy trick which simplify the implementation but degrade the dev experience.

Also I didn't understand why the heathchecker should knock only once for startup. Why not keeping it for the whole life of the node, reporting healthiness for all running services ?

It can be something simple inspired from the gossip protocols: each second, each life-checker of each node chose a random service from the whole cluster, and check its liveliness. It then update the correct key in etcd with last time update and status. (it's the way [Serf](http://serfdom.io) works with membership).

We need to run the numbers to see when statistically we would hit a new service considering different coreos configurations. But a prioritization can also be done: the new-service list is prioritized against the list of already running services. This way we know sooner when they are up without degrading the global health-checking too much.

Last but not least, it would be nice to have more than one healthcheck point. The configuration can be trivial: the `GET` request on the healthcheck endpoint _can_ return one or more `Location:` headers, which would then be checked. That one would be neat.

I as well agree with @jchauncey . 

Secondly, another thing, the `/up` endpoint:

```
GET /up
```

which returns this list:

```
[
    "foo_v2.web.1",
    "foo_v2.web.3",
    "bar_v3.cmd.1"
]
```

Wouldn't it be better to filter this, at least by application, and possibly by version?

E.g.

```
GET /up/foo
GET /up/bar
```

and/or

```
GET /up/foo/v2
GET /up/foo/v3
GET /up/bar/v2
```

I don't see the point of A) sending a list of every living container to the controller, and B) Have the controller iterate through this list.

So the ideas of liveness and readiness have gotten confused again.

_Readiness_ should check for when a container is ready to serve.
_Liveness_ checks whether a container which was previously marked "ready" is still alive.

One healthz endpoint can be used for both. But the logic behind checking for readiness and checking for liveness are different. Readiness is checked only during container deployment, and not until the initial timeout has passed. Liveness is checked repeatedly over the lifetime of the app.

Kubernetes does 3 retries of each type of probe (Liveness or readiness). I don't think they insert delays between probes, but they might. I think we should probably try an incremental backoff strategy that starts with 10% of the HEALTCHECK_INITIAL_DELAY, then 50%, then 100%.

The max retries is not configurable, and I don't actually see any compelling reason to make it so for Deis -- at least not for the Deis user, who could do something horrendous like set it to a million.

(For Kubernetes, see https://github.com/GoogleCloudPlatform/kubernetes/blob/0048eae8ebd4311f66de786f24ae56ec97bc0040/pkg/kubelet/prober/prober.go)

As far as the container developer is concerned, only one healthz endpoint needs to be implemented. Readiness and liveness can both be checked on that same URL. So nothing in this model makes life more complex for the Deis user.

> So the ideas of liveness and readiness have gotten confused again.

The state of healthy/not-healthy is _less_ subtle than ready/alive.
What I suggest, is to separate the logic of testing "is it healthy ?", store this info somewhere, and it's the job to someone else to understand what it means (ready/not-ready or alive/not-alive).

What do you think ?

So to explain this more clearly (thanks @technosophos), the health checker indicates whether the container is ready to service requests. If it fails, publisher will not publish it to etcd. The default state before the initial delay is `Down`. The state for a container when no health check is provided is assumed to be `Up` (assuming the container is actually "up"). This will run forever, continuously checking if the container is up or down, publishing and unpublishing as needed. This is the _Liveness_ check.

Now, here's where the controller comes into play: the controller is the unit that deploys and recycles containers as needed, since it manages an application's lifecycle. Because of this, it needs to perform a _Readiness_ check after an initial delay. For Heroku, this is at 60 seconds, which then if the container is not listening on the expected port then the dyno is killed off. For us, we need to check to ensure the new release is up and responding. If it is down, then we need to kill it and keep the old containers.

> I think we should probably try an incremental backoff strategy that starts with 10% of the HEALTHCHECK_INITIAL_DELAY, then 50%, then 100%.

That sounds like a happy medium.

> Wouldn't it be better to filter this, at least by application, and possibly by version?

Sure. The idea of a global list is to show all containers being published on a single host (since publisher works on a host-by-host basis), but I can add query filtering.

@mackwic that is indeed what's being proposed. Publisher is the component that tests "is it healthy?" and the controller is the one that determines the state of the application's containers.

For a 60 second intial delay I would wait again for 6 seconds, then 30 seconds then 60 seconds? So the last check would happen 1 minute and 36 seconds after the initial check (60 seconds). 

Yes, that would the theoretical maximum your app has to respond before your app is considered kaput.

I guess that means your application could be up but your still waiting. If you app comes up in 97 seconds then you have to wait 59 seconds before its added into the rotation. Still think it should be configurable but know that brings in more trouble for you guys because you need to store that info somewhere. 

Just an FYI… I’m not the @mbutcher you are looking for. _chuckle_ How did I get on this project?

From: Jonathan Chauncey [mailto:notifications@github.com]
Sent: Thursday, July 23, 2015 1:18 PM
To: deis/deis
Cc: Mike Butcher
Subject: Re: [deis] Design Doc: HTTP Healthcheck on Deploy (#4045)

I guess that means your application could be up but your still waiting. If you app comes up in 97 seconds then you have to wait 59 seconds before its added into the rotation. Still think it should be configurable but know that brings in more trouble for you guys because you need to store that info somewhere.

—
Reply to this email directly or view it on GitHubhttps://github.com/deis/deis/issues/4045#issuecomment-124171080.

@mbutcher: @technosophos is mbutcher elsewhere. I'm just happy I haven't made that mistake yet :smiley:.

@jchauncey that is how incremental backoff works. If you're hitting the last of the backoff intervals, you really need to tune your `HEALTHCHECK_INITIAL_DELAY`. But in the meantime, the app isn't getting slammed with healthcheck requests.

We don't _have_ to do it that way, but it is an effort to tolerate the 250% case while not doing tons of healthchecks.

but wont you be slamming hte same endpoint anyways when you do liveness checks?

The assumption with a liveness test is that the container is already in a stable state (while startup is not a stable state).

@bacongobbler Do you think we should set max values on `HEALTHCHECK_INITIAL_DELAY` and `HEALTHCHECK_TIMEOUT`? Timeouts in particular consume network connections, so making sure nobody sets theirs to an extraordinarily high value might be nice.

That might be a good idea. A total sum up to 20 minutes would [match the rest](https://github.com/deis/deis/blob/master/controller/scheduler/fleet.py#L191) of our [timeout values](https://github.com/deis/deis/blob/master/controller/templates/gconf.py#L11), so that would be a good maximum. To caclulate what could be the theoretical maximum before a timeout occurs on the load balancer (ignoring route table latency, request propagation through the API etc.):

1 + .1 + .5 + 1 = 2.6 (total backoff percentage)

20 / 2.6 =  7.6 minutes ~= 7m41s as a theoretical maximum for the inital timeout

7m41s \* 2.6 = 1198.6 seconds, which leaves 1 second left over as the HEALTHCHECK_TIMEOUT value. Anything higher than that and we hit the gunicorn worker timeout.

Any other thoughts on this?

I've read this a few times and I'm happy with this proposal and the great discussion around it. So long as we don't intentionally code anything that makes working more directly with k8s difficult, I'd say let's not let perfect be the enemy of good. :+1: 

I think there's one thing wrong with this proposal and it's the idea of the controller hitting publisher. publisher is an agent running on every host, so if we wanted to perform the readiness check, it would have to connect to every host in the cluster, aggregate the information and then determine if the app is running. I'm thinking that it'd be easier to just connect to etcd and check the keyspace to see if the new version of the app is running instead.


Resolving patch:
diff --git a/controller/api/exceptions.py b/controller/api/exceptions.py
new file mode 100644
--- /dev/null
+++ b/controller/api/exceptions.py
@@ -0,0 +1,3 @@
+class HealthcheckException(Exception):
+    """Exception class used for when the application's health check fails"""
+    pass
diff --git a/controller/api/models.py b/controller/api/models.py
--- a/controller/api/models.py
+++ b/controller/api/models.py
@@ -31,7 +31,7 @@
 import requests
 from rest_framework.authtoken.models import Token
 
-from api import fields, utils
+from api import fields, utils, exceptions
 from registry import publish_release
 from utils import dict_diff, fingerprint
 
@@ -118,6 +118,20 @@ def validate_certificate(value):
         raise ValidationError('Could not load certificate: {}'.format(e))
 
 
+def get_etcd_client():
+    if not hasattr(get_etcd_client, "client"):
+        # wire up etcd publishing if we can connect
+        try:
+            get_etcd_client.client = etcd.Client(
+                host=settings.ETCD_HOST,
+                port=int(settings.ETCD_PORT))
+            get_etcd_client.client.get('/deis')
+        except etcd.EtcdException:
+            logger.log(logging.WARNING, 'Cannot synchronize with etcd cluster')
+            get_etcd_client.client = None
+    return get_etcd_client.client
+
+
 class AuditedModel(models.Model):
     """Add created and updated fields to a model."""
 
@@ -353,6 +367,58 @@ def _start_containers(self, to_add):
         if set([c.state for c in to_add]) != set(['up']):
             err = 'warning, some containers failed to start'
             log_event(self, err, logging.WARNING)
+        # if the user specified a health check, try checking to see if it's running
+        try:
+            config = self.config_set.latest()
+            if 'HEALTHCHECK_URL' in config.values.keys():
+                self._healthcheck(to_add, config.values)
+        except Config.DoesNotExist:
+            pass
+
+    def _healthcheck(self, containers, config):
+        # if at first it fails, back off and try again at 10%, 50% and 100% of INITIAL_DELAY
+        intervals = [1.0, 0.1, 0.5, 1.0]
+        # HACK (bacongobbler): we need to wait until publisher has a chance to publish each
+        # service to etcd, which can take up to 20 seconds.
+        time.sleep(20)
+        for i in range(0, 4):
+            delay = int(config.get('HEALTHCHECK_INITIAL_DELAY', 0))
+            try:
+                # sleep until the initial timeout is over
+                if delay > 0:
+                    time.sleep(delay * intervals[i])
+                self._do_healthcheck(containers, config)
+                break
+            except exceptions.HealthcheckException as e:
+                try:
+                    new_delay = delay * intervals[i+1]
+                    msg = "{}; trying again in {} seconds".format(e, new_delay)
+                    log_event(self, msg, logging.WARNING)
+                except IndexError:
+                    log_event(self, e, logging.WARNING)
+        else:
+            self._destroy_containers(containers)
+            msg = "aborting, app containers failed to respond to health check"
+            log_event(self, msg, logging.ERROR)
+            raise RuntimeError(msg)
+
+    def _do_healthcheck(self, containers, config):
+        path = config.get('HEALTHCHECK_URL', '/')
+        timeout = int(config.get('HEALTHCHECK_TIMEOUT', 1))
+        if not _etcd_client:
+            raise exceptions.HealthcheckException('no etcd client available')
+        for container in containers:
+            try:
+                key = "/deis/services/{self}/{container.job_id}".format(**locals())
+                url = "http://{}{}".format(_etcd_client.get(key).value, path)
+                response = requests.get(url, timeout=timeout)
+                if response.status_code != requests.codes.OK:
+                    raise exceptions.HealthcheckException(
+                        "app failed health check (got '{}', expected: '200')".format(
+                            response.status_code))
+            except (requests.Timeout, requests.ConnectionError, KeyError) as e:
+                raise exceptions.HealthcheckException(
+                    'failed to connect to container ({})'.format(e))
 
     def _restart_containers(self, to_restart):
         """Restarts containers via the scheduler"""
@@ -452,7 +518,7 @@ def _default_scale(self, user, release):
 
         self.scale(user, structure)
 
-    def logs(self, log_lines):
+    def logs(self, log_lines=str(settings.LOG_LINES)):
         """Return aggregated log data for this application."""
         path = os.path.join(settings.DEIS_LOG_DIR, self.id + '.log')
         if not os.path.exists(path):
@@ -513,7 +579,7 @@ def _scheduler(self):
 
     @property
     def state(self):
-        return self._scheduler.state(self._job_id).name
+        return self._scheduler.state(self.job_id).name
 
     def short_name(self):
         return "{}.{}.{}".format(self.app.id, self.type, self.num)
@@ -526,15 +592,10 @@ class Meta:
         get_latest_by = '-created'
         ordering = ['created']
 
-    def _get_job_id(self):
-        app = self.app.id
-        release = self.release
-        version = "v{}".format(release.version)
-        num = self.num
-        job_id = "{app}_{version}.{self.type}.{num}".format(**locals())
-        return job_id
-
-    _job_id = property(_get_job_id)
+    @property
+    def job_id(self):
+        version = "v{}".format(self.release.version)
+        return "{self.app.id}_{version}.{self.type}.{self.num}".format(**locals())
 
     def _get_command(self):
         try:
@@ -566,45 +627,41 @@ def create(self):
         kwargs = {'memory': self.release.config.memory,
                   'cpu': self.release.config.cpu,
                   'tags': self.release.config.tags}
-        job_id = self._job_id
         try:
             self._scheduler.create(
-                name=job_id,
+                name=self.job_id,
                 image=image,
                 command=self._command,
                 **kwargs)
         except Exception as e:
-            err = '{} (create): {}'.format(job_id, e)
+            err = '{} (create): {}'.format(self.job_id, e)
             log_event(self.app, err, logging.ERROR)
             raise
 
     @close_db_connections
     def start(self):
-        job_id = self._job_id
         try:
-            self._scheduler.start(job_id)
+            self._scheduler.start(self.job_id)
         except Exception as e:
-            err = '{} (start): {}'.format(job_id, e)
+            err = '{} (start): {}'.format(self.job_id, e)
             log_event(self.app, err, logging.WARNING)
             raise
 
     @close_db_connections
     def stop(self):
-        job_id = self._job_id
         try:
-            self._scheduler.stop(job_id)
+            self._scheduler.stop(self.job_id)
         except Exception as e:
-            err = '{} (stop): {}'.format(job_id, e)
+            err = '{} (stop): {}'.format(self.job_id, e)
             log_event(self.app, err, logging.ERROR)
             raise
 
     @close_db_connections
     def destroy(self):
-        job_id = self._job_id
         try:
-            self._scheduler.destroy(job_id)
+            self._scheduler.destroy(self.job_id)
         except Exception as e:
-            err = '{} (destroy): {}'.format(job_id, e)
+            err = '{} (destroy): {}'.format(self.job_id, e)
             log_event(self.app, err, logging.ERROR)
             raise
 
@@ -614,7 +671,6 @@ def run(self, command):
             raise EnvironmentError('No build associated with this release '
                                    'to run this command')
         image = self.release.image
-        job_id = self._job_id
         entrypoint = '/bin/bash'
         # if this is a procfile-based app, switch the entrypoint to slugrunner's default
         # FIXME: remove slugrunner's hardcoded entrypoint
@@ -626,10 +682,10 @@ def run(self, command):
         else:
             command = "-c '{}'".format(command)
         try:
-            rc, output = self._scheduler.run(job_id, image, entrypoint, command)
+            rc, output = self._scheduler.run(self.job_id, image, entrypoint, command)
             return rc, output
         except Exception as e:
-            err = '{} (run): {}'.format(job_id, e)
+            err = '{} (run): {}'.format(self.job_id, e)
             log_event(self.app, err, logging.ERROR)
             raise
 
@@ -1103,6 +1159,34 @@ def _etcd_purge_cert(**kwargs):
         pass
 
 
+def _etcd_publish_config(**kwargs):
+    config = kwargs['instance']
+    # we purge all existing config when adding the newest instance. This is because
+    # deis config:unset would remove an existing value, but not delete the
+    # old config object
+    try:
+        _etcd_client.delete('/deis/config/{}'.format(config.app),
+                            prevExist=True, dir=True, recursive=True)
+    except KeyError:
+        pass
+    if kwargs['created']:
+        for k, v in config.values.iteritems():
+            _etcd_client.write(
+                '/deis/config/{}/{}'.format(
+                    config.app,
+                    unicode(k).encode('utf-8').lower()),
+                unicode(v).encode('utf-8'))
+
+
+def _etcd_purge_config(**kwargs):
+    config = kwargs['instance']
+    try:
+        _etcd_client.delete('/deis/config/{}'.format(config.app),
+                            prevExist=True, dir=True, recursive=True)
+    except KeyError:
+        pass
+
+
 def _etcd_publish_domains(**kwargs):
     domain = kwargs['instance']
     if kwargs['created']:
@@ -1134,13 +1218,9 @@ def create_auth_token(sender, instance=None, created=False, **kwargs):
     if created:
         Token.objects.create(user=instance)
 
-# wire up etcd publishing if we can connect
-try:
-    _etcd_client = etcd.Client(host=settings.ETCD_HOST, port=int(settings.ETCD_PORT))
-    _etcd_client.get('/deis')
-except etcd.EtcdException:
-    logger.log(logging.WARNING, 'Cannot synchronize with etcd cluster')
-    _etcd_client = None
+
+_etcd_client = get_etcd_client()
+
 
 if _etcd_client:
     post_save.connect(_etcd_publish_key, sender=Key, dispatch_uid='api.models')
@@ -1152,3 +1232,5 @@ def create_auth_token(sender, instance=None, created=False, **kwargs):
     post_delete.connect(_etcd_purge_app, sender=App, dispatch_uid='api.models')
     post_save.connect(_etcd_publish_cert, sender=Certificate, dispatch_uid='api.models')
     post_delete.connect(_etcd_purge_cert, sender=Certificate, dispatch_uid='api.models')
+    post_save.connect(_etcd_publish_config, sender=Config, dispatch_uid='api.models')
+    post_delete.connect(_etcd_purge_config, sender=Config, dispatch_uid='api.models')


Test patch:
diff --git a/controller/api/tests/test_config.py b/controller/api/tests/test_config.py
--- a/controller/api/tests/test_config.py
+++ b/controller/api/tests/test_config.py
@@ -13,18 +13,44 @@
 
 from django.contrib.auth.models import User
 from django.test import TransactionTestCase
+import etcd
 from rest_framework.authtoken.models import Token
 
-from api.models import Config
+import api.exceptions
+from api.models import App, Config
 
 
-def mock_import_repository_task(*args, **kwargs):
+def mock_status_ok(*args, **kwargs):
     resp = requests.Response()
     resp.status_code = 200
     resp._content_consumed = True
     return resp
 
 
+def mock_status_not_found(*args, **kwargs):
+    resp = requests.Response()
+    resp.status_code = 404
+    resp._content_consumed = True
+    return resp
+
+
+def mock_request_connection_error(*args, **kwargs):
+    raise requests.exceptions.ConnectionError("connection error")
+
+
+class MockEtcdClient:
+
+    def __init__(self, app):
+        self.app = app
+
+    def get(self, key, *args, **kwargs):
+        node = {
+            'key': '/deis/services/{}/{}_v2.web.1'.format(self.app, self.app),
+            'value': '127.0.0.1:1234'
+        }
+        return etcd.EtcdResult(None, node)
+
+
 class ConfigTest(TransactionTestCase):
 
     """Tests setting and updating config values"""
@@ -34,8 +60,12 @@ class ConfigTest(TransactionTestCase):
     def setUp(self):
         self.user = User.objects.get(username='autotest')
         self.token = Token.objects.get(user=self.user).key
+        url = '/v1/apps'
+        response = self.client.post(url, HTTP_AUTHORIZATION='token {}'.format(self.token))
+        self.assertEqual(response.status_code, 201)
+        self.app = App.objects.all()[0]
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_config(self):
         """
         Test that config is auto-created for a new app and that
@@ -107,7 +137,7 @@ def test_config(self):
         self.assertEqual(response.status_code, 405)
         return config5
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_response_data(self):
         """Test that the serialized response contains only relevant data."""
         body = {'id': 'test'}
@@ -132,7 +162,7 @@ def test_response_data(self):
         }
         self.assertDictContainsSubset(expected, response.data)
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_response_data_types_converted(self):
         """Test that config data is converted into the correct type."""
         body = {'id': 'test'}
@@ -164,7 +194,7 @@ def test_response_data_types_converted(self):
         self.assertEqual(response.status_code, 400)
         self.assertIn('CPU shares must be an integer', response.data['cpu'])
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_config_set_same_key(self):
         """
         Test that config sets on the same key function properly
@@ -188,7 +218,7 @@ def test_config_set_same_key(self):
         self.assertIn('PORT', response.data['values'])
         self.assertEqual(response.data['values']['PORT'], '5001')
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_config_set_unicode(self):
         """
         Test that config sets with unicode values are accepted.
@@ -219,14 +249,14 @@ def test_config_set_unicode(self):
         self.assertIn('INTEGER', response.data['values'])
         self.assertEqual(response.data['values']['INTEGER'], '1')
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_config_str(self):
         """Test the text representation of a node."""
         config5 = self.test_config()
         config = Config.objects.get(uuid=config5['uuid'])
         self.assertEqual(str(config), "{}-{}".format(config5['app'], config5['uuid'][:7]))
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_admin_can_create_config_on_other_apps(self):
         """If a non-admin creates an app, an administrator should be able to set config
         values for that app.
@@ -246,7 +276,7 @@ def test_admin_can_create_config_on_other_apps(self):
         self.assertIn('PORT', response.data['values'])
         return response
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_limit_memory(self):
         """
         Test that limit is auto-created for a new app and that
@@ -334,7 +364,7 @@ def test_limit_memory(self):
         self.assertEqual(response.status_code, 405)
         return limit4
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_limit_cpu(self):
         """
         Test that CPU limits can be set
@@ -405,7 +435,7 @@ def test_limit_cpu(self):
         self.assertEqual(response.status_code, 405)
         return limit4
 
-    @mock.patch('requests.post', mock_import_repository_task)
+    @mock.patch('requests.post', mock_status_ok)
     def test_tags(self):
         """
         Test that tags can be set on an application
@@ -509,3 +539,135 @@ def test_unauthorized_user_cannot_modify_config(self):
         response = self.client.post(url, json.dumps(body), content_type='application/json',
                                     HTTP_AUTHORIZATION='token {}'.format(unauthorized_token))
         self.assertEqual(response.status_code, 403)
+
+    def _test_app_healthcheck(self,):
+        # post a new build, expecting it to pass as usual
+        url = "/v1/apps/{self.app}/builds".format(**locals())
+        body = {'image': 'autotest/example'}
+        response = self.client.post(url, json.dumps(body), content_type='application/json',
+                                    HTTP_AUTHORIZATION='token {}'.format(self.token))
+        self.assertEqual(response.status_code, 201)
+        # mock out the etcd client
+        api.models._etcd_client = MockEtcdClient(self.app)
+        # set an initial healthcheck url.
+        url = "/v1/apps/{self.app}/config".format(**locals())
+        body = {'values': json.dumps({'HEALTHCHECK_URL': '/'})}
+        return self.client.post(url, json.dumps(body), content_type='application/json',
+                                HTTP_AUTHORIZATION='token {}'.format(self.token))
+
+    @mock.patch('requests.get', mock_status_ok)
+    @mock.patch('time.sleep', lambda func: func)
+    def test_app_healthcheck_good(self):
+        """
+        If a user deploys an app with a config value set for HEALTHCHECK_URL, the controller
+        should check that the application responds with a 200 OK.
+        """
+        response = self._test_app_healthcheck()
+        self.assertEqual(response.status_code, 201)
+        self.assertEqual(self.app.release_set.latest().version, 3)
+
+    @mock.patch('requests.get', mock_status_not_found)
+    @mock.patch('api.models.get_etcd_client', lambda func: func)
+    @mock.patch('time.sleep', lambda func: func)
+    def test_app_healthcheck_bad(self):
+        """
+        If a user deploys an app with a config value set for HEALTHCHECK_URL, the controller
+        should check that the application responds with a 200 OK. If it's down, the app should be
+        rolled back.
+        """
+        response = self._test_app_healthcheck()
+        self.assertEqual(response.status_code, 503)
+        self.assertEqual(
+            response.data,
+            {'detail': 'aborting, app containers failed to respond to health check'})
+        # check that only the build and initial release exist
+        self.assertEqual(self.app.release_set.latest().version, 2)
+        # assert that the reason why the containers failed was because
+        # they failed the health check 4 times
+        self.assertEqual(
+            self.app.logs().count("app failed health check (got '404', expected: '200')"),
+            4)
+
+    @mock.patch('requests.get', mock_status_not_found)
+    @mock.patch('api.models.get_etcd_client', lambda func: func)
+    @mock.patch('time.sleep')
+    def test_app_backoff_interval(self, mock_time):
+        """
+        Ensure that when a healthcheck fails, a backoff strategy is used before trying again.
+        """
+        # post a new build, expecting it to pass as usual
+        url = "/v1/apps/{self.app}/builds".format(**locals())
+        body = {'image': 'autotest/example'}
+        response = self.client.post(url, json.dumps(body), content_type='application/json',
+                                    HTTP_AUTHORIZATION='token {}'.format(self.token))
+        self.assertEqual(response.status_code, 201)
+        # mock out the etcd client
+        api.models._etcd_client = MockEtcdClient(self.app)
+        # set an initial healthcheck url.
+        url = "/v1/apps/{self.app}/config".format(**locals())
+        body = {'values': json.dumps({'HEALTHCHECK_URL': '/'})}
+        return self.client.post(url, json.dumps(body), content_type='application/json',
+                                HTTP_AUTHORIZATION='token {}'.format(self.token))
+        self.assertEqual(mock_time.call_count, 5)
+
+    @mock.patch('requests.get', mock_status_ok)
+    @mock.patch('time.sleep')
+    def test_app_healthcheck_initial_delay(self, mock_time):
+        """
+        Ensure that when an initial delay is set, the request will sleep for x seconds, where
+        x is the number of seconds in the initial timeout.
+        """
+        # post a new build, expecting it to pass as usual
+        url = "/v1/apps/{self.app}/builds".format(**locals())
+        body = {'image': 'autotest/example'}
+        response = self.client.post(url, json.dumps(body), content_type='application/json',
+                                    HTTP_AUTHORIZATION='token {}'.format(self.token))
+        self.assertEqual(response.status_code, 201)
+        # mock out the etcd client
+        api.models._etcd_client = MockEtcdClient(self.app)
+        # set an initial healthcheck url.
+        url = "/v1/apps/{self.app}/config".format(**locals())
+        body = {'values': json.dumps({'HEALTHCHECK_URL': '/'})}
+        return self.client.post(url, json.dumps(body), content_type='application/json',
+                                HTTP_AUTHORIZATION='token {}'.format(self.token))
+        # mock_time increments by one each time its called, so we should expect 2 calls to
+        # mock_time; one for the call in the code, and one for this invocation.
+        mock_time.assert_called_with(0)
+        app = App.objects.all()[0]
+        url = "/v1/apps/{app}/config".format(**locals())
+        body = {'values': json.dumps({'HEALTHCHECK_INITIAL_DELAY': 10})}
+        self.client.post(url, json.dumps(body), content_type='application/json',
+                         HTTP_AUTHORIZATION='token {}'.format(self.token))
+        mock_time.assert_called_with(10)
+
+    @mock.patch('requests.get')
+    @mock.patch('time.sleep', lambda func: func)
+    def test_app_healthcheck_timeout(self, mock_request):
+        """
+        Ensure when a timeout value is set, the controller respects that value
+        when making a request.
+        """
+        self._test_app_healthcheck()
+        app = App.objects.all()[0]
+        url = "/v1/apps/{app}/config".format(**locals())
+        body = {'values': json.dumps({'HEALTHCHECK_TIMEOUT': 10})}
+        self.client.post(url, json.dumps(body), content_type='application/json',
+                         HTTP_AUTHORIZATION='token {}'.format(self.token))
+        mock_request.assert_called_with('http://127.0.0.1:1234/', timeout=10)
+
+    @mock.patch('requests.get', mock_request_connection_error)
+    @mock.patch('time.sleep', lambda func: func)
+    def test_app_healthcheck_connection_error(self):
+        """
+        If a user deploys an app with a config value set for HEALTHCHECK_URL but the app
+        returns a connection error, the controller should continue checking until either the app
+        responds or the app fails to respond within the timeout.
+
+        NOTE (bacongobbler): the Docker userland proxy listens for connections and returns a
+        ConnectionError, hence the unit test.
+        """
+        response = self._test_app_healthcheck()
+        self.assertEqual(response.status_code, 503)
+        self.assertEqual(
+            response.data,
+            {'detail': 'aborting, app containers failed to respond to health check'})
diff --git a/publisher/server/publisher_test.go b/publisher/server/publisher_test.go
--- a/publisher/server/publisher_test.go
+++ b/publisher/server/publisher_test.go
@@ -1,7 +1,10 @@
 package server
 
 import (
+	"fmt"
 	"net"
+	"net/http"
+	"net/http/httptest"
 	"testing"
 )
 
@@ -45,3 +48,25 @@ func TestIsPortOpen(t *testing.T) {
 		t.Errorf("Port should be closed")
 	}
 }
+
+func TestHealthCheckOK(t *testing.T) {
+	s := &Server{}
+
+	// good server
+	ts1 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		fmt.Fprintln(w, "Hello, client")
+	}))
+	defer ts1.Close()
+	if !s.HealthCheckOK(ts1.URL, 0, 0) {
+		t.Errorf("healthcheck should be OK")
+	}
+
+	// bad server
+	ts2 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		http.NotFound(w, r)
+	}))
+	defer ts2.Close()
+	if s.HealthCheckOK(ts2.URL, 0, 0) {
+		t.Errorf("healthcheck should be NOT OK")
+	}
+}
diff --git a/tests/Makefile b/tests/Makefile
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -23,6 +23,7 @@ test-full: test-style
 	godep go test -tags integration -v -run TestBuilds
 	godep go test -tags integration -v -run TestConfig
 	godep go test -tags integration -v -run TestDomains
+	godep go test -tags integration -v -run TestHealthcheck
 	godep go test -tags integration -v -run TestKeys
 	godep go test -tags integration -v -run TestPerms
 	godep go test -tags integration -v -run TestPs
diff --git a/tests/bin/test-setup.sh b/tests/bin/test-setup.sh
--- a/tests/bin/test-setup.sh
+++ b/tests/bin/test-setup.sh
@@ -115,8 +115,6 @@ function dump_logs {
     #echo "$CURRENT_APP"
     get_journal_logs $CURRENT_APP
   done
-  # etcd keyspace
-  get_logs deis-controller "etcdctl ls / --recursive" etcdctl-dump
   # component logs
   get_logs deis-builder
   get_logs deis-controller
@@ -141,7 +139,15 @@ function dump_logs {
   get_logs deis-router@3 deis-store-volume deis-store-volume-3
   get_logs deis-store-gateway
 
-  # get debug-etcd logs
+  # docker logs
+  fleetctl -strict-host-key-checking=false ssh deis-router@1 journalctl --no-pager -u docker \
+    > $FAILED_LOGS_DIR/docker-1.log
+  fleetctl -strict-host-key-checking=false ssh deis-router@2 journalctl --no-pager -u docker \
+    > $FAILED_LOGS_DIR/docker-2.log
+  fleetctl -strict-host-key-checking=false ssh deis-router@3 journalctl --no-pager -u docker \
+    > $FAILED_LOGS_DIR/docker-3.log
+
+  # etcd logs
   fleetctl -strict-host-key-checking=false ssh deis-router@1 journalctl --no-pager -u etcd \
     > $FAILED_LOGS_DIR/debug-etcd-1.log
   fleetctl -strict-host-key-checking=false ssh deis-router@2 journalctl --no-pager -u etcd \
@@ -149,6 +155,10 @@ function dump_logs {
   fleetctl -strict-host-key-checking=false ssh deis-router@3 journalctl --no-pager -u etcd \
     > $FAILED_LOGS_DIR/debug-etcd-3.log
 
+  # etcdctl dump
+  fleetctl -strict-host-key-checking=false ssh deis-router@1 etcdctl ls / --recursive \
+    > $FAILED_LOGS_DIR/etcdctl-dump.log
+
   # tarball logs
   BUCKET=jenkins-failure-logs
   FILENAME=deis-test-failure-$TIMESTAMP.tar.gz
diff --git a/tests/healthcheck_test.go b/tests/healthcheck_test.go
new file mode 100644
--- /dev/null
+++ b/tests/healthcheck_test.go
@@ -0,0 +1,74 @@
+// +build integration
+
+package tests
+
+import (
+	"fmt"
+	"net/http"
+	"testing"
+
+	"github.com/deis/deis/tests/utils"
+)
+
+var (
+	healthcheckGoodCmd = "config:set HEALTHCHECK_URL=/ --app={{.AppName}}"
+)
+
+func TestHealthcheck(t *testing.T) {
+	client := utils.HTTPClient()
+	cfg := healthcheckSetup(t)
+	done := make(chan bool, 1)
+	url := fmt.Sprintf("http://%s.%s", cfg.AppName, cfg.Domain)
+
+	utils.Execute(t, healthcheckGoodCmd, cfg, false, "/")
+	go func() {
+		// there should never be any downtime during these health check operations
+		psScaleTest(t, cfg, psScaleCmd)
+		cfg.ProcessNum = "1"
+		psScaleTest(t, cfg, psScaleCmd)
+		// kill healthcheck goroutine
+		done <- true
+	}()
+
+	// run health checks in parallel while performing operations
+	fmt.Printf("starting health checks at %s\n", url)
+loop:
+	for {
+		select {
+		case <-done:
+			fmt.Println("done performing health checks")
+			break loop
+		default:
+			doHealthCheck(t, client, url)
+		}
+	}
+	utils.AppsDestroyTest(t, cfg)
+}
+
+func healthcheckSetup(t *testing.T) *utils.DeisTestConfig {
+	cfg := utils.GetGlobalConfig()
+	cfg.AppName = "healthchecksample"
+	utils.Execute(t, authLoginCmd, cfg, false, "")
+	utils.Execute(t, gitCloneCmd, cfg, false, "")
+	if err := utils.Chdir(cfg.ExampleApp); err != nil {
+		t.Fatal(err)
+	}
+	utils.Execute(t, appsCreateCmd, cfg, false, "")
+	utils.Execute(t, gitPushCmd, cfg, false, "")
+	utils.CurlApp(t, *cfg)
+	if err := utils.Chdir(".."); err != nil {
+		t.Fatal(err)
+	}
+	return cfg
+}
+
+func doHealthCheck(t *testing.T, client *http.Client, url string) {
+	response, err := client.Get(url)
+	if err != nil {
+		t.Fatalf("could not retrieve response from %s: %v\n", url, err)
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusOK {
+		t.Fatalf("app had some downtime while undergoing health checks (got %d response)", response.StatusCode)
+	}
+}
diff --git a/tests/utils/itutils.go b/tests/utils/itutils.go
--- a/tests/utils/itutils.go
+++ b/tests/utils/itutils.go
@@ -123,12 +123,17 @@ func GetGlobalConfig() *DeisTestConfig {
 	return &envCfg
 }
 
-func doCurl(url string) ([]byte, error) {
+// HTTPClient returns a client for use with the integration tests.
+func HTTPClient() *http.Client {
 	// disable security check for self-signed certificates
 	tr := &http.Transport{
 		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
 	}
-	client := &http.Client{Transport: tr}
+	return &http.Client{Transport: tr}
+}
+
+func doCurl(url string) ([]byte, error) {
+	client := HTTPClient()
 	response, err := client.Get(url)
 	if err != nil {
 		return nil, err
@@ -300,6 +305,7 @@ func Execute(t *testing.T, cmd string, params interface{}, failFlag bool, expect
 
 // AppsDestroyTest destroys a Deis app and checks that it was successful.
 func AppsDestroyTest(t *testing.T, params *DeisTestConfig) {
+	fmt.Printf("destroying app %s...\n", params.ExampleApp)
 	cmd := "apps:destroy --app={{.AppName}} --confirm={{.AppName}}"
 	if err := Chdir(params.ExampleApp); err != nil {
 		t.Fatal(err)