mesosphere/marathon — pull request #386 by ConnorDoyle (2014-07-11 22:04:29+00:00)
Title: Exponential Task Launch Back-off (per-app)

Adds a configurable (per-app) exponentially increasing task launch cool down interval.

See issues #20 and #258
Supersedes #345

Two new optional parameters are now exposed in an app. The values shown below are the defaults.

```json
{
"launchDelaySeconds": 1,
"launchDelayFactor": 1.15
}
```

Each time a task fails, is lost, or is killed due to failing too many health checks, the launch delay for that app's tasks increases by the supplied factor.

See the following log snippet to get a feel for how this plays out:

```
[INFO] [07/11/2014 14:46:29.095] [marathon-akka.actor.default-dispatcher-24] [akka://marathon/user/$a] Received health result: [Healthy(toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:29.094Z)]
[INFO] [07/11/2014 14:46:31.115] [marathon-akka.actor.default-dispatcher-22] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:31.115Z)]
[INFO] [07/11/2014 14:46:31.116] [marathon-akka.actor.default-dispatcher-28] [akka://marathon/user/$a] Received health result: [Unhealthy(toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:31.116Z,410 Gone)]
[INFO] [07/11/2014 14:46:33.136] [marathon-akka.actor.default-dispatcher-4] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:33.135Z)]
[INFO] [07/11/2014 14:46:33.136] [marathon-akka.actor.default-dispatcher-4] [akka://marathon/user/$a] Received health result: [Unhealthy(toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:33.135Z,410 Gone)]
[INFO] [07/11/2014 14:46:35.154] [marathon-akka.actor.default-dispatcher-22] [akka://marathon/user/$a] Received health result: [Unhealthy(toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:35.154Z,410 Gone)]
[INFO] [07/11/2014 14:46:35.154] [marathon-akka.actor.default-dispatcher-22] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:35.154Z)]
[INFO] [07/11/2014 14:46:37.175] [marathon-akka.actor.default-dispatcher-26] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:37.175Z)]
[INFO] [07/11/2014 14:46:37.175] [marathon-akka.actor.default-dispatcher-26] [akka://marathon/user/$a] Received health result: [Unhealthy(toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:37.175Z,410 Gone)]
[INFO] [07/11/2014 14:46:37.175] [marathon-akka.actor.default-dispatcher-26] [akka://marathon/user/$a] Killing task toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac on host localhost
[2014-07-11 14:46:37,178] WARN Task [toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac] for app [toggle] was killed for failing too many health checks (mesosphere.marathon.MarathonScheduler:198)
[2014-07-11 14:46:37,180] INFO Task launch delay for [toggle] is now [1] seconds (mesosphere.util.RateLimiter:39)
[2014-07-11 14:46:37,334] INFO Received status update for task toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac: TASK_KILLED (Command terminated with signal Terminated: 15) (mesosphere.marathon.MarathonScheduler:144)
[2014-07-11 14:46:37,343] INFO Task toggle.5f73d3d1-0944-11e4-a3a5-685b35a05cac removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:81)
[2014-07-11 14:46:37,345] INFO Need to scale toggle from 1 up to 2 instances (mesosphere.marathon.MarathonScheduler:352)
[2014-07-11 14:46:37,346] INFO Queueing 1 new tasks for toggle (0 queued) (mesosphere.marathon.MarathonScheduler:358)
[INFO] [07/11/2014 14:46:39.195] [marathon-akka.actor.default-dispatcher-31] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:39.194Z)]
[INFO] [07/11/2014 14:46:41.214] [marathon-akka.actor.default-dispatcher-22] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:41.214Z)]
[INFO] [07/11/2014 14:46:43.235] [marathon-akka.actor.default-dispatcher-10] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:43.234Z)]
[2014-07-11 14:46:44,341] INFO Received status update for task toggle.d92898bf-0944-11e4-9801-685b35a05cac: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler:144)
[2014-07-11 14:46:44,341] INFO Task launch delay for [toggle] reset to zero (mesosphere.util.RateLimiter:46)
[INFO] [07/11/2014 14:46:45.254] [marathon-akka.actor.default-dispatcher-22] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:45.254Z)]
[INFO] [07/11/2014 14:46:45.256] [marathon-akka.actor.default-dispatcher-27] [akka://marathon/user/$a] Received health result: [Healthy(toggle.d92898bf-0944-11e4-9801-685b35a05cac,2014-07-11T21:46:45.255Z)]
[2014-07-11 14:46:46,662] INFO Syncing tasks for all apps (mesosphere.marathon.MarathonScheduler:296)
[2014-07-11 14:46:46,663] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.MarathonScheduler:313)
[2014-07-11 14:46:46,663] INFO Already running 2 instances of toggle. Not scaling. (mesosphere.marathon.MarathonScheduler:377)
[2014-07-11 14:46:46,669] INFO Received status update for task toggle.d92898bf-0944-11e4-9801-685b35a05cac: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler:144)
[2014-07-11 14:46:46,680] INFO Received status update for task toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler:144)
[INFO] [07/11/2014 14:46:47.275] [marathon-akka.actor.default-dispatcher-10] [akka://marathon/user/$a] Received health result: [Healthy(toggle.d92898bf-0944-11e4-9801-685b35a05cac,2014-07-11T21:46:47.275Z)]
[INFO] [07/11/2014 14:46:47.275] [marathon-akka.actor.default-dispatcher-10] [akka://marathon/user/$a] Received health result: [Healthy(toggle.76b7ca63-0944-11e4-a3a5-685b35a05cac,2014-07-11T21:46:47.275Z)]
```