Repository: deis/deis

Issue:
Failure to create a container leaves others created
If `deis scale cmd=5` creates 4 containers but fails on the 5th, it raises an exception and leaves 4 containers in the "created" state, not running. The code in question is here: https://github.com/deis/deis/blob/master/controller/api/models.py#L272-L292

It would be more intuitive and useful behavior if instead Deis made a "best effort" to start whichever containers it was able to create. In this particular case we used `deis limits:set -c cmd=1024` to ensure some containers wouldn't be created due to resource limitations, then scaled up, and were left with this:

``` console
$ deis info
=== zigzag-drumroll Application
{
  "updated": "2015-04-30T22:05:12UTC", 
  "uuid": "0c0dd591-5d9c-45c4-914d-52d703886cb3", 
  "created": "2015-04-30T21:53:21UTC", 
  "url": "zigzag-drumroll.local5.deisapp.com", 
  "owner": "test", 
  "id": "zigzag-drumroll", 
  "structure": {
    "cmd": 1
  }
}

=== zigzag-drumroll Processes

--- cmd: 
cmd.1 created (v3)
cmd.2 created (v3)
cmd.3 created (v3)
cmd.4 created (v3)
cmd.5 destroyed (v3)
```



Discussion:
Interesting. In this particular scenario, they should've been removed from fleet because we roll back changes if they fail to create. If they were starting, then it would've done a "best effort" and just warned if they failed to start.

Does `deis logs` confirm https://github.com/deis/deis/blob/master/controller/api/models.py#L284-L287 was triggered? If so, why weren't they removed? Did we accidentally remove that rollback feature?

It happened with Swarm not yet hit this particular use case with Fleet. @mboersma  set the cpu limits to 1024 and scaled the app to 5 on a 4 node cluster . 
Expected behavior is 4 Containers of the app should be started on 4 nodes and 5th container should hit failure case.

Maybe this is effectively specific to the Swam Scheduler then, but it looks like we could fall into the same code regardless. I'll try to fix it since I have a reproducible case here using Swarm.

There is a high probability of hitting this scenario if the scheduler is resource aware. 

It's not hard to fix the [models.py code](https://github.com/deis/deis/blob/master/controller/api/models.py#L272-L292) to start the containers which were created successfully, which fixes this case. The tricky part is bubbling up the particular errors so the `deis` CLI user can see that some containers started, but also know why others were not.

Thinking of the two use cases, if someone calls `deis scale web=20` with a resource-constrained scheduler like Swarm, we could
1. abort the operation saying that there's not enough resources for this request, rolling back
2. scale up as high as possible, then report that there's not enough resources for the rest of the containers (@mboersma's proposal)

@carmstrong any thoughts on which is the best solution? Both are correct, it's just a matter of what behaviour we should enforce.

EDIT: we should also be aware of `heroku scale web+1`, which scales up the dyno count by one. If no containers get created then we should return a proper error message if we're shooting for (2).

@bacongobbler I would think in your examples, number 2 is the most correct. We could always roll back, and the user could try a different scale number.

I strongly suggest we abort the operation if there are not enough resources (option 1).  If I ask for 20 containers and I only get 10, that should be rolled back immediately since a) it's not what I asked for and b) the cluster might be resource-starved and we should release the resources.

@gabrtv best effort scaling is one of the solutions but with deis we want users to be cluster agnostic. So if `deis scale web=x` failed how would a user know what is the best 'x' to scale next if the previous 'x' value failed.  I think we should give best effort as an option while scaling the containers, `deis scale web=x`. Here user is stressing that I want either 'x' containers or nothing and `deis scale web=~x` or `deis scale web <= x` A soft constraint where user wants to scale upto some value less than or equal to 'x' .  I also agree with @bacongobbler `deis scale web++` or `deis scale web+1`.

Its also hard for the user to guess the best value of 'x' if the resources in the cluster are changin dynamically

> failed how would a user know what is the best 'x' to scale next if the previous 'x' value failed

Maybe an error response with `Cannot scale web=20, requested mem=2G, available=1G`? Basically a "I cannot accept your request. I got x resources, but I only have y resources available for your use." I agree with @gabrtv that we should completely fall back when the cluster cannot handle the amount of requested resources.

> Here user is stressing that I want either 'x' containers or nothing and deis scale web=~x or deis scale web <= x A soft constraint where user wants to scale upto some value less than or equal to 'x'.

That sounds like a better way to handle "soft" scaling. I still agree that `deis scale web=20` should abort, but maybe `deis scale web=~20` would be a soft scale operation.

yeah but we should not make user calculate the X value by himself and by the time he calculated the X value the available resources in the cluster might change. We should either give him an option to schedule to X or <= X. Basically we cannot guarantee to what value user can scale unless we have our own custom scheduler that locks the available resources until he makes the next decision.    

> yeah but we should not make user calculate the X value by himself and by the time he calculated the X value the available resources in the cluster might change.

That is true, but it's also a rare case that only occurs when all users are sharing all of the resources of a cluster. In a multi-tenant Deis environment (which is something I've briefly mentioned in https://github.com/deis/deis/issues/2007#issuecomment-57721203), each user (or a group of users) would be given a slice of the cluster's resources which only they would have access to, so the resource pool wouldn't necessarily change all that much. I still think that having a strict _and_ a lazy way to scale would still be the best way to solve this.

> We should either give him an option to schedule to X or <= X

Yes, I think we're both agreeing on the same thing here. We can have both worlds with `deis scale web=20` being a **hard** scale (today's behaviour, which is what most people would expect anyways) and `deis scale web=~20` being a **soft** scale. Does that sound good to you, or is there something else missing?

Yup exactly. We should also give a feature like soft scale or come up with our own arithmetic operators (<= , ++ , -- , web(+/-)y )

If I have an app with 40 containers running, and I do `deis scale cmd=41` and it can't create the 41st container...what should happen? Currently Deis leaves the existing 40 alone and tries to create only the one needed, then fails and leaves this:

``` console
$ deis info
...
=== rugged-teaspoon Processes

--- cmd: 
cmd.1 up (v3)
cmd.2 up (v3)
...
cmd.39 up (v3)
cmd.40 up (v3)
cmd.41 destroyed (v3)
```

But the behavior described above indicates we should destroy the 40 running containers as well.

Which is fine if that's how it should behave. I'm just pointing out this is a tricky area and we must proceed cautiously.

> But the behavior described above indicates we should destroy the 40 running containers as well.

Sorry, I meant a roll back to the previous state, so only the 41st container would be destroyed.


Resolving patch:
diff --git a/controller/api/models.py b/controller/api/models.py
--- a/controller/api/models.py
+++ b/controller/api/models.py
@@ -292,6 +292,7 @@ def _start_containers(self, to_add):
         if any(c.state != 'created' for c in to_add):
             err = 'aborting, failed to create some containers'
             log_event(self, err, logging.ERROR)
+            self._destroy_containers(to_add)
             raise RuntimeError(err)
         [t.start() for t in start_threads]
         [t.join() for t in start_threads]
diff --git a/controller/scheduler/swarm.py b/controller/scheduler/swarm.py
--- a/controller/scheduler/swarm.py
+++ b/controller/scheduler/swarm.py
@@ -33,7 +33,8 @@ def create(self, name, image, command='', template=None, **kwargs):
                 mem = mem[:-1]
         cpu = kwargs.get('cpu', {}).get(l['c_type'])
         self.docker_cli.create_container(image=cimage, name=name,
-                                         command=command.encode('utf-8'), mem_limit=mem,
+                                         command=command.encode('utf-8'),
+                                         mem_limit=mem,
                                          cpu_shares=cpu,
                                          environment=[affinity],
                                          host_config={'PublishAllPorts': True})


Test patch:
diff --git a/controller/api/tests/test_scheduler.py b/controller/api/tests/test_scheduler.py
--- a/controller/api/tests/test_scheduler.py
+++ b/controller/api/tests/test_scheduler.py
@@ -75,10 +75,7 @@ def test_create_chaos(self):
         url = "/v1/apps/{app_id}/containers".format(**locals())
         response = self.client.get(url, HTTP_AUTHORIZATION='token {}'.format(self.token))
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(len(response.data['results']), 20)
-        # make sure some failed
-        states = set([c['state'] for c in response.data['results']])
-        self.assertEqual(states, set(['error', 'created']))
+        self.assertEqual(len(response.data['results']), 0)
 
     def test_start_chaos(self):
         url = '/v1/apps'
@@ -304,7 +301,6 @@ def test_build_chaos(self):
         response = self.client.get(url, HTTP_AUTHORIZATION='token {}'.format(self.token))
         self.assertEqual(response.status_code, 200)
         self.assertEqual(len(response.data['results']), 20)
-
         # make sure all old containers are still up
         states = set([c['state'] for c in response.data['results']])
         self.assertEqual(states, set(['up']))