Repository: deis/deis

Issue:
AuthenticationException when scaling down a layer
Create a formation, then run:

```
deis layers:scale proxy=1 runtime=1
```

Wait until that finishes, then rescale to fewer nodes:

```
deis layers:scale proxy=1 runtime=0
```

Deis will still try to SSH to the terminating node to run a chef converge:

``` python
==> /var/log/deis/server.log <==
Internal Server Error: /api/formations/luxury-sailfish/scale/layers
Traceback (most recent call last):
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/rest_framework/viewsets.py", line 78, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/rest_framework/views.py", line 327, in dispatch
    response = self.handle_exception(exc)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/rest_framework/views.py", line 324, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/deis/controller/api/views.py", line 184, in scale_layers
    databag = formation.scale_layers()
  File "/opt/deis/controller/api/models.py", line 290, in scale_layers
    self.converge(databag)
  File "/opt/deis/controller/api/models.py", line 437, in converge
    job.apply_async().join()
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/result.py", line 495, in join
    interval=interval))
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/result.py", line 108, in get
    interval=interval)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/backends/amqp.py", line 151, in wait_for
    raise self.exception_to_python(meta['result'])
AuthenticationException: Authentication failed.
```



Discussion:

Resolving patch:
diff --git a/api/models.py b/api/models.py
--- a/api/models.py
+++ b/api/models.py
@@ -267,6 +267,7 @@ def scale_layers(self, **kwargs):
             diff = requested - len(nodes)
             if diff == 0:
                 continue
+            new_nodes = False
             while diff < 0:
                 node = nodes.pop(0)
                 funcs.append(node.terminate)
@@ -276,6 +277,7 @@ def scale_layers(self, **kwargs):
                 nodes.append(node)
                 funcs.append(node.launch)
                 diff = requested - len(nodes)
+                new_nodes = True
         # http://docs.celeryproject.org/en/latest/userguide/canvas.html#groups
         job = [func() for func in funcs]
         # balance containers
@@ -285,8 +287,8 @@ def scale_layers(self, **kwargs):
             group(*job).apply_async().join()
         # once nodes are in place, recalculate the formation and update the data bag
         databag = self.calculate()
-        # force-converge nodes if there were changes
-        if job or containers_balanced:
+        # force-converge nodes if there were new nodes or container rebalancing
+        if new_nodes or containers_balanced:
             self.converge(databag)
         # save the formation with updated layers
         self.save()


Test patch: