Repository: deis/deis

Issue:
deis ps cli reporting incorrectly
Thanks for the fix on here @carmstrong 
https://github.com/deis/deis/pull/1269

but seems we have a byproduct issue: deis cli now seems to be reporting the apps are down, even if they are up according to `fleetctl status`.

```
$ deis ps --app=stats
=== stats Processes

--- web: 
web.1 up (v19)

--- worker: 
worker.1 down (v19)

--- clock: 
clock.1 down (v19)
```

Fleet:

```
$ fleetctl status stats_v19.worker.1
● stats_v19.worker.1.service - stats_v19.worker.1
   Loaded: loaded (/run/fleet/units/stats_v19.worker.1.service; linked-runtime)
   Active: active (running) since Tue 2014-07-08 14:22:38 UTC; 8min ago
 Main PID: 26805 (sh)
   CGroup: /system.slice/stats_v19.worker.1.service
           ├─26805 /bin/sh -c port=$(docker inspect -f '{{range $k, $v := .ContainerConfig.ExposedPorts }}{{$k}}{{end}}' 172.31.22.109:5000/stats:v19 | cut -d/ -f1) ; docker run --name stats_v19.worker.1 -P -e PORT=$port 172.31.22.109:5000/stats:v19 start worker
           └─26815 docker run --name stats_v19.worker.1 -P -e PORT=5000 172.31.22.109:5000/stats:v19 start worker

```

bug(controller) when deploying a new app getting errors during release
from 

```
$ docker logs deis-controller
...
INFO stats: Release stats-v20 created
[2014-07-08 09:17:27,989: INFO/MainProcess] Received task: api.tasks.deploy_release[3442d174-768f-401b-abf5-34b7410f3cc5]
[2014-07-08 09:17:28,138: WARNING/Worker-7] Creating stats_v20.web.1
[2014-07-08 09:17:28,140: WARNING/Worker-7] Creating stats_v20.worker.1
[2014-07-08 09:17:28,173: WARNING/Worker-7] Creating stats_v20.clock.1
[2014-07-08 09:17:29,479: WARNING/Worker-7] -- skipping announcer for stats_v20.worker.1 - cmd type is start worker
[2014-07-08 09:17:29,480: WARNING/Worker-7] Starting stats_v20.worker.1
[2014-07-08 09:17:29,581: WARNING/Worker-7] -- skipping announcer for stats_v20.clock.1 - cmd type is start clock
[2014-07-08 09:17:29,583: WARNING/Worker-7] Starting stats_v20.clock.1
Triggered job stats_v20.worker.1.service start
Triggered job stats_v20.worker.1-log.service start
Triggered job stats_v20.clock.1.service start
Error creating jobs: unable to find Unit(stats_v20.worker.1-announce.service) in Registry or on filesystem
[2014-07-08 09:17:29,926: WARNING/Worker-7] Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/app/api/models.py", line 47, in _inner
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 267, in _change_state
    return fsm_meta.field.change_state(instance, func, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 161, in change_state
    result = method(instance, *args, **kwargs)
  File "/app/api/models.py", line 336, in deploy
    self._scheduler.start(new_job_id)
  File "/app/scheduler/coreos.py", line 98, in start
    self._start_announcer(name, env)
  File "/app/scheduler/coreos.py", line 114, in _start_announcer
    shell=True, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'fleetctl.sh start -no-block stats_v20.worker.1-announce.service' returned non-zero exit status 1
[2014-07-08 09:17:30,093: WARNING/Worker-7] Starting stats_v20.web.1
Triggered job stats_v20.clock.1-log.service start
Triggered job stats_v20.web.1.service start
Error creating jobs: unable to find Unit(stats_v20.clock.1-announce.service) in Registry or on filesystem
[2014-07-08 09:17:30,471: WARNING/Worker-7] Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/app/api/models.py", line 47, in _inner
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 267, in _change_state
    return fsm_meta.field.change_state(instance, func, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 161, in change_state
    result = method(instance, *args, **kwargs)
  File "/app/api/models.py", line 336, in deploy
    self._scheduler.start(new_job_id)
  File "/app/scheduler/coreos.py", line 98, in start
    self._start_announcer(name, env)
  File "/app/scheduler/coreos.py", line 114, in _start_announcer
    shell=True, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'fleetctl.sh start -no-block stats_v20.clock.1-announce.service' returned non-zero exit status 1
Triggered job stats_v20.web.1-log.service start
Triggered job stats_v20.web.1-announce.service start

Restarting celery worker (/usr/local/bin/celery worker --app=deis --loglevel=INFO --workdir=/app --pidfile=/tmp/celery.pid)
2014-07-08 09:17:49 [69] [INFO] Handling signal: hup
2014-07-08 09:17:49 [69] [INFO] Hang up: Master
2014-07-08 09:17:49 [25940] [INFO] Booting worker with pid: 25940
2014-07-08 09:17:49 [25941] [INFO] Booting worker with pid: 25941
2014-07-08 09:17:49 [25942] [INFO] Booting worker with pid: 25942
2014-07-08 09:17:49 [25943] [INFO] Booting worker with pid: 25943
2014-07-08 09:17:49 [25944] [INFO] Booting worker with pid: 25944
2014-07-08 09:17:49 [25945] [INFO] Booting worker with pid: 25945
2014-07-08 09:17:49 [25946] [INFO] Booting worker with pid: 25946
2014-07-08 09:17:49 [25947] [INFO] Booting worker with pid: 25947
"45819ccada74e8f0c0e3d3de218d1c63b23581eef47c8de358aff8c082214660"
True
True
True
"d7e6dbba0871ed3668d41d34a53caed05bc382d3341bccc364789d5b5c843355"
True
True
True
2014-07-08 09:17:49 [88] [INFO] Worker exiting (pid: 88)
"7b72c8a5dc327c12d28e2963a7ebc31a2d33d8ffd61246a737ad4314d2412031"
True
True
True
2014-07-08 09:17:49 [89] [INFO] Worker exiting (pid: 89)
2014-07-08 09:17:49 [94] [INFO] Worker exiting (pid: 94)
2014-07-08 09:17:49 [97] [INFO] Worker exiting (pid: 97)
"ccc076f026d256d540e927d2560ae822ffe28512e5c0cec2532e0d405f960296"
True
True
True
2014-07-08 09:17:49 [104] [INFO] Worker exiting (pid: 104)
2014-07-08 09:17:49 [114] [INFO] Worker exiting (pid: 114)
"2244296144a4effeec6fb66c7e5a39ad6d2f19a3627718ee0cbd534eb69db06d"
True
True
True
2014-07-08 09:17:49 [120] [INFO] Worker exiting (pid: 120)

Restarting celery worker (/usr/local/bin/celery worker --app=deis --loglevel=INFO --workdir=/app --pidfile=/tmp/celery.pid)
2014-07-08 09:17:54 [69] [INFO] Handling signal: hup
2014-07-08 09:17:54 [69] [INFO] Hang up: Master
2014-07-08 09:17:54 [25989] [INFO] Booting worker with pid: 25989
2014-07-08 09:17:54 [25990] [INFO] Booting worker with pid: 25990
2014-07-08 09:17:54 [25991] [INFO] Booting worker with pid: 25991
2014-07-08 09:17:54 [25992] [INFO] Booting worker with pid: 25992
2014-07-08 09:17:54 [25993] [INFO] Booting worker with pid: 25993
2014-07-08 09:17:54 [25994] [INFO] Booting worker with pid: 25994
2014-07-08 09:17:54 [25995] [INFO] Booting worker with pid: 25995
2014-07-08 09:17:54 [25940] [INFO] Worker exiting (pid: 25940)
2014-07-08 09:17:54 [25941] [INFO] Worker exiting (pid: 25941)
2014-07-08 09:17:55 [25942] [INFO] Worker exiting (pid: 25942)
2014-07-08 09:17:55 [25943] [INFO] Worker exiting (pid: 25943)
2014-07-08 09:17:55 [25944] [INFO] Worker exiting (pid: 25944)
2014-07-08 09:17:55 [25945] [INFO] Worker exiting (pid: 25945)
2014-07-08 09:17:55 [25946] [INFO] Worker exiting (pid: 25946)
2014-07-08 09:17:55 [25947] [INFO] Worker exiting (pid: 25947)
2014-07-08 09:17:55 [25996] [INFO] Booting worker with pid: 25996
[2014-07-08 09:18:52,315: WARNING/Worker-7] Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/app/api/models.py", line 47, in _inner
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 267, in _change_state
    return fsm_meta.field.change_state(instance, func, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 161, in change_state
    result = method(instance, *args, **kwargs)
  File "/app/api/models.py", line 336, in deploy
    self._scheduler.start(new_job_id)
  File "/app/scheduler/coreos.py", line 99, in start
    self._wait_for_announcer(name, env)
  File "/app/scheduler/coreos.py", line 126, in _wait_for_announcer
    raise RuntimeError('Container failed to start')
RuntimeError: Container failed to start
[2014-07-08 09:18:52,754: INFO/MainProcess] Task api.tasks.deploy_release[3442d174-768f-401b-abf5-34b7410f3cc5] succeeded in 84.7626976409s: None
"5d6c2d8479411272910a04edf40929687351bcb27e303626c61c2ac900d032d8"
True
True
True
2014-07-08 09:18:53 [99] [INFO] Worker exiting (pid: 99)

 -------------- celery@b6fe59dd0b6e v3.1.11 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.14.6+-x86_64-with-Ubuntu-12.04-precise
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         deis:0x29c2bd0
- ** ---------- .> transport:   redis://172.31.41.76:6379/0
- ** ---------- .> results:     redis://172.31.41.76:6379/0
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . api.tasks.create_cluster
  . api.tasks.deploy_release
  . api.tasks.destroy_cluster
  . api.tasks.run_command
  . api.tasks.start_containers
  . api.tasks.stop_containers

[2014-07-08 09:18:56,631: INFO/MainProcess] Connected to redis://172.31.41.76:6379/0
[2014-07-08 09:18:56,639: INFO/MainProcess] mingle: searching for neighbors
[2014-07-08 09:18:57,645: INFO/MainProcess] mingle: all alone
[2014-07-08 09:18:57,656: WARNING/MainProcess] celery@b6fe59dd0b6e ready.

```

Jenkins log calling Deis-builder:

```
remote: Successfully built 5d6c2d847941        
remote: -----> Pushing image to private registry        
remote: 
remote:        Launching... Connection to deis.stats-production.cronycle.net closed by remote host.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
To ssh://git@deis.stats-production.cronycle.net:2222/stats.git
 + ebb0b67...8081e80 HEAD -> master (forced update)
error: failed to push some refs to 'ssh://git@deis.stats-production.cronycle.net:2222/stats.git'
Build step 'Execute shell' marked build as failure
Finished: FAILURE
```

could be related to the announcers being turning off. (https://github.com/deis/deis/pull/1269)



Discussion:
Good point--they're still logically "up" even if they are (intentionally) not announced and routed.

i guess we should just get the status off of fleet. if it says 'active', it should show up as 'up'

@bacongobbler Started looking through the logic to address this -  looks like the state machine is what handles a container being marked as "down"?

correct. If the function that is decorated raises an Exception i.e. `_wait_for_announcer` crashes during https://github.com/deis/deis/blob/master/controller/api/models.py#L314-L319 (see #1275), then the FSM reverts to the `crashed` state, which in this case is `down`. IOW, the fix is to stop waiting for the announce container to come up on non-web/cmd process types.

Yep. In a lot of cases in coreos.py we're still referencing the announcer. I'm refactoring those to have the same condition logic so we skip handling announcers.

:+1: 

Maybe its shorter to just add a condition on _wait_for_announcer on top (https://github.com/deis/deis/blob/master/controller/scheduler/coreos.py#L117) so it just returns with a positive value instead of doing anything when its 'web' or 'cmd'. 
eg.

```
if (name == 'web') or (name == 'cmd'):
  return true
else:
  ...      
```

this way if we do refactor things in future so we have announcers (maybe for other types of buildpacks where there ARE non-web/cmd processes that needs to be exposed to router we have just one place to edit)

This is unrelated to announcers - most definitely it's the ELB timeout issue.

Ah, well it's actually both issues. Just saw:

```
  File "/app/api/models.py", line 336, in deploy
    self._scheduler.start(new_job_id)
  File "/app/scheduler/coreos.py", line 99, in start
    self._wait_for_announcer(name, env)
  File "/app/scheduler/coreos.py", line 126, in _wait_for_announcer
    raise RuntimeError('Container failed to start')
RuntimeError: Container failed to start
```

Looks like we're calling `_wait_for_announcer` when starting an idle job.


Resolving patch:
diff --git a/controller/api/models.py b/controller/api/models.py
--- a/controller/api/models.py
+++ b/controller/api/models.py
@@ -304,19 +304,25 @@ def _get_command(self):
 
     _command = property(_get_command)
 
+    def _command_announceable(self):
+        return self._command.lower() in ['start web', '']
+
     @close_db_connections
     @transition(field=state, source=INITIALIZED, target=CREATED)
     def create(self):
         image = self.release.image
         c_type = self.type
-        self._scheduler.create(self._job_id, image, self._command.format(**locals()))
+        self._scheduler.create(name=self._job_id,
+                               image=image,
+                               command=self._command.format(**locals()),
+                               use_announcer=self._command_announceable())
 
     @close_db_connections
     @transition(field=state,
                 source=[CREATED, UP, DOWN],
                 target=UP, crashed=DOWN)
     def start(self):
-        self._scheduler.start(self._job_id)
+        self._scheduler.start(self._job_id, self._command_announceable())
 
     @close_db_connections
     @transition(field=state,
@@ -332,15 +338,18 @@ def deploy(self, release):
         new_job_id = self._job_id
         image = self.release.image
         c_type = self.type
-        self._scheduler.create(new_job_id, image, self._command.format(**locals()))
-        self._scheduler.start(new_job_id)
+        self._scheduler.create(name=new_job_id,
+                               image=image,
+                               command=self._command.format(**locals()),
+                               use_announcer=self._command_announceable())
+        self._scheduler.start(new_job_id, self._command_announceable())
         # destroy old container
-        self._scheduler.destroy(old_job_id)
+        self._scheduler.destroy(old_job_id, self._command_announceable())
 
     @close_db_connections
     @transition(field=state, source=UP, target=DOWN)
     def stop(self):
-        self._scheduler.stop(self._job_id)
+        self._scheduler.stop(self._job_id, self._command_announceable())
 
     @close_db_connections
     @transition(field=state,
@@ -348,7 +357,7 @@ def stop(self):
                 target=DESTROYED)
     def destroy(self):
         # TODO: add check for active connections before killing
-        self._scheduler.destroy(self._job_id)
+        self._scheduler.destroy(self._job_id, self._command_announceable())
 
     @transition(field=state,
                 source=[INITIALIZED, CREATED, DESTROYED],
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -46,9 +46,17 @@ def tearDown(self):
         """
         return
 
+    # announcer helpers
+
+    def _log_skipped_announcer(self, action, name):
+        """
+        Logs a message stating that this operation doesn't require an announcer
+        """
+        print "-- skipping announcer {} for {}".format(action, name)
+
     # job api
 
-    def create(self, name, image, command='', template=None):
+    def create(self, name, image, command='', template=None, use_announcer=True):
         """
         Create a new job
         """
@@ -57,11 +65,10 @@ def create(self, name, image, command='', template=None):
         self._create_container(name, image, command, template or CONTAINER_TEMPLATE, env)
         self._create_log(name, image, command, LOG_TEMPLATE, env)
 
-        # only announce web and cmd processes
-        if command.lower() in ['start web', '']:
+        if use_announcer:
             self._create_announcer(name, image, command, ANNOUNCE_TEMPLATE, env)
         else:
-            print "-- skipping announcer for {} - cmd type is {}".format(name,command)
+            self._log_skipped_announcer('create', name)
 
     def _create_container(self, name, image, command, template, env):
         l = locals().copy()
@@ -87,7 +94,7 @@ def _create_log(self, name, image, command, template, env):
         return subprocess.check_call('fleetctl.sh submit {name}-log.service'.format(**locals()),  # noqa
                                      shell=True, env=env)
 
-    def start(self, name):
+    def start(self, name, use_announcer=True):
         """
         Start an idle job
         """
@@ -95,8 +102,12 @@ def start(self, name):
         env = self.env.copy()
         self._start_container(name, env)
         self._start_log(name, env)
-        self._start_announcer(name, env)
-        self._wait_for_announcer(name, env)
+
+        if use_announcer:
+            self._start_announcer(name, env)
+            self._wait_for_announcer(name, env)
+        else:
+            self._log_skipped_announcer('start', name)
 
     def _start_log(self, name, env):
         subprocess.check_call(
@@ -125,13 +136,18 @@ def _wait_for_announcer(self, name, env):
         else:
             raise RuntimeError('Container failed to start')
 
-    def stop(self, name):
+    def stop(self, name, use_announcer=True):
         """
         Stop a running job
         """
         print 'Stopping {name}'.format(**locals())
         env = self.env.copy()
-        self._stop_announcer(name, env)
+
+        if use_announcer:
+            self._stop_announcer(name, env)
+        else:
+            self._log_skipped_announcer('stop', name)
+
         self._stop_container(name, env)
         self._stop_log(name, env)
 
@@ -150,13 +166,18 @@ def _stop_log(self, name, env):
             'fleetctl.sh stop -block-attempts=600 {name}-log.service'.format(**locals()),
             shell=True, env=env)
 
-    def destroy(self, name):
+    def destroy(self, name, use_announcer=True):
         """
         Destroy an existing job
         """
         print 'Destroying {name}'.format(**locals())
         env = self.env.copy()
-        self._destroy_announcer(name, env)
+
+        if use_announcer:
+            self._destroy_announcer(name, env)
+        else:
+            self._log_skipped_announcer('destroy', name)
+
         self._destroy_container(name, env)
         self._destroy_log(name, env)
 
diff --git a/controller/scheduler/mock.py b/controller/scheduler/mock.py
--- a/controller/scheduler/mock.py
+++ b/controller/scheduler/mock.py
@@ -26,25 +26,25 @@ def tearDown(self):
 
     # job api
 
-    def create(self, name, image, command):
+    def create(self, name, image, command, use_announcer):
         """
         Create a new job
         """
         return {'state': 'inactive'}
 
-    def start(self, name):
+    def start(self, name, use_announcer):
         """
         Start an idle job
         """
         return {'state': 'active'}
 
-    def stop(self, name):
+    def stop(self, name, use_announcer):
         """
         Stop a running job
         """
         return {'state': 'inactive'}
 
-    def destroy(self, name):
+    def destroy(self, name, use_announcer):
         """
         Destroy an existing job
         """


Test patch: