Repository: deis/deis

Issue:
Healthcheck config changes aren't removed from etcd
I've strange issue with my deisv1, we've had healthchecks turned on once in the past, but it was failing with more than 3-4 containers, so we've turned it off, and removed all healthcheck related config values. However from time to time, application is failing to start because healthcheck error, there is no healthcheck-related configuration when running `deis config -a <app name>`, but after looking directly into etcd I've noticed this:

```
$ etcdctl ls deis/config/app | grep health
/deis/config/app/healthcheck_url
/deis/config/app/healthcheck_initial_delay
/deis/config/app/healthcheck_timeout
```

I remember we've removed it manualy from etcd in the past (etcdctl rm...), but now it's back. I've thought it might be related to recent controller restart (IIRC all etcd config values are propagated on restart) I've also looked into Deis postgresql database, and config attached to this particular app doesn't have any healthcheck related settings.

Any advice on this would be greatly appreciated, because it breaks our production environement from time to time. 



Discussion:
I recall observing this same behavior as well a while back, and had to perform similar steps to manually remove the keys from etcd because the HEALTHCHECK_URL was not showing up in the `deis config` output, but was there in etcd. One additional datapoint is that we had performed a migration upgrade since the last time we removed the healthchecks, so the etcd keys had to have been populated from the db at some point, since the etcd cluster was replaced during the migration upgrade.

I believe I found the issue: in https://github.com/deis/deis/blob/master/controller/api/management/commands/load_db_state_to_etcd.py#L13-L15 I see that the command loads _ALL CONFIGS_ into etcd, not just the latest config on a per app basis. In addition to generating a large amount of unnecessary calls to etcd, this will also cause the invalid behavior observed here for _any_ config value that is ever set and then later removed from the config.


Resolving patch:
diff --git a/controller/api/management/commands/load_db_state_to_etcd.py b/controller/api/management/commands/load_db_state_to_etcd.py
--- a/controller/api/management/commands/load_db_state_to_etcd.py
+++ b/controller/api/management/commands/load_db_state_to_etcd.py
@@ -1,6 +1,8 @@
+from __future__ import print_function
+
 from django.core.management.base import BaseCommand
 
-from api.models import Key, App, Domain, Certificate, Config
+from api.models import Key, App, Domain, Certificate
 
 
 class Command(BaseCommand):
@@ -9,8 +11,11 @@ class Command(BaseCommand):
     """
     def handle(self, *args, **options):
         """Publishes Deis platform state from the database to etcd."""
-        print "Publishing DB state to etcd..."
-        for model in (Key, App, Domain, Certificate, Config):
+        print("Publishing DB state to etcd...")
+        for app in App.objects.all():
+            app.save()
+            app.config_set.latest().save()
+        for model in (Key, Domain, Certificate):
             for obj in model.objects.all():
                 obj.save()
-        print "Done Publishing DB state to etcd."
+        print("Done Publishing DB state to etcd.")


Test patch: