Repository: deis/deis

Issue:
Unable to Delete Application
Cannot delete an application.

```
jwaldrip at Jasons-MacBook-Pro in ~/dev/src/github.com/brandfolder/boulder (master)
$ deis apps:destroy

 !    WARNING: Potentially Destructive Action
 !    This command will destroy the application: brandfolder-production
 !    To proceed, type "brandfolder-production" or re-run this command with --confirm=brandfolder-production

> brandfolder-production
Destroying brandfolder-production...
500 INTERNAL SERVER ERROR
<h1>Server Error (500)</h1>
```

Info from `deisctl journal controller`:

```
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: INFO brandfolder-production: domain admin.brandfolder.tv removed
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: ERROR Internal Server Error: /v1/apps/brandfolder-production
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: Traceback (most recent call last):
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: return view_func(*args, **kwargs)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/rest_framework/viewsets.py", line 85, in view
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: return self.dispatch(request, *args, **kwargs)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 407, in dispatch
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: response = self.handle_exception(exc)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 404, in dispatch
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: response = handler(request, *args, **kwargs)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/rest_framework/mixins.py", line 85, in destroy
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: self.perform_destroy(instance)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/rest_framework/mixins.py", line 89, in perform_destroy
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: instance.delete()
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/app/api/models.py", line 190, in delete
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: return super(App, self).delete(*args, **kwargs)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 695, in delete
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: collector.delete()
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/django/db/models/deletion.py", line 282, in delete
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: sender=model, instance=obj, using=self.using
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/django/dispatch/dispatcher.py", line 185, in send
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: response = receiver(signal=self, sender=sender, **named)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/app/api/models.py", line 930, in _etcd_purge_domains
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: _etcd_client.delete('/deis/domains/{}'.format(app))
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 389, in delete
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: self.key_endpoint + key, self._MDELETE, params=kwds)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 582, in api_execute
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: return self._handle_server_response(response)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 598, in _handle_server_response
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: etcd.EtcdError.handle(**r)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: File "/usr/local/lib/python2.7/dist-packages/etcd/__init__.py", line 142, in handle
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: raise exc(msg)
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: KeyError: 'Key not found : /deis/domains/brandfolder-production'
Mar 23 14:01:40 ip-10-21-1-108.ec2.internal sh[6985]: 10.21.2.21 "DELETE /v1/apps/brandfolder-production HTTP/1.0" 500 27 "python-requests/2.4.3 CPython/2.7.5 Darwin/14.1.0"
```



Discussion:
@bacongobbler would you mind taking a look at this?  We should be ignoring missing key errors on app deletion.

We proposed that in an earlier bug but it was closed: https://github.com/deis/deis/pull/2865

After looking at #2865, I think it should be merged.  It's clearly an etcd issue, but it can also be triggered by out-of-band manipulation of the etcd keyspace -- something we shouldn't just barf on.  Not only would this resolve a 500 error users are clearly seeing in the wild, but it's also the right logic because the net effect of swallowing the `KeyError` is the same (no more key!).

@jwaldrip as a workaround, you should be able to `etcdctl set /deis/domains/brandfolder-production` then redo the `apps:destroy`.


Resolving patch:
diff --git a/controller/api/models.py b/controller/api/models.py
--- a/controller/api/models.py
+++ b/controller/api/models.py
@@ -886,8 +886,11 @@ def _etcd_publish_key(**kwargs):
 
 def _etcd_purge_key(**kwargs):
     key = kwargs['instance']
-    _etcd_client.delete('/deis/builder/users/{}/{}'.format(
-        key.owner.username, fingerprint(key.public)))
+    try:
+        _etcd_client.delete('/deis/builder/users/{}/{}'.format(
+            key.owner.username, fingerprint(key.public)))
+    except KeyError:
+        pass
 
 
 def _etcd_purge_user(**kwargs):
@@ -908,7 +911,10 @@ def _etcd_create_app(**kwargs):
 
 def _etcd_purge_app(**kwargs):
     appname = kwargs['instance']
-    _etcd_client.delete('/deis/services/{}'.format(appname), dir=True, recursive=True)
+    try:
+        _etcd_client.delete('/deis/services/{}'.format(appname), dir=True, recursive=True)
+    except KeyError:
+        pass
 
 
 def _etcd_publish_domains(**kwargs):
@@ -926,7 +932,10 @@ def _etcd_purge_domains(**kwargs):
         _etcd_client.write('/deis/domains/{}'.format(app),
                            ' '.join(str(d.domain) for d in app_domains))
     else:
-        _etcd_client.delete('/deis/domains/{}'.format(app))
+        try:
+            _etcd_client.delete('/deis/domains/{}'.format(app))
+        except KeyError:
+            pass
 
 
 # Log significant app-related events


Test patch: