Issue 1:
(k8s 1.1.4)

After restarting my master server components, 2 out of 5 of my DaemonSet pods were terminated (why?) and never recreated. It seems like all ds pods were requeued during this event, however the IDs of the currently running pods do not even match what is reported by the DaemonSet.

Restarting the kubelet on faulty nodes does nothing but re-adding a SyncLoop

Restarting the kube-controller-manager again shows Waiting for pods controller to sync, requeuing ds ingress-apis

kube-scheduler shows no relevant information

The first message in the kubelet log ("... The image pull may not succeed.") makes me think it might be related to #18947, however:

a pull should not have been triggered since the imagePullPolicy is IfNotPresent, and the image was present before the restart

SyncLoop (ADD) shouldn't have happened in the first place since 5 pods were already running before the restart

Current state of the ds:

Name: ingress-apis
Image(s): nginx:1.9.7
Selector: app=ingress,tier=backend
Node-Selector: <none>
Labels: app=ingress,tier=backend
Desired Number of Nodes Scheduled: 5
Current Number of Nodes Scheduled: 5
Number of Nodes Misscheduled: 0
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 2 Failed
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
40m 40m 1 {daemon-set } SuccessfulCreate Created pod: ingress-apis-0svoi
40m 40m 1 {daemon-set } SuccessfulCreate Created pod: ingress-apis-qn8tk
39m 39m 1 {daemon-set } SuccessfulCreate Created pod: ingress-apis-4p04e
39m 39m 1 {daemon-set } SuccessfulCreate Created pod: ingress-apis-2ncfy
39m 39m 1 {daemon-set } SuccessfulCreate Created pod: ingress-apis-v6iuf

Current pods:

NAME READY STATUS RESTARTS AGE
ingress-apis-jwlqo 1/1 Running 0 19h
ingress-apis-mabp8 1/1 Running 0 17h
ingress-apis-rte9q 1/1 Running 0 19h

kubelet logs on a faulty node:

--Jan 20 09:44:11--
kubelet.go:1477] Unable to retrieve pull secret default/docker-registry for default/ingress-apis-mensq due to Get https://k8s.freeletics.com/api/v1/namespaces/default/secrets/docker-registry: dial tcp 10.0.31.59:443: connection refused. The image pull may not succeed.
kubelet.go:2012] SyncLoop (ADD): "ingress-apis-0svoi_default"
kubelet.go:1862] Pod "ingress-apis-0svoi_default": HostPort is already allocated, ignoring: [[0].port: duplicate value '9080/TCP']
kubelet.go:2015] SyncLoop (UPDATE): "ingress-apis-mensq_default"
manager.go:1404] Killing container "bac67ab2623eb60b6d23e714feee4d74be6b5c3e7417f07b08bd874d2183245f nginx default/ingress-apis-mensq" with 30 second grace period
manager.go:1438] Container "bac67ab2623eb60b6d23e714feee4d74be6b5c3e7417f07b08bd874d2183245f nginx default/ingress-apis-mensq" exited after 68.511739ms
manager.go:1444] No ref for pod '"bac67ab2623eb60b6d23e714feee4d74be6b5c3e7417f07b08bd874d2183245f nginx default/ingress-apis-mensq"'
manager.go:1404] Killing container "eb0b4d358a9afe7aa2ac18d7f6eec57a6d2cd54506e4ae18096c7996c011de77 default/ingress-apis-mensq" with 30 second grace period
manager.go:1438] Container "eb0b4d358a9afe7aa2ac18d7f6eec57a6d2cd54506e4ae18096c7996c011de77 default/ingress-apis-mensq" exited after 213.225172ms
manager.go:1444] No ref for pod '"eb0b4d358a9afe7aa2ac18d7f6eec57a6d2cd54506e4ae18096c7996c011de77 default/ingress-apis-mensq"'
kubelet.go:2015] SyncLoop (UPDATE): "ingress-apis-mensq_default"
kubelet.go:2018] SyncLoop (REMOVE): "ingress-apis-mensq_default"
kubelet.go:2108] Failed to delete pod "ingress-apis-mensq_default", err: pod not found
--Jan 20 09:44:22--

kube-controller-manager logs:

--Jan 20 09:44:14--
controller.go:493] Waiting for pods controller to sync, requeuing ds ingress-apis
controller.go:493] Waiting for pods controller to sync, requeuing ds ingress-apis
controller.go:493] Waiting for pods controller to sync, requeuing ds ingress-apis
controller.go:493] Waiting for pods controller to sync, requeuing ds ingress-apis
controller.go:493] Waiting for pods controller to sync, requeuing ds ingress-apis
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"4b05bf3e-9c18-11e5-b4d2-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"63618203", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-0svoi
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"4b05bf3e-9c18-11e5-b4d2-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"63618203", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-qn8tk
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"4b05bf3e-9c18-11e5-b4d2-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"63618203", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-4p04e
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"4b05bf3e-9c18-11e5-b4d2-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"63618203", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-2ncfy
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"4b05bf3e-9c18-11e5-b4d2-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"63618203", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-v6iuf
--Jan 20 09:44:20--

Before someone asks, although this is obvious, I solved it by deleting and recreating the ds (identically):

kubelet logs on a previously faulty node:

--Jan 20 10:52:33--
kubelet.go:2015] SyncLoop (UPDATE): "ingress-apis-0svoi_default"
kubelet.go:2018] SyncLoop (REMOVE): "ingress-apis-0svoi_default"
kubelet.go:2108] Failed to delete pod "ingress-apis-0svoi_default", err: pod not found
kubelet.go:2012] SyncLoop (ADD): "ingress-apis-9us1a_default"
manager.go:1707] Need to restart pod infra container for "ingress-apis-9us1a_default" because it is not found
--Jan 20 10:52:34--

kube-controller-manager logs:

--Jan 20 10:52:40--
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"ed06f7f4-bf63-11e5-91dc-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"64118991", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-hgcn1
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"ed06f7f4-bf63-11e5-91dc-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"64118991", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-9us1a
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"ed06f7f4-bf63-11e5-91dc-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"64118991", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-lsobz
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"ed06f7f4-bf63-11e5-91dc-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"64118991", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-ogjrw
event.go:206] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"ingress-apis", UID:"ed06f7f4-bf63-11e5-91dc-0a59d1e77755", APIVersion:"extensions", ResourceVersion:"64118991", FieldPath:""}): reason: 'SuccessfulCreate' Created pod: ingress-apis-9tae0
--Jan 20 10:52:41--

Issue 2:
https://travis-ci.org/GoogleCloudPlatform/kubernetes/jobs/69776608

panic: test timed out after 5m0s
goroutine 37 [running]:
runtime.panic(0xaef900, 0xc2080aa020)
/usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
testing.func·008()
/usr/local/go/src/pkg/testing/testing.go:629 +0x12e
created by time.goFunc
/usr/local/go/src/pkg/time/sleep.go:121 +0x55
goroutine 16 [chan receive, 2 minutes]:
testing.RunTests(0xee2130, 0x13b34a0, 0x4, 0x4, 0xe3fe01)
/usr/local/go/src/pkg/testing/testing.go:505 +0xbb8
testing.Main(0xee2130, 0x13b34a0, 0x4, 0x4, 0x13e7900, 0x0, 0x0, 0x13e7900, 0x0, 0x0)
/usr/local/go/src/pkg/testing/testing.go:435 +0xa3
main.main()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_testmain.go:105 +0x18c
goroutine 19 [finalizer wait, 5 minutes]:
runtime.park(0x435ff0, 0x13de610, 0x13b8349)
/usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x13de610, 0x13b8349)
/usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
/usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1445
goroutine 33 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x13e1280)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x96
created by github.com/golang/glog.init·1
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2d6
goroutine 17 [syscall, 5 minutes]:
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1445
goroutine 52 [select, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).run(0xc20804d1a0, 0xd24260, 0x3, 0xd21a40, 0x3)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:49 +0x3db
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·001()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0xb9
github.com/GoogleCloudPlatform/kubernetes/pkg/util.func·004()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:112 +0x5f
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Until(0xc20804d1c0, 0x12a05f200, 0x0)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:113 +0xbc
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Forever(0xc20804d1c0, 0x12a05f200)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:94 +0x48
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).Elect
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0x1f6
goroutine 49 [select, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).run(0xc20804d0e0, 0xd24260, 0x3, 0xd21a40, 0x3)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:49 +0x3db
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·001()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0xb9
github.com/GoogleCloudPlatform/kubernetes/pkg/util.func·004()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:112 +0x5f
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Until(0xc20804d100, 0x12a05f200, 0x0)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:113 +0xbc
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Forever(0xc20804d100, 0x12a05f200)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:94 +0x48
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).Elect
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0x1f6
goroutine 55 [select, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).run(0xc20804d320, 0xd24260, 0x3, 0xd21a40, 0x3)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:49 +0x3db
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·001()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0xb9
github.com/GoogleCloudPlatform/kubernetes/pkg/util.func·004()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:112 +0x5f
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Until(0xc20804d340, 0x12a05f200, 0x0)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:113 +0xbc
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Forever(0xc20804d340, 0x12a05f200)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:94 +0x48
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*etcdMasterElector).Elect
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/etcd_master.go:37 +0x1f6
goroutine 58 [select, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/pkg/watch.(*Broadcaster).distribute(0xc2080b1110, 0xd372d0, 0x8, 0x7f1745fd6ef0, 0xc208132d20)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/watch/mux.go:182 +0x3bd
github.com/GoogleCloudPlatform/kubernetes/pkg/watch.(*Broadcaster).loop(0xc2080b1110)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/watch/mux.go:163 +0xc3
created by github.com/GoogleCloudPlatform/kubernetes/pkg/watch.NewBroadcaster
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/watch/mux.go:70 +0x127
goroutine 57 [chan receive, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.Test(0xc2080a6750)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/master_test.go:83 +0x306
testing.tRunner(0xc2080a6750, 0x13b34e8)
/usr/local/go/src/pkg/testing/testing.go:422 +0x110
created by testing.RunTests
/usr/local/go/src/pkg/testing/testing.go:504 +0xb46
goroutine 59 [select, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*notifier).serviceLoop(0xc208093b30, 0xc2080977a0)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:105 +0x363
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·005()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:87 +0x90
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.func·008()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:115 +0x5f
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.Until(0x7f174580bec0, 0x0, 0xc2080976e0)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:116 +0xd3
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.Notify(0x7f1745fd7030, 0xc20804d460, 0xd16620, 0x0, 0xd26520, 0x2, 0x7f1745fd7058, 0xc20804d480, 0xc2080976e0)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:87 +0x429
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·006()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/master_test.go:72 +0x145
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.func·007()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:95 +0xaa
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.After
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:97 +0xf7
goroutine 61 [chan send, 2 minutes]:
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.(*Fake).Elect(0xc20804d460, 0xd16620, 0x0, 0xd26520, 0x2, 0x0, 0x0)
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/fake.go:38 +0x23e
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·003()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:45 +0x13c
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.func·008()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:115 +0x5f
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.Until(0x7f1745807f50, 0x0, 0xc2080976e0)
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:116 +0xd3
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election.func·004()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:84 +0x14f
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.func·007()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:95 +0xaa
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.After
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:97 +0xf7
goroutine 62 [semacquire, 2 minutes]:
sync.runtime_Syncsemacquire(0xc2080a54d0)
/usr/local/go/src/pkg/runtime/sema.goc:257 +0xc0
sync.(*Cond).Wait(0xc2080a54c0)
/usr/local/go/src/pkg/sync/cond.go:62 +0xb5
sync.*Cond.Wait·fm()
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election/_test/_obj_test/master.go:104 +0x34
github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.func·007()
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:95 +0xaa
created by github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime.After
/home/travis/gopath/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/runtime/util.go:97 +0xf7
FAIL github.com/GoogleCloudPlatform/kubernetes/contrib/mesos/pkg/election 300.121s

/cc @jdef, @davidopp