Issue 1:
FYI. we should add a secret manager to store the pull secrets for different namespaces. Thus for duplicate secrets, we have no need to talk to api server. Actually I am working on this manager, but I have no idea about when to remove a secret.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

v1.4.0

Environment:

Cloud provider or hardware configuration: GCE + custome kubernetes

OS (e.g. from /etc/os-release): "CoreOS 1068.8.0 (MoreOS)"

Kernel (e.g. uname -a): 4.6.3-coreos

Install tools:

Others:

What happened:

kube-scheduler entered keeps entering restarting due to nil pointer exception. It seems like it happens when a node or pod gets removed from the cluster.

Here is the journal output for the kube-scheduler

event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service1-4068773324-24s7s", UID:"1268223f-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"47435", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service1-4068773324-24s7s to h-stg-wk1-3jvo
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service2-3685785901-549bk", UID:"12b068d6-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"47449", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service2-3685785901-549bk to h-stg-wk1-vkym
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service3-1539738932-4zdq2", UID:"8ffd5a12-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"50531", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service3-1539738932-4zdq2 to h-stg-wk1-y8gt
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:479
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:458
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:62
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/sigpanic_unix.go:24
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182
Sep 29 14:04:15 node-123 docker[15022]: <autogenerated>:52
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:2086
Sep 29 14:04:15 node-123 docker[15022]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Sep 29 14:04:15 node-123 docker[15022]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 29 14:04:15 node-123 docker[15022]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1aa376c]
Sep 29 14:04:15 node-123 docker[15022]: goroutine 16 [running]:
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:500 +0x1a1
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:458 +0x243
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*NodeInfo).removePod(0x0, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192 +0x7c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).removePod(0xc420ce3d40, 0xc420eacc80, 0xc4210509a0, 0x1d)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229 +0x88
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).RemovePod(0xc420ce3d40, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252 +0x22d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).deletePodFromCache(0xc42009ebb0, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196 +0xa9
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).(k8s.io/kubernetes/plugin/pkg/scheduler/factory.deletePodFromCache)-fm(0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134 +0x3e
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.ResourceEventHandlerFuncs.OnDelete(0xc4203f31d0, 0xc4203f31f0, 0xc4203f3200, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182 +0x49
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc420d5cb20, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]: <autogenerated>:52 +0x78
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.NewIndexerInformer.func1(0x2d35cc0, 0xc421050960, 0xc421050960, 0x2d35cc0)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318 +0x4ee
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Pop(0xc42009ec60, 0xc4200152c0, 0x0, 0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420 +0x22a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).processLoop(0xc420150150)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126 +0x3c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).(k8s.io/kubernetes/pkg/client/cache.processLoop)-fm()
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x2a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc4210d7f70)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84 +0x19
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc4210d7f70, 0x3b9aca00, 0x0, 0x2b48701, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85 +0xad
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.Until(0xc4210d7f70, 0x3b9aca00, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x4d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).Run(0xc420150150, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x1af
Sep 29 14:04:15 node-123 docker[15022]: created by k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).Run
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391 +0x12f
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Unit entered failed state.
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Failed with result 'exit-code'.

Based on the source code it looks like here:

https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/cache.go#L228

we get pod from cache without checking if it actually exists there. Since it is a pointer we can call removePod on it. But once inside https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/node_info.go#L192 we try to get filed of a nil pointer and so it crashes.

What you expected to happen: Expected not to get a nil pointer exception.

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

This is also related to #31968 but this time exception is in a different place. Also it looks like this might cause some of our pods stay in pending state indefinitely but I am not sure.