Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.

Discussion:
c.registeredPods[key] = pod if prev != nil { for name := range getConfigMapNames(prev) { <ins> // On an update, the .Add() call above will have re-incremented the // ref count of any existing items, so any configmaps that are in both // names and prev need to have their ref counts decremented. Any that // are only in prev need to be completely removed. This unconditional // call takes care of both cases. </ins> c.configMapStore.Delete(prev.Namespace, name) } }

volumes                []string } <del> func podWithConfigMaps(ns, name string, toAttach configMapsToAttach) *v1.Pod { </del> <ins> func podWithConfigMaps(ns, podName string, toAttach configMapsToAttach) *v1.Pod { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: ns, <del> Name:      name, </del> <ins> Name:      podName, </ins> }, Spec: v1.PodSpec{}, }

manager.RegisterPod(podWithConfigMaps("ns1", "other-name", s2)) manager.UnregisterPod(podWithConfigMaps("ns1", "other-name", s2)) <ins> s5 := configMapsToAttach{ containerEnvConfigMaps: []envConfigMaps{ {envVarNames: []string{"s7"}}, {envFromNames: []string{"s70"}}, }, } // Check the no-op update scenario manager.RegisterPod(podWithConfigMaps("ns1", "noop-pod", s5)) manager.RegisterPod(podWithConfigMaps("ns1", "noop-pod", s5)) </ins> refs := func(ns, name string) int { store.lock.Lock() defer store.lock.Unlock()

} return item.refCount } <del> assert.Equal(t, refs("ns1", "s1"), 1) assert.Equal(t, refs("ns1", "s10"), 1) assert.Equal(t, refs("ns1", "s2"), 1) assert.Equal(t, refs("ns1", "s3"), 3) assert.Equal(t, refs("ns1", "s30"), 2) assert.Equal(t, refs("ns1", "s4"), 2) assert.Equal(t, refs("ns1", "s5"), 4) assert.Equal(t, refs("ns1", "s50"), 2) assert.Equal(t, refs("ns1", "s6"), 0) assert.Equal(t, refs("ns1", "s60"), 0) assert.Equal(t, refs("ns1", "s7"), 0) </del> <ins> assert.Equal(t, 1, refs("ns1", "s1")) assert.Equal(t, 1, refs("ns1", "s10")) assert.Equal(t, 1, refs("ns1", "s2")) assert.Equal(t, 3, refs("ns1", "s3")) assert.Equal(t, 2, refs("ns1", "s30")) assert.Equal(t, 2, refs("ns1", "s4")) assert.Equal(t, 4, refs("ns1", "s5")) assert.Equal(t, 2, refs("ns1", "s50")) assert.Equal(t, 0, refs("ns1", "s6")) assert.Equal(t, 0, refs("ns1", "s60")) assert.Equal(t, 1, refs("ns1", "s7")) assert.Equal(t, 1, refs("ns1", "s70")) </ins> } func TestCachingConfigMapManager(t *testing.T) {

c.registeredPods[key] = pod if prev != nil { for name := range getSecretNames(prev) { <ins> // On an update, the .Add() call above will have re-incremented the // ref count of any existing secrets, so any secrets that are in both // names and prev need to have their ref counts decremented. Any that // are only in prev need to be completely removed. This unconditional // call takes care of both cases. </ins> c.secretStore.Delete(prev.Namespace, name) } }

containerEnvSecrets  []envSecrets } <del> func podWithSecrets(ns, name string, toAttach secretsToAttach) *v1.Pod { </del> <ins> func podWithSecrets(ns, podName string, toAttach secretsToAttach) *v1.Pod { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: ns, <del> Name:      name, </del> <ins> Name:      podName, </ins> }, Spec: v1.PodSpec{}, }

manager.RegisterPod(podWithSecrets("ns1", "other-name", s2)) manager.UnregisterPod(podWithSecrets("ns1", "other-name", s2)) <ins> s5 := secretsToAttach{ containerEnvSecrets: []envSecrets{ {envVarNames: []string{"s7"}}, {envFromNames: []string{"s70"}}, }, } // Check the no-op update scenario manager.RegisterPod(podWithSecrets("ns1", "noop-pod", s5)) manager.RegisterPod(podWithSecrets("ns1", "noop-pod", s5)) </ins> // Now we have: 3 pods with s1, 2 pods with s2 and 2 pods with s3, 0 pods with s4. <del> verify := func(ns, name string, count int) bool { </del> <ins> refs := func(ns, name string) int { </ins> store.lock.Lock() defer store.lock.Unlock() item, ok := store.items[objectKey{ns, name}] if !ok { <del> return count == 0 </del> <ins> return 0 </ins> } <del> return item.refCount == count } assert.True(t, verify("ns1", "s1", 3)) assert.True(t, verify("ns1", "s10", 1)) assert.True(t, verify("ns1", "s2", 3)) assert.True(t, verify("ns1", "s3", 3)) assert.True(t, verify("ns1", "s30", 2)) assert.True(t, verify("ns1", "s4", 2)) assert.True(t, verify("ns1", "s5", 4)) assert.True(t, verify("ns1", "s50", 2)) assert.True(t, verify("ns1", "s6", 0)) assert.True(t, verify("ns1", "s60", 0)) assert.True(t, verify("ns1", "s7", 0)) </del> <ins> return item.refCount } assert.Equal(t, 3, refs("ns1", "s1")) assert.Equal(t, 1, refs("ns1", "s10")) assert.Equal(t, 3, refs("ns1", "s2")) assert.Equal(t, 3, refs("ns1", "s3")) assert.Equal(t, 2, refs("ns1", "s30")) assert.Equal(t, 2, refs("ns1", "s4")) assert.Equal(t, 4, refs("ns1", "s5")) assert.Equal(t, 2, refs("ns1", "s50")) assert.Equal(t, 0, refs("ns1", "s6")) assert.Equal(t, 0, refs("ns1", "s60")) assert.Equal(t, 1, refs("ns1", "s7")) assert.Equal(t, 1, refs("ns1", "s70")) </ins> } func TestCachingSecretManager(t *testing.T) {