Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? static manifest Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Bare metal and reproduced in Vagrant OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: custom scratch install and reproduced with kubeadm Others: Calico for networking. Used (Also able to reproduce without calico) What happened: Modified one of the static manifest files in . The kubelet killed the running pod and did NOT create a new pod. (waited 30 minutes, still no pod) What you expected to happen: Kubelet should have killed the pod and started a new one. How to reproduce it (as minimally and precisely as possible): Started with single node, vanilla Ubuntu Xenial, installed kubernetes via kubeadm, nothing special. Modify , for example change the logging level or even just whitespace at the end of the file. Observe the kubelet kill the controller manager pod, and does not restart it. Anything else we need to know: If I remove the file from the directory, and then replace the file, kubelet will start it up. Kubelet logs from the period where I updated the file:
Did you modify the yaml file using an editor (e.g., vim)? If the editor creates a temporary file in the same directory, kubelet can get confused and thinking that's a new pod (with the same name & namespace). When the temporary file gets removed, kubelet'd inadvertently delete the pod from its cache. Since kubelet relies on inotify for changes, if it doesn't get notified for any new changes from the original yaml file, it'd forever ignore the pod. There is an issue open to add periodic listing in addition to inotify (introduced in 1.5). This would make sure kubelet eventually recover from the situation. I can't find the issue, but there is another one for inotify: . Another fix is to make sure kubelet ignore such known temporary files (it already ignores files starting with ). In general, don't edit the file directly. Copy the file to another place, edit, and copy it back to the manifests directory.
I'm seeing a similar problem except I never edited the file, might be fit for a separate issue
Yes I was using vim, and this explanation makes sense. Would be nice if it was smart about it. Would a PR to ignore the vim temp file be acceptable
I think it's reasonable. I doubt if anyone uses those file names on purpose.
The dot ignore should have covered the vim temp file too, but it doesn't work because the filename passed in is the absolute filename including the path. So the first character is not a dot. I'll try to submit a PR fix.
PR submitted
fixed?
How to work around? I coudn't start api server when using kubeadm to deploy k8s. Thanks!

Discussion:
import ( "fmt" "os" <ins> "path/filepath" </ins> "strings" "github.com/golang/glog"

func (s *sourceFile) processEvent(e *inotify.Event) error { // Ignore file start with dots <del> if strings.HasPrefix(e.Name, ".") { </del> <ins> if strings.HasPrefix(filepath.Base(e.Name), ".") { </ins> glog.V(4).Infof("Ignored pod manifest: %s, because it starts with dots", e.Name) return nil }