Change:
docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:d83594ecd85345144584523e7fa5388467edf5d2dfa30d0a1bcbf184cddf4a7b", </del> <ins> digest = "sha256:cc782ed16599000ca4c85d47ec6264753747ae1e77520894dca84b104a7621e2", </ins> registry = "k8s.gcr.io", repository = "debian-hyperkube-base-amd64", <del> tag = "0.9",  # ignored, but kept here for documentation </del> <ins> tag = "0.10",  # ignored, but kept here for documentation </ins> ) docker_pull(

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Cloud providers are using the command-line tool to retrieve status and format volume mounts. is missing from the hyperkube image resulting in missing volumes. Issues: What you expected to happen: Volume mounts on cloud providers are expected to be formatted and mounted. How to reproduce it (as minimally and precisely as possible): GCE and Openstack volume mounts are effected. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9, 1.10+ Cloud provider or hardware configuration: GCE and Openstack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/sig storage

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.

Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me.

thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by .

For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log.

Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks!

Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec.

We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found