Change:
TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet. The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks! Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command. This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!
Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.
because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:

Discussion:
} else if !volMounted || cache.IsRemountRequiredError(err) { // Volume is not mounted, or is already mounted, but requires remounting remountingLogStr := "" <del> if cache.IsRemountRequiredError(err) { </del> <ins> isRemount := cache.IsRemountRequiredError(err) if isRemount { </ins> remountingLogStr = "Volume is already mounted to pod, but remount was requested." } glog.V(12).Infof(volumeToMount.GenerateMsgDetailed("Starting operationExecutor.MountVolume", remountingLogStr)) err := rc.operationExecutor.MountVolume( rc.waitForAttachTimeout, volumeToMount.VolumeToMount, <del> rc.actualStateOfWorld) </del> <ins> rc.actualStateOfWorld, isRemount) </ins> if err != nil && !nestedpendingoperations.IsAlreadyExists(err) && !exponentialbackoff.IsExponentialBackoff(err) {

// * Mount the volume to the pod specific path. // * Update actual state of world to reflect volume is mounted to the pod //   path. <del> MountVolume(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorld ActualStateOfWorldMounterUpdater) error </del> <ins> // The parameter "isRemount" is informational and used to adjust logging // verbosity. An initial mount is more log-worthy than a remount, for // example. MountVolume(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorld ActualStateOfWorldMounterUpdater, isRemount bool) error </ins> // UnmountVolume unmounts the volume from the pod specified in // volumeToUnmount and updates the actual state of the world to reflect that.

func (oe *operationExecutor) MountVolume( waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, <del> actualStateOfWorld ActualStateOfWorldMounterUpdater) error { </del> <ins> actualStateOfWorld ActualStateOfWorldMounterUpdater, isRemount bool) error { </ins> mountFunc, err := oe.operationGenerator.GenerateMountVolumeFunc( <del> waitForAttachTimeout, volumeToMount, actualStateOfWorld) </del> <ins> waitForAttachTimeout, volumeToMount, actualStateOfWorld, isRemount) </ins> if err != nil { return err }

PluginIsAttachable: false, // this field determines whether the plugin is attachable ReportedInUse:      true, } <del> oe.MountVolume(0 /* waitForAttachTimeOut */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */) </del> <ins> oe.MountVolume(0 /* waitForAttachTimeOut */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */, false /* isRemount */) </ins> } // Assert

PluginIsAttachable: true, // this field determines whether the plugin is attachable ReportedInUse:      true, } <del> oe.MountVolume(0 /* waitForAttachTimeout */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */) </del> <ins> oe.MountVolume(0 /* waitForAttachTimeout */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */, false /* isRemount */) </ins> } // Assert

} } <del> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater) (func() error, error) { </del> <ins> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (func() error, error) { </ins> return func() error { startOperationAndBlock(fopg.ch, fopg.quit) return nil

// OperationGenerator interface that extracts out the functions from operation_executor to make it dependency injectable type OperationGenerator interface { // Generates the MountVolume function needed to perform the mount of a volume plugin <del> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater) (func() error, error) </del> <ins> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (func() error, error) </ins> // Generates the UnmountVolume function needed to perform the unmount of a volume plugin GenerateUnmountVolumeFunc(volumeToUnmount MountedVolume, actualStateOfWorld ActualStateOfWorldMounterUpdater) (func() error, error)

func (og *operationGenerator) GenerateMountVolumeFunc( waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, <del> actualStateOfWorld ActualStateOfWorldMounterUpdater) (func() error, error) { </del> <ins> actualStateOfWorld ActualStateOfWorldMounterUpdater, isRemount bool) (func() error, error) { </ins> // Get mounter plugin volumePlugin, err := og.volumePluginMgr.FindPluginBySpec(volumeToMount.VolumeSpec)

return detailedErr } <del> glog.Infof(volumeToMount.GenerateMsgDetailed("MountVolume.SetUp succeeded", "")) </del> <ins> verbosity := glog.Level(1) if isRemount { verbosity = glog.Level(7) } glog.V(verbosity).Infof(volumeToMount.GenerateMsgDetailed("MountVolume.SetUp succeeded", "")) </ins> // Update actual state of world markVolMountedErr := actualStateOfWorld.MarkVolumeAsMounted(