Code change:
on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs. The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else. i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.) What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies. I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA. though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:

Related discussion:
text: assert.NoError(t, err) assert.True(t, f.cpuHardcapping, "cpu hardcapping is expected to be enabled") } <ins> func TestGetCapacity(t *testing.T) { ephemeralStorageFromCapacity := int64(2000) ephemeralStorageFromCadvisor := int64(8000) mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCtrlError := gomock.NewController(t) defer mockCtrlError.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) rootfs := cadvisorapiv2.FsInfo{ Capacity: 8000, } mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisorError := cadvisortest.NewMockInterface(mockCtrlError) mockCadvisorError.EXPECT().RootFsInfo().Return(cadvisorapiv2.FsInfo{}, errors.New("Unable to get rootfs data from cAdvisor interface")) cases := []struct { name                                string cm                                  *containerManagerImpl expectedResourceQuantity            *resource.Quantity expectedNoEphemeralStorage          bool enableLocalStorageCapacityIsolation bool }{ { name: "capacity property has ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceEphemeralStorage: *resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), }, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity:          v1.ResourceList{}, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCadvisor, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, error from rootfs", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisorError, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, cadvisor interface is nil", cm: &containerManagerImpl{ cadvisorInterface: nil, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "LocalStorageCapacityIsolation feature flag is disabled", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("4"), v1.ResourceMemory: resource.MustParse("16G"), }, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: false, }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, kubefeatures.LocalStorageCapacityIsolation, c.enableLocalStorageCapacityIsolation)() ret := c.cm.GetCapacity() if v, exists := ret[v1.ResourceEphemeralStorage]; !exists { if !c.expectedNoEphemeralStorage { t.Errorf("did not get any ephemeral storage data") } } else { if v.Value() != c.expectedResourceQuantity.Value() { t.Errorf("got unexpected %s value, expected %d, got %d", v1.ResourceEphemeralStorage, c.expectedResourceQuantity.Value(), v.Value()) } } }) } } </ins>