Change:
When trying to join a second or sometimes third control plane node, fails with: something that looks like an etcd hangup ("Initial timeout of 40s passed") CrashLoopBackOff for one of the kube-apiserver pods The k8s suggest opening an issue here. Successful join op Here goes... First to init the first control plane node (1 of 3 in an HA cluster). This is successful. Contents of : Apply CNI plugin: Looks fine at this point. shows all Running. Next join 2nd control plane node: This seems to go off without a hitch also. The new node comes up as Ready and all new pods Running. Contents of here: Now for the third, failing node: same exact join command, same exact config. It is really a replica node of the 2nd control plane node, and yet times out. Output: Then going back to initial control node. Output of : Uh-oh. So CrashLoopBackOff shows up and one etcd pod is missing completely. No. Kubernetes version (use ): Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Ubuntu 18.04.5 LTS Kernel (e.g. ): Network plugin and version (if this is a network-related bug): weave-net
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/sig usability
Can you check etcd yaml in node in ? Also check the log on node to know why the etcd pod is not created. A known issue is for ephemeral-storage check, you can remove the ephemeral storage requests in and see if the etcd pod can be created.
Okay, I re-ran this exact setup on a newly provisioned set of nodes and ran into the same error. xx-xx-xx-248: initial control plane node (ok) xx-xx-xx-131: second control plane node (ok) xx-xx-xx-176: third control plane node (fails to join) Also want to add the comment that I am joining the nodes in serial rather than in parallel, since I have in other issues seen parallel joins described as an issue for etcd. With this latest run I ran with and fetched some additional log info as referenced by Contents of on the failing node: Output from the failed on the failing node: is same as original post. I am -ing the output to and guessing I might need to capture stderr too. But that's outside the scope of this issue. And lastly kubelet log on the failing node (from ): As shown above this is littered wth (newlines for readability): Final note: the IP changes in this syslog from .65 because the nodes are rebooted before any init or join operations are ran.
/sig node storage
Going to give a try. In my situation here, each node is 65G disk and 4G RAM with nothing else really going on on them besides the k8s initialization.
It looks like applying as outlined in provided a band-aid and the final control plane node joined successfully. The param is successfully removed from . For other kubeadm users looking to apply this fix, the for is: The key is that the argument is a directory, not the patch file specifically. In my case, the file sat at and the invocation is then
I've updated PR to reflect that it will fix this issue Also, note that was merged yesterday, and will be cherry-picked into future 1.20 and 1.21 releases.
thanks for checking the logs. let's keep a single tracking issue for this problem: /close
Closing this issue. <detailsIn response to : Jun 09 11:49:50 ip-xx-xx-xx-176 kubelet[4641]: I0609 11:49:50. 4641 ] "Failed to admit pod, unexpected error while attempting to recover from admission failure" pod="kube-system/etcd-ip-xx-xx-xx-176" err="preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ]" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
return err != nil && os.IsNotExist(err) } <ins> // GetCapacity returns node capacity data for "cpu", "memory", "ephemeral-storage", and "huge-pages*" // At present this method is only invoked when introspecting ephemeral storage </ins> func (cm *containerManagerImpl) GetCapacity() v1.ResourceList { <ins> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.LocalStorageCapacityIsolation) { // We store allocatable ephemeral-storage in the capacity property once we Start() the container manager if _, ok := cm.capacity[v1.ResourceEphemeralStorage]; !ok { // If we haven't yet stored the capacity for ephemeral-storage, we can try to fetch it directly from cAdvisor, if cm.cadvisorInterface != nil { rootfs, err := cm.cadvisorInterface.RootFsInfo() if err != nil { klog.ErrorS(err, "Unable to get rootfs data from cAdvisor interface") // If the rootfsinfo retrieval from cAdvisor fails for any reason, fallback to returning the capacity property with no ephemeral storage data return cm.capacity } // We don't want to mutate cm.capacity here so we'll manually construct a v1.ResourceList from it, // and add ephemeral-storage capacityWithEphemeralStorage := v1.ResourceList{} for rName, rQuant := range cm.capacity { capacityWithEphemeralStorage[rName] = rQuant } capacityWithEphemeralStorage[v1.ResourceEphemeralStorage] = cadvisor.EphemeralStorageCapacityFromFsInfo(rootfs)[v1.ResourceEphemeralStorage] return capacityWithEphemeralStorage } } } </ins> return cm.capacity }

package cm import ( <ins> "errors" </ins> "io/ioutil" "os" "path" "testing" <ins> gomock "github.com/golang/mock/gomock" cadvisorapiv2 "github.com/google/cadvisor/info/v2" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" kubefeatures "k8s.io/kubernetes/pkg/features" </ins> "github.com/opencontainers/runc/libcontainer/cgroups" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" <ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" cadvisortest "k8s.io/kubernetes/pkg/kubelet/cadvisor/testing" </ins> "k8s.io/mount-utils" )

assert.NoError(t, err) assert.True(t, f.cpuHardcapping, "cpu hardcapping is expected to be enabled") } <ins> func TestGetCapacity(t *testing.T) { ephemeralStorageFromCapacity := int64(2000) ephemeralStorageFromCadvisor := int64(8000) mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCtrlError := gomock.NewController(t) defer mockCtrlError.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) rootfs := cadvisorapiv2.FsInfo{ Capacity: 8000, } mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisorError := cadvisortest.NewMockInterface(mockCtrlError) mockCadvisorError.EXPECT().RootFsInfo().Return(cadvisorapiv2.FsInfo{}, errors.New("Unable to get rootfs data from cAdvisor interface")) cases := []struct { name                                string cm                                  *containerManagerImpl expectedResourceQuantity            *resource.Quantity expectedNoEphemeralStorage          bool enableLocalStorageCapacityIsolation bool }{ { name: "capacity property has ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceEphemeralStorage: *resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), }, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity:          v1.ResourceList{}, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCadvisor, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, error from rootfs", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisorError, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, cadvisor interface is nil", cm: &containerManagerImpl{ cadvisorInterface: nil, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "LocalStorageCapacityIsolation feature flag is disabled", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("4"), v1.ResourceMemory: resource.MustParse("16G"), }, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: false, }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, kubefeatures.LocalStorageCapacityIsolation, c.enableLocalStorageCapacityIsolation)() ret := c.cm.GetCapacity() if v, exists := ret[v1.ResourceEphemeralStorage]; !exists { if !c.expectedNoEphemeralStorage { t.Errorf("did not get any ephemeral storage data") } } else { if v.Value() != c.expectedResourceQuantity.Value() { t.Errorf("got unexpected %s value, expected %d, got %d", v1.ResourceEphemeralStorage, c.expectedResourceQuantity.Value(), v.Value()) } } }) } } </ins>