Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In 1.11, removed externalID from Node.Spec which used to be used by kubelet to detect node recreation in certain environments during node upgrade. This changed node upgrade behavior in those environments because in tryRegisterWithAPIServer(node *v1.Node), without externalID checking, we now consider the recreated node after upgrade as a previously registered node and thus will get old node state from API server instead of regenerating them based on node local state. For extended resource reported by a device plugin, this causes problem because after node upgrade/recreation, the device plugin DaemonSet pod needs to be restarted by Kubelet, finishes certain setup, and then report to kubelet that its resource is available at the node. However, the node status capacity and allocatable from the API server still contains the old state before the node upgrade. After kubelet syncs up with the API server, the node status field gets overwritten and previously reported extended resources will appear on the node status capacity/allocatable even though the node is not ready for pods to consume such resources. As the result, Kubelet may start a pod requesting such resource without proper container runtime setup it needs to learn from device plugin. Note to cope with the externalID removal change, has been updated () to explicitly uncordon the node after checking it is restarted and becomes ready. This however doesn't help device plugin case because node readiness doesn't depend on individual extended resource readiness. This issue is to explore how we can fix this problem on head and 1.11. In particular, I wonder whether we may switch to the model that we consider Kubelet as the only source to update extended resource capacity/allocatable in node status. I.e., don't support manually updating node status capacity/allocatable fields as documented in This would allow us to re-generate node status capacity/allocatable every time in node status update, which I think is much simpler and more robust. As I heard, some folks have been using this mechanism to do simple resource exporting/accounting through a central controller. I wonder whether people are open to switch to the device plugin model in those cases. Even though those extended resources may not require special container setup, it seems a more secure model to have Kubelet totally own its node status. FYI, here is the related OSS issue: where we first introduced extended resource concept. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Running the gpu upgrade test from with --upgrade-target set to any 1.11+ version would hit the issue. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
/cc
Having extended resources be a concept encapsulating both device plugin resources and user set resources (opaque int resources) has been a hassle to deal with (as it introduced unexpected behavior in the device plugin lifecycle). We should also consider a few alternatives : separating them from a code perspective (i.e: different fields in node status) . separating them from a name (/dns) perspective.
Agree with If we do need to support the use case of exporting a resource in node status capacity/allocatable not through Kubelet, seems better to introduce special naming convention for such resources.
We had a discussion in another issue about using the check point to keep the node in not ready until all expected device plugins have re-registered. Is that option still on the table? I’m hesitant to suggest changing the semantics of extended resources again, at least until we are sure it’s a valid long term solution.
We did implement the mechanism to use check point data to determine previously registered device plugin resource and keep the resource not ready till the expected device plugin re-register. It helps normal node reboot case but not for node upgrade/recreation case, because boot disk is recreated in the later case and the check point data is also gone.
Kubelet's device manager should recover gracefully from a deleted checkpoint. It should be able to identify which resources in the node object are extended resources and it should remove them if the checkpoint disappears. Checkpoints can disappear for any number of reasons and kubelet needs to react appropriately. cc since this is fallout from . Device manager was relying on node object deletion/recreation to function after GCE upgrades.
+1 Node API object deletion/recreation was only for the case where you have a completely different instance (and deletion was only done because the external ID was made immutable). Deletion/recreation in response to an upgrade or machine restart is a form of data loss (admin- taints, labels, etc, are unrecoverable). Additionally, depending on Node object deletion on upgrade as previously seen in GCE is highly environment-dependent and will not be consistent between cloud-providers, environments, etc.
how about during node registration reconciliation, we reconcile extended resources by setting their capacity to zero? That means any external controller that exports extended resources by directly patching node status capacity/allocatable needs to re-do that after any kubelet restart, which I guess is already what they are doing to handle node upgrade. The required change should be small:
what are the implications of doing that on every kubelet process restart? does available resource flutter impact already-running workloads on a node (does the kubelet evaluation of PodFitsResources evict running workloads? does any controller notice workloads running on a node are using resources the kubelet says it doesn't have, and evict them?)
For extended resources exported through device plugin, kubelet device manager will handle their restarts, currently based on checkpoint data and in the future, we will explore whether we may store this information in some form of API. If the checkpoint data is gone, pods will fail kubelet admission handling and the pod's controller should recreate a new pod. For extended resources exported by an external controller through direct node status capacity patching, the controller should watch for kubelet readiness change, and repatch the field after kubelet becomes ready again.
Having an external controller watching the nodes status update to repatch the nodes sounds like a bad "user" experience. Maybe looking at it from a user's perspective and then figuring out what we'll do is a better way to look at this? The behavior I'd expect to see is the following: For non device plugin extended resources A user patches the node status with the extended resource If the node restarts I don't expect the Allocatable / Capacity fields to change If the node is upgraded I don't expect the Allocatable / Capacity fields to change If the node is recreated I don't expect the Allocatable / Capacity fields to change For device plugin extended resources Kubelet advertises the extended resources when the device plugin registers itself against kubelet If the node restarts I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is upgraded I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is recreated I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices As for the case where the checkpoint data is gone, should we expect the pod's controller to recreate a new pod? That seems like a surprising behavior are we doing this somewhere else? Does the general behavior outlined here seem sane to the people in this thread?
I think in 1.12 time frame, we should start to scope the work on improving extended resource life cycle. Right now, as we are approaching 1.11 code freeze timeline, I am trying to figure out a solution that is least surprise to any existing workloads. Till 1.11, if there is indeed a real use case where an external component directly patches node status capacity to export a resource, the solution already needs to handle the capacity loss after node upgrade by repatching the field. That is the least surprising solution I can think of now. If folks have concerns on the breaking behavior this may introduce, we should seriously consider reverting part of in 1.11.
+1 for considering reverting for v1.11 timeframe. What would be the side-effect of that ? Ownership of fields in the Node object has been a long standing issue. For example node labels suffered from the same issue. I do agree that kubelet has to be resilient and has to recover from control plane data loss with minimal impact on running applications. Having to expose arbitrary extended resources as device plugin resources is also not ideal. For instance, if one wants to expose inodes or network bandwidth as a resource, device plugins would not be an ideal interface (we can consider fixing this). k8s node upgrades should ideally involve cordoning (or tainting) the node during the duration of the upgrade. During this upgrade phase the node is primarily expected to keep existing pods running (if the upgrades are in place) and the system (scheduler) is not expected to place additional workloads. Since the node object is a shared resource, every component that relies on the node object must ensure eventual consistency. Arbitrary users are not expected to update/patch the node object since it is core to k8s functionality. does node feature discovery continually update the state on the node object to reflect the current state on the node? Until we have a better resource representation format, kubelet will not be able to tag ownership for individual extended resources attached to the node object. I think this limitation is acceptable and the users who introduce additional extended compute resources are expected to be sophisticated enough that they can run long running processes to ensure their additional compute resources are accurately reflected in the node object. As part of Resource Classes, and v2 compute resource APIs, we should consider incorporating a owner reference into concrete compute resource objects which can help with the ownership issues. Node upgrades also need non-trivial automation today to ensure that all node dependencies (runtime, kernel, OS, network, storage plugins, etc.) are logically ready prior to adding a node back to the scheduling pool. Assuming patches to node compute resources are continually updated and nodes are cordoned during the upgrade phase, node upgrade automation can ensure that all the expected extended compute resources on certain nodes are updated to a non-zero capacity prior to adding a node back to the scheduling pool (uncordon) after an upgrade. In addition, additional automation can be employed to auto-taint nodes that do not expose non-zero capacity for expected extended resources. For the near future, kubelet can on-startup can only try to restore state on a best effort basis. If checkpoints are not around, kubelet may override existing patched fields. Device plugins and other extended resource plugins will have to ideally update capacity/allocatable to reflect the state on the local node prior to marking an upgrade as complete.
it would delay removal of the self-deletion permission for yet another release (already pushed to 1.12 because of version-skew), meaning node taints would not be usable for isolating compromised nodes until at least 1.14 (because the associated kubelet could simply delete/re-register their Node object). I also don't think it would actually solve this issue. According to , even were we to revert and restore kubelet self-deletion, changes made in stabilized the computation of external id in GCE such that it no longer changes as part of node upgrades in 1.11+. Undoing that, destabilizing the GCE computed external id, and reintroducing self-deletion cases to solve the stale resource issue in one particular cloud environment doesn't seem like a good approach.
The feature that would be slipping is which is particularly critical for and because it's required to run a cluster with dedicated nodes securely. We would be slipping another release on something that is already painfully slow due to backward compatibility considerations. The PR () that changed the behavior device manager was depending on merged two months ago. It's really unfortunate that this report came in so late. I'm pretty convinced that this is a device manager issue that is only mitigated by clearing the node object. Is there anything I can do to help fix this? I'm happy to argue that changes here should be exempt from the codefreeze deadline.
The node feature discovery project only deals with node labels, and not extended resources. NFD fairly recently the ability to run as a daemon, but also retained the option to publish labels once and exit.
So far, it sounds is perhaps the most realistic solution for 1.11 time frame. Although we want to support good user experience, I do feel we need to balance the complexity we need to add into the system v.s. the use cases we want to optimize for. I feel the ask for the external component to repatch node status capacity after kubelet restart seems reasonable and perhaps is already implemented by any serious use cases.
The change that was backported to 1.10 (and maybe all versions?) for this seems to have had the unexpected consequence of zeroing out all of our extended resources upon kubelet restart and causing a large number of our pods to get evicted in our QA cluster since they all use extended resources. I filed an issue here: if you have any advice on intentions/workarounds. For now, we reverted that commit in our internal build as a workaround. Thanks!
How is your extended resource exported on a node? Are they being exported by a device plugin or do you directly update your node objects? If it is the later case, do you have a central controller that may watch for node updates and re-export your resources?
When the host is first configured, our configuration management system running on the node starts the kubelet service, waits for kubelet to register itself in the API by polling and then patches the node capacity directly. We do not have a central controller to rexport the resources. It also sounds like watching and rexporting is not sufficient, since the value would still hit zero for some brief period of time before the central controller could react, at which point kubelet will have already begun hard evicting all the pods. The only central solution I think would work is doing it from an admission controller that ignores the kubelet's attempt to set the extended resources in our namespace to zero.
Can the configuration management system re-patch node capacity after each Kubelet registration?
The kubelet service can restart at any time, whether the config management system is currently configuring the system or not. Ex: kubelet could get OOM-killed, crash, fail healthchecks, etc. If every kubelet restart is a kubelet registration, then no, unless we made the kubelet service also launch a script every time it restarts. So lets say that we hypothetically did configure a script to re-patch the node capacity every time kubelet restarts: there is still an unsolvable race condition where the node temporarily has its capacity set to zero. Consider the situation: is up and healthy. The value for . kubelet service restarts reconciliation logic sets . kubelet immediately begins hard-evicting every pod that requested a . script patches the node capacity back to , but it's too late, every pod has already been killed non-gracefully by kubelet due to the resource having been reset back to 0 at step 3.
Sigh, this is the deficiency we documented in Unfortunately, I couldn't think of a clean solution without API change. One thing we may consider is to instead of zeroing out extended resource capacity, we delete them. Because we are currently taking an open policy on extended resource admission, pods already assigned on the node wouldn't be affected, and your configuration manager can re-patch node later. Another possible workaround you may consider is to implement device plugin API in your configuration manager. Because Kubelet maintains device plugin checkpoint, it would allow pods to pass admission based on the checkpointed state. cc for thoughts on this issue.
Note for the first mentioned workaround, i.e., instead of zeroing out extended resource capacity, we delete them, I think we will rely on to re-populate device plugin resource runtime states during container restart retries. This is not ideal but we already need this to handle the edge cases that node API object may get deleted but pods are still assigned to node, or pods can be admitted before node re-register with API server.
Unfortunately, I couldn't think of a clean solution without API change. The effect of this change as it stands is that kubernetes users run the risk of having 100% of pods in their cluster completely wiped out with no respect for PDB, terminationGracePeriodSeconds, and preStop hooks just by restarting the kubelet processes. This really feels like a critical bug that has much worse consequences than the bugs the change sought to solve. If this is going to continue being the behavior, there should be a really loud disclaimer on which currently makes it appear that extended resources are trivial to manage. I had a team of 5 people restoring 350 stateful database clusters to a clean state for 3+ hours on Monday night. Restarting kubelets was never a destructive action before, so we weren't restarting them as slowly and carefully as if we were rolling out a config change. If this had happened in our production cluster instead, it could have brought the company down for hours. So you're saying that simply removing the entry from the capacity map instead of zeroing it would allow pods using those resources to continue running? When you re-patch the capacity, does that break the accounting at all? (Ex if 3 pods were consuming an extended resource, and the resource is deleted and re-, does the scheduler realize that 3 of that resource are still consumed by pods, or would it reset to thinking zero pods are using it or would it count them correctly?) I'm willing to do that if it's really necessary. That said, I couldn't find the code that does that during kubelet registration in 1.10, what happens if the device plugin is down during a kubelet restart? Does kubelet then zero out the resource? Does the device plugin need to be running for a container using the resource to successfully start? As a possible suggestion that doesn't require an (http) API change or the cluster admins to run a device plugin daemon, has it been considered to implement similar to ? on kubelet. That way the admin would have a trivial way to explicitly tell kubelet to always register with a static extended resource capacity value, making the reconciliation process much more straightforward.
Paul Furtado <: state. It would see three pods are already assigned to the node and wouldn't assign more. being scheduled on the node till the resource is ready and device plugin re-registers. However, in Kubelet device plugin manager, we use check-pointed state to bypass predicate admission handler so running pods wouldn't be affected. The function that does this is at: Jiaying
Could we add a whitelist configration to skip reconcile specified extended resource? Or skip a specific prefix
That would require changing kubelet flags, which I am hoping to avoid.
but we have no way to distinguish the extended resource is reported by device plugin or patched by users directly. Seems we can't use non device plugin extended resources anymore. b/c we can't guarantee to repatch resources before pods evicted.
It is indeed a question whether we want to support the use case of directly patching node object to export a resource. Capacity/Allocatable are node status fields. By definition, they should be observable from the node. I.e., even when Kubelet is isolated from the API server, it should be able to reconstruct its status fields. I think even without zeroing out extended resource capacity during API registration, there is perhaps already a race between pod admission and API registration. They are performed by separate go routines on Kubelet and there is no guaranteed ordering between these two processes. This wan't a problem in early versions of 1.10 because we changed extended resource admission into open policies. It remains a question whether we want to keep this open admission policies or not. Adding an explicit Kubelet flag to specify what extended resources (and perhaps also their capacities) it should export may be a better way than directly patching node API object, because it would make that information truly observable. Whether we have enough use cases to justify this flag is perhaps a question we should discuss in sig-node.
Could we have a "generic device plugin" / "extended resource device plugin"? e.g: You deploy this device plugin and start interacting with a CRD rather than patching the node object in the spec?
For this case, shall we take the following ways to avoid pod restarts when kubelet restart? (1) we can use the default namespace prefix ,"" to avoid being zero out (2) remove extended resource before kubelet start, and patch add again when kubelet started Both of the two ways can make pod with extended resource not evicted then kubelet restart

Discussion:
// annotation. requiresUpdate := kl.reconcileCMADAnnotationWithExistingNode(node, existingNode) requiresUpdate = kl.updateDefaultLabels(node, existingNode) || requiresUpdate <ins> requiresUpdate = kl.reconcileExtendedResource(node, existingNode) || requiresUpdate </ins> if requiresUpdate { if _, _, err := nodeutil.PatchNodeStatus(kl.kubeClient.CoreV1(), types.NodeName(kl.nodeName), originalNode, existingNode); err != nil { glog.Errorf("Unable to reconcile node %q with API server: error updating node: %v", kl.nodeName, err)

return true } <ins> // Zeros out extended resource capacity during reconciliation. func (kl *Kubelet) reconcileExtendedResource(initialNode, node *v1.Node) bool { requiresUpdate := false for k := range node.Status.Capacity { if v1helper.IsExtendedResourceName(k) { node.Status.Capacity[k] = *resource.NewQuantity(int64(0), resource.DecimalSI) node.Status.Allocatable[k] = *resource.NewQuantity(int64(0), resource.DecimalSI) requiresUpdate = true } } return requiresUpdate } </ins> // updateDefaultLabels will set the default labels on the node func (kl *Kubelet) updateDefaultLabels(initialNode, existingNode *v1.Node) bool { defaultLabels := []string{

} } <ins> func TestReconcileExtendedResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.kubeClient = nil // ensure only the heartbeat client is used extendedResourceName1 := v1.ResourceName("test.com/resource1") extendedResourceName2 := v1.ResourceName("test.com/resource2") cases := []struct { name         string existingNode *v1.Node expectedNode *v1.Node needsUpdate  bool }{ { name: "no update needed without extended resource", existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, }, }, needsUpdate: false, }, { name: "extended resource capacity is zeroed", existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(2), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(10), resource.DecimalSI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(2), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(10), resource.DecimalSI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(0), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(0), resource.DecimalSI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(0), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(0), resource.DecimalSI), }, }, }, needsUpdate: true, }, } for _, tc := range cases { defer testKubelet.Cleanup() kubelet := testKubelet.kubelet initialNode := &v1.Node{} needsUpdate := kubelet.reconcileExtendedResource(initialNode, tc.existingNode) assert.Equal(t, tc.needsUpdate, needsUpdate, tc.name) assert.Equal(t, tc.expectedNode, tc.existingNode, tc.name) } } </ins> func TestValidateNodeIPParam(t *testing.T) { type test struct { nodeIP   string