Change:
Follow up from , tracked in Upgrade process for is not sufficiently clear in user-facing documentation: [ ] add details to kubelet upgrade procedure (whether drain is required, whether skip-level kubelet upgrades are supported, etc) Page to Update:
Is this about in-place updates of the kubelet? Do the kubeadm upgrade tests cover that scenario? The kube-up GCE upgrade tests just replace machines running the old kubelet with machines running a newer one, bypassing the in-place upgrade questions here. Unless or until we have testing for in-place upgrades, the conservative answer is that the upgrade process for a kubelet is to provision a new machine with the desired kubelet version.
Yes I don't know.
yes, but our tests are failing due to problems with the upgrade framework in k/k (e.g. ginkgo skipping); possibly some other reasons too. the current tests are pretty much unmaintained and there are plans to replace them with something else next cycle, hopefully. we do recommend draining in our 12 -13 upgrade process. skipping minor versions is claimed as unsupported. (edit: not claimed -claimed)
is the informal guidance I've seen until now. There may be other deployment-specific reasons to destroy and rebuild nodes (node-level improvements that only take effect for new nodes, etc), but from the kubelet's perspective, drain has been sufficient, as far as I know.
/wg lts
/language en
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/kind feature /priority backlog
cc:
/remove-lifecycle stale
There is evidence of users upgrading kubelets between minor versions without draining pods (). If that is required, the kubelet upgrade docs need to be made explicit.
Is it required? Do we have a formal decision on that?
Please, let also us address whether drain is required when upgrading between patch versions, e.g., 1.17.0 to 1.17.1. These upgrades are, arguably, more frequent than upgrades between minor versions, and therefore users have a greater incentive to skip drain.
WRT workload stability, the process is the same for PATCH iterations, so a drain would be required there too. one potential difference with MINOR updates is the CPU checkpoint format of the kubelet is not supposed to change on PATCH iterations.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign Routing to sig-node leads. If we require draining nodes before minor version upgrade or reconfiguration (and that is required, as far as I can tell), that needs to be made explicit
My opinion, a cordon -drain -upgrade -uncordon path is the safest thing to document for all situations. We should be able to do patch level upgrades (z-stream, in x.y.z) without draining but, imho, there is no point in complicating the guidance.
cordon -drain -upgrade kubelet -uncordon is the only path supported by SIG Node today. In the past, there were efforts to do in-place upgrade kubelet, including containerized kubelet by CoreOS team to simplify the upgrading flow, but none of them are officially supported by the community. Let's make the upgrade flow explicit in the doc for now, but open for the enhancement.
Now that it's official, I'll work up a docs PR :slightlysmilingface:
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Opened to update the doc. The cluster-upgrade doc already included this information

Discussion:
Optionally upgrade `kubelet` instances to **{{< skew latestVersion >}}** (or they can be left at **{{< skew prevMinorVersion >}}** or **{{< skew oldestMinorVersion >}}**) <ins> {{< note >}} Before performing a minor version `kubelet` upgrade, [drain](/docs/tasks/administer-cluster/safely-drain-node/) pods from that node. In-place minor version `kubelet` upgrades are not supported. {{</ note >}} </ins> {{< warning >}} Running a cluster with `kubelet` instances that are persistently two minor versions behind `kube-apiserver` is not recommended: