Change:
Reported in Scenario (CentOS 7): install 1.7.x packages for kubelet on nodes. install cluster with kubeadm init stable-1.7 use kubeadm to upgrade control plane to 1.8.0-rc.1 on nodes which still run 1.7.x kubelet, install new rpms for where certificate-rotation is enabled. systemctl daemon-reload, systemctl restart kubelet. result: kubelet is in wait loop, waiting for csr to be approved. node is in NotReady state. The kubelet blocks on startup, despite having a valid kubeconfig file. It should start up, attempt to rotate in the background, but not block.
I think this is because of the bootstrap kubeconfig flag. If you only provide the cert rotation flag it should only bootstrap in the background. That's what we where doing for bootkube where we need to start static pods while we're doing our initial cert rotation
a bit of debug shown that returns on the last line . So, kubelet has valid cert, but still wants to rotate it. That in theory should be a valid scenario. Question is: why it's not automatically approved then ? Even if cert will be in proper place, does that mean that automatic rotation wouldn't work for expired certs ? Or I'm missing something here.
no, it's because when a file is provided containing only cert/key data, the cert manager blocks on Start() until a CSR is approved
proposed fix for unblocking Start() in , which runs rotation in the background if you have a starting certificate
/cc
what does set up for auto-approval of node self certificate requests during an upgrade to 1.8?
for cert manager hang for kubeadm auto-approval setup issues upgrading to 1.8
looks like kubeadm upgrade instructions indicate approval of CSRs for upgrades nodes is expected: http://kubernetes-io-vnext- I'd expect that only to be required once certs expired, not automatically required as part of upgrade
Yeah, I that step to the kubeadm upgrade instructions last-minute today when the problem was discovered and we didn't think we would have a solution in time for 1.8.0. Manually approving the CSRs seems to work fine, but is a less-than-stellar upgrade experience.
lowering priority a notch, since the upgrade procedure is at least documented, and does not default to true for all installations (specific deployment methods like kubeadm enable it)
In case it helps, kubeadm 1.8 code for starting the controller manager is , and the RBAC rules for auto-approval are .
Also, I'm not sure if we have any other docs that need to be updated regarding upgrading kubelet, since this isn't a kubeadm-specific problem.
the combination of enabling rotation and not setting up auto-approval is what bit kubeadm here is a deployment-specific choice, and is not true by default. is the only deployment I see that enables it on upgrade to 1.8 the auto-approval rules at do not enable nodes to rotate their client certificates, that only sets up auto-approval for the initial TLS bootstrapping token I agree the kubelet should behave better in this scenario if it already has a valid credential (like ), but the configured setup by kubeadm doesn't seem correct
Ah, understood. It wasn't until I was looking at the RBAC rules myself just now that I realized we must have missed that. Definitely needs to get fixed in kubeadm.
I've created a PR for the documentation fix for kubeadm upgrades: We still need a kubeadm code fix to add the RBAC rules, hopefully to get into 1.8.1.
Yeah, not creating the ClusterRoleBinding between and was an oversight, but isn't that critical in any way. We'll make that change in v1.8.1, and document things meanwhile. Upgrading your kubelets is a manual process anyway right now, so it's reasonable to add one extra step to create this binding. Is it expected that a kubelet that already has credentials should renew them on restart if is set?
Not in a blocking way, only in the background if needed because of expiration, see
Reopening. Blocking on start is still an issue if auto approval is not enabled.
Fix in
Should this have the v1.8 milestone attached to it?
is everything here now fixed or are we waiting for
ping on the status of
- needs rebase and the mentioned in there is merged. What else needs to be done?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I think this issue is now solved right?

Discussion:
return err } <ins> // Create/update RBAC rules that makes the nodes to rotate certificates and get their CSRs approved automatically if err := nodebootstraptokenphase.AutoApproveNodeCertificateRotation(client, k8sVersion); err != nil { return err } </ins> // Create the cluster-info ConfigMap with the associated RBAC rules if err := clusterinfophase.CreateBootstrapConfigMapIfNotExists(client, adminKubeConfigPath); err != nil { return err

// TODO: This value should be defined in an other, generic authz package instead of here // Starting from v1.8, CSRAutoApprovalClusterRoleName is automatically created by the API server on startup CSRAutoApprovalClusterRoleName = "system:certificates.k8s.io:certificatesigningrequests:nodeclient" <ins> // NodeSelfCSRAutoApprovalClusterRoleName is a role defined in default 1.8 RBAC policies for automatic CSR approvals for automatically rotated node certificates NodeSelfCSRAutoApprovalClusterRoleName = "system:certificates.k8s.io:certificatesigningrequests:selfnodeclient" </ins> // NodeAutoApproveBootstrapClusterRoleBinding defines the name of the ClusterRoleBinding that makes the csrapprover approve node CSRs NodeAutoApproveBootstrapClusterRoleBinding = "kubeadm:node-autoapprove-bootstrap" <ins> // NodeAutoApproveCertificateRotationClusterRoleBinding defines name of the ClusterRoleBinding that makes the csrapprover approve node auto rotated CSRs NodeAutoApproveCertificateRotationClusterRoleBinding = "kubeadm:node-autoapprove-certificate-rotation" </ins> ) // AllowBootstrapTokensToPostCSRs creates RBAC rules in a way the makes Node Bootstrap Tokens able to post CSRs

}, }) } <ins> // AutoApproveNodeCertificateRotation creates RBAC rules in a way that makes Node certificate rotation CSR auto-approved by the csrapprover controller func AutoApproveNodeCertificateRotation(client clientset.Interface, k8sVersion *version.Version) error { // Create autorotation cluster role binding only if we deploying or upgrading to version that supports it. if k8sVersion.AtLeast(constants.MinimumCSRAutoApprovalClusterRolesVersion) { fmt.Println("[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster") return apiclient.CreateOrUpdateClusterRoleBinding(client, &rbac.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: NodeAutoApproveCertificateRotationClusterRoleBinding, }, RoleRef: rbac.RoleRef{ APIGroup: rbac.GroupName, Kind:     "ClusterRole", Name:     NodeSelfCSRAutoApprovalClusterRoleName, }, Subjects: []rbac.Subject{ { Kind: "Group", Name: constants.NodesGroup, }, }, }) } return nil } </ins>

errs = append(errs, err) } <ins> // Create/update RBAC rules that makes the 1.8.0+ nodes to rotate certificates and get their CSRs approved automatically if err := nodebootstraptoken.AutoApproveNodeCertificateRotation(client, k8sVersion); err != nil { errs = append(errs, err) } </ins> // TODO: Is this needed to do here? I think that updating cluster info should probably be separate from a normal upgrade // Create the cluster-info ConfigMap with the associated RBAC rules // if err := clusterinfo.CreateBootstrapConfigMapIfNotExists(client, kubeadmconstants.GetAdminKubeConfigPath()); err != nil {