Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm testing the new option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new CSR and tries to rotate the certificates when it is started. This is happening even though there's a valid (brand new) certificate on disk issued by kube-controller-manager. From what I can tell, this is happening because kubelet implements and is set after a successful certificate rotation, but is never persisted to disk. If kubelet is restarted, then the last requested CSR is nil. What you expected to happen: Ideally, kubelet should only request the signing of a new CSR if it doesn't already have a serving certificate or it's about to expire. How to reproduce it (as minimally and precisely as possible): Restart a kubelet instance that had it's serving certificate issued by kube-controller-manager Check and see that it requested the signing of a new CSR Anything else we need to know?: kubelet is being run with . , , and are all omitted to use the default values. The TLS bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates. Environment: Kubernetes version: v1.12.0 - built from source at the tag () Cloud provider or hardware configuration: Bare metal OS: CentOS 7.5.1804 Kernel: Install tools: Custom RPMs built from this git repo with no additional patches Others:
/sig node
/sig auth
Not sure if this is the ideal fix, but I've patched it so that it compares the current CSR template to the last requested template only if it doesn't already match the current cert.
cc
thanks for the proposed fix. I think it looks good. Could you send a PR? We should also remove entirely and only look at current cert and template. does that sound reasonable?
Sure thing, I'd be happy to do that. EDIT: I will submit a PR after approval from my employer.
I think that's probably ok, as long as it checks everything we include in the template and we consider a certificate missing anything we request to be cause for re-requesting another cert (an issuer couldn't choose to issue a cert that drops some SANs and us not request another cert) left some comments in
This regression is a frequent flyer. I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:

Discussion:
if m.dynamicTemplate { go wait.Forever(func() { // check if the current template matches what we last requested <del> if !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </del> <ins> if !m.certSatisfiesTemplate() && !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </ins> // if the template is different, queue up an interrupt of the rotation deadline loop. // if we've requested a CSR that matches the new template by the time the interrupt is handled, the interrupt is disregarded. templateChanged <- struct{}{}

return true, nil } <del> // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() </del> <ins> // Check that the current certificate on disk satisfies the requests from the // current template. // // Note that extra items in the certificate's SAN or orgs that don't exist in // the template will not trigger a renewal. // // Requires certAccessLock to be locked. func (m *manager) certSatisfiesTemplateLocked() bool { </ins> if m.cert == nil { <del> return time.Now() </del> <ins> return false </ins> } <del> // Ensure the currently held certificate satisfies the requested subject CN and SANs </del> if template := m.getTemplate(); template != nil { if template.Subject.CommonName != m.cert.Leaf.Subject.CommonName { <del> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s), rotating now", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s)", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return false </ins> } currentDNSNames := sets.NewString(m.cert.Leaf.DNSNames...) desiredDNSNames := sets.NewString(template.DNSNames...) missingDNSNames := desiredDNSNames.Difference(currentDNSNames) if len(missingDNSNames) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested DNS names %v, rotating now", missingDNSNames.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested DNS names %v", missingDNSNames.List()) return false </ins> } currentIPs := sets.NewString()

} missingIPs := desiredIPs.Difference(currentIPs) if len(missingIPs) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested IP addresses %v, rotating now", missingIPs.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested IP addresses %v", missingIPs.List()) return false </ins> } <ins> currentOrgs := sets.NewString(m.cert.Leaf.Subject.Organization...) desiredOrgs := sets.NewString(template.Subject.Organization...) missingOrgs := desiredOrgs.Difference(currentOrgs) if len(missingOrgs) > 0 { glog.V(2).Infof("Current certificate is missing requested orgs %v", missingOrgs.List()) return false } } return true } func (m *manager) certSatisfiesTemplate() bool { m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() return m.certSatisfiesTemplateLocked() } // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() if !m.certSatisfiesTemplateLocked() { return time.Now() </ins> } notAfter := m.cert.Leaf.NotAfter

"crypto/x509" "crypto/x509/pkix" "fmt" <ins> "net" </ins> "strings" "testing" "time"

} } <ins> func TestCertSatisfiesTemplate(t *testing.T) { testCases := []struct { name          string cert          *x509.Certificate template      *x509.CertificateRequest shouldSatisfy bool }{ { name:          "No certificate, no template", cert:          nil, template:      nil, shouldSatisfy: false, }, { name:          "No certificate", cert:          nil, template:      &x509.CertificateRequest{}, shouldSatisfy: false, }, { name: "No template", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, template:      nil, shouldSatisfy: true, }, { name: "Mismatched common name", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name-2", }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, shouldSatisfy: false, }, { name: "Missing orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, shouldSatisfy: false, }, { name: "Extra orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, shouldSatisfy: true, }, { name: "Missing DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, shouldSatisfy: false, }, { name: "Extra DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, shouldSatisfy: true, }, { name: "Missing IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, shouldSatisfy: false, }, { name: "Extra IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, { name: "Matching certificate", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { var tlsCert *tls.Certificate if tc.cert != nil { tlsCert = &tls.Certificate{ Leaf: tc.cert, } } m := manager{ cert:        tlsCert, getTemplate: func() *x509.CertificateRequest { return tc.template }, } result := m.certSatisfiesTemplate() if result != tc.shouldSatisfy { t.Errorf("cert: %+v, template: %+v, certSatisfiesTemplate returned %v, want %v", m.cert, tc.template, result, tc.shouldSatisfy) } }) } } </ins> func TestRotateCertCreateCSRError(t *testing.T) { now := time.Now() m := manager{