Change:
go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.
adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:

Discussion:
import ( "crypto/x509" "errors" <ins> "fmt" </ins> "net/http" "reflect" "strings" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/klog/v2" )

// it does not have to be reimplemented. type counterRaiser struct { counter *metrics.Counter <del> reason  string </del> <ins> // programmatic id used in log and audit annotations prefixes id string // human readable explanation reason string </ins> } func (c *counterRaiser) IncreaseMetricsCounter(req *http.Request) { if req != nil && req.URL != nil { if hostname := req.URL.Hostname(); len(hostname) > 0 { <del> klog.Infof("invalid certificate detected while connecting to %q: %s", req.URL.Hostname(), c.reason) </del> <ins> prefix := fmt.Sprintf("%s.invalid-cert.kubernetes.io", c.id) klog.Infof("%s: invalid certificate detected connecting to %q: %s", prefix, hostname, c.reason) audit.AddAuditAnnotation(req.Context(), prefix+"/"+hostname, c.reason) </ins> } } c.counter.Inc()

return &missingSANChecker{ counterRaiser: counterRaiser{ counter: counter, <ins> id:      "missing-san", </ins> reason:  "relies on a legacy Common Name field instead of the SAN extension for subject validation", }, }

return &sha1SignatureChecker{ counterRaiser: &counterRaiser{ counter: counter, <ins> id:      "insecure-sha1", </ins> reason:  "uses an insecure SHA-1 signature", }, }

"testing" "github.com/stretchr/testify/require" <ins> auditapi "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/testutil" )

if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> _, err = client.Transport.RoundTrip(req) if sanChecker.CheckRoundTripError(err) { sanChecker.IncreaseMetricsCounter(req) <ins> if len(auditCtx.Event.Annotations["missing-san.invalid-cert.kubernetes.io/"+req.URL.Hostname()]) == 0 { t.Errorf("expected audit annotations, got %#v", auditCtx.Event.Annotations) } </ins> } errorCounterVal := getSingleCounterValueFromRegistry(t, registry, "Test_checkForHostnameError")

if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> // can't use tlsServer.Client() as it contains the server certificate // in tls.Config.Certificates. The signatures are, however, only checked

if sha1checker.CheckRoundTripError(err) { sha1checker.IncreaseMetricsCounter(req) <ins> if len(auditCtx.Event.Annotations["insecure-sha1.invalid-cert.kubernetes.io/"+req.URL.Hostname()]) == 0 { t.Errorf("expected audit annotations, got %#v", auditCtx.Event.Annotations) } </ins> } errorCounterVal := getSingleCounterValueFromRegistry(t, registry, "Test_checkForInsecureAlgorithmError")

}{ { name:     "no error, resp w/ cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseWithCert(), }, { name:     "no error, resp w/o cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseNoCert(), }, { name:            "no error, resp w/ cert, counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:            httpResponseWithCert(), counterIncrease: true, }, { name:     "unrelated error, no resp, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:      fmt.Errorf("error"), }, { name:            "related error, no resp,  counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:             fmt.Errorf("error"), counterIncrease: true, },