Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk. I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security). Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?

Related discussion:
text: s.Authorization.RemoteKubeConfigFileOptional = true s.Authorization.AlwaysAllowPaths = []string{"/healthz"} <del> s.SecureServing.ServerCert.CertDirectory = "/var/run/kubernetes" </del> <ins> // Set the PairName but leave certificate directory blank to generate in-memory by default s.SecureServing.ServerCert.CertDirectory = "" </ins> s.SecureServing.ServerCert.PairName = "cloud-controller-manager" s.SecureServing.BindPort = ports.CloudControllerManagerPort