Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image. This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm

Discussion:
// WriteStaticPodManifests builds manifest objects based on user provided configuration and then dumps it to disk // where kubelet will pick and schedule them. func WriteStaticPodManifests(cfg *kubeadmapi.MasterConfiguration) error { <ins> volumes := []api.Volume{k8sVolume(cfg)} volumeMounts := []api.VolumeMount{k8sVolumeMount()} if isCertsVolumeMountNeeded() { volumes = append(volumes, certsVolume(cfg)) volumeMounts = append(volumeMounts, certsVolumeMount()) } if isPkiVolumeMountNeeded() { volumes = append(volumes, pkiVolume(cfg)) volumeMounts = append(volumeMounts, pkiVolumeMount()) } </ins> // Prepare static pod specs staticPodSpecs := map[string]api.Pod{ kubeAPIServer: componentPod(api.Container{ Name:          kubeAPIServer, Image:         images.GetCoreImage(images.KubeAPIServerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getAPIServerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager, Image:         images.GetCoreImage(images.KubeControllerManagerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getControllerManagerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler, Image:         images.GetCoreImage(images.KubeSchedulerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage),

} } <ins> func isCertsVolumeMountNeeded() bool { // Always return true for now. We may add conditional logic here for images which do not require host mounting /etc/ssl // hyperkube for example already has valid ca-certificates installed return true } </ins> // certsVolume exposes host SSL certificates to pod containers. func certsVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{

} } <del> func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </del> <ins> func isPkiVolumeMountNeeded() bool { // On some systems were we host-mount /etc/ssl/certs, it is also required to mount /etc/pki. This is needed // due to symlinks pointing from files in /etc/ssl/certs into /etc/pki/ if _, err := os.Stat("/etc/pki"); err == nil { return true } return false } func pkiVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </ins> return api.Volume{ Name: "pki", VolumeSource: api.VolumeSource{ <ins> // TODO(phase1+) make path configurable HostPath: &api.HostPathVolumeSource{Path: "/etc/pki"}, }, } } func pkiVolumeMount() api.VolumeMount { return api.VolumeMount{ Name:      "pki", MountPath: "/etc/pki", } } func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{ Name: "k8s", VolumeSource: api.VolumeSource{ </ins> HostPath: &api.HostPathVolumeSource{Path: kubeadmapi.GlobalEnvParams.KubernetesDir}, }, }

func k8sVolumeMount() api.VolumeMount { return api.VolumeMount{ <del> Name:      "pki", </del> <ins> Name:      "k8s", </ins> MountPath: "/etc/kubernetes/", ReadOnly:  true, }

{ cfg: &kubeadmapi.MasterConfiguration{}, expected: api.Volume{ <del> Name: "pki", </del> <ins> Name: "k8s", </ins> VolumeSource: api.VolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: kubeadmapi.GlobalEnvParams.KubernetesDir},

}{ { expected: api.VolumeMount{ <del> Name:      "pki", </del> <ins> Name:      "k8s", </ins> MountPath: "/etc/kubernetes/", ReadOnly:  true, },