Repository: prometheus-operator/kube-prometheus

Issue #718: KubeControllerManagerDown & kubeSchedulerDown firing on kubeadm 1.18 cluster
Created: 2020-10-08T19:26:56Z
What happened?
Deploying kube-prometheus release-0.6 to a kubeadm boostrapped bare-metal cluster causes KubeControllerManagerDown and kubeSchedulerDown alerts to fire.
Did you expect to see some different?
Alerts should not fire as everything is up.
How to reproduce it (as minimally and precisely as possible):
Deploy release-0.6 with the bellow config to a kubeadm boostrapped cluster running 1.18.x


Prometheus Operator version:
prometheus-operator:v0.42.1


Manifests:


local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet')
  {
    _config+:: {
      namespace: 'monitoring',
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Anything else we need to know?:
This issue is related to Kubernetes 1.18.x, it appears a few changes were made to the kube-controller-manager and kube-scheduler.
Firstly, version 1.18+ now uses the more secure https port of 10257 and disables http by default. Unfortunately, the --secure-port used by both kube-controller-manager and kube-scheduler is bound to 127.0.0.1 and not 0.0.0.0.
As a result, metrics cannot be collected until the bound address for both of these is updated.
The workaround:
Updating the manifests in /etc/kubernetes/manifests/ to use the --bind-address 0.0.0.0 for both the scheduler and the controller manager will relaunch the the pods with the correct bind address, but these settings will not survive a kubeadm upgrade.
In order to persist the settings, the kubeadm-config configmap in the kube-system namespace should also be edited to include the following:
    controllerManager:
      extraArgs:
        bind-address: 0.0.0.0
    scheduler:
      extraArgs:
        bind-address: 0.0.0.0

I understand this isn't a bug directly related to kube-prometheus, but I didn't find this documented anywhere and had been scratching my head for a day looking at this. Hoping this will help someone else in the future.

Duplicate issue #1006: SD kube-controller-manager + kubescheduler not  being discovered
Created: 2021-03-05T13:32:56Z
Prometheus servicediscovery says:
monitoring/kube-prometheus-stack-kube-controller-manager/0 (0 / 37 active targets)
monitoring/kube-prometheus-stack-kube-scheduler/0 (0 / 37 active targets)
all other targets are discovered just fine.
What happened?
I installed this using community helm chart v13.13.0 - and all other SD scrape jobs work fine.
Did you expect to see something different?
I did expect it to discover these two services.
How to reproduce it (as minimally and precisely as possible):
Setup is kops 1.18.3 installation of k8s 1.18.15
Environment


Prometheus Operator version:
quay.io/prometheus-operator/prometheus-operator:v0.45.0


Kubernetes version information:
1.18.15


Kubernetes cluster kind:


kops 1.18.3
Used community helm chart: kube-prometheus-stack

Prometheus Operator Logs:

nothing relevant. job is setup - it just does not match correctly for these two services it seems.
 apiVersion: v1                                                                                                                                                                                                                                        
 kind: Service                                                                                                                                                                                                                                         
 metadata:                                                                                                                                                                                                                                             
   annotations:                                                                                                                                                                                                                                        
     kubectl.kubernetes.io/last-applied-configuration: |                                                                                                                                                                                               
       {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kube-prometheus-stack-kube-controller-manager","argocd.argoproj.io/instance":"kube-prometheus-stack","chart":"kube-prometheus-stack-13.13.0","heritage":"Helm 
 ","jobLabel":"kube-controller-manager","release":"kube-prometheus-stack"},"name":"kube-prometheus-stack-kube-controller-manager","namespace":"kube-system"},"spec":{"clusterIP":"None","ports":[{"name":"http-metrics","port":10252,"protocol":"TCP", 
 "targetPort":10252}],"selector":{"component":"kube-controller-manager"},"type":"ClusterIP"}}                                                                                                                                                          
   creationTimestamp: "2021-02-26T10:24:28Z"                                                                                                                                                                                                           
   labels:                                                                                                                                                                                                                                             
     app: kube-prometheus-stack-kube-controller-manager                                                                                                                                                                                                
     argocd.argoproj.io/instance: kube-prometheus-stack                                                                                                                                                                                                
     chart: kube-prometheus-stack-13.13.0                                                                                                                                                                                                              
     heritage: Helm                                                                                                                                                                                                                                    
     jobLabel: kube-controller-manager                                                                                                                                                                                                                 
     release: kube-prometheus-stack                                                                                                                                                                                                                    
   name: kube-prometheus-stack-kube-controller-manager                                                                                                                                                                                                 
   namespace: kube-system                                                                                                                                                                                                                              
   resourceVersion: "1106635"                                                                                                                                                                                                                          
   selfLink: /api/v1/namespaces/kube-system/services/kube-prometheus-stack-kube-controller-manager                                                                                                                                                     
   uid: 01b648aa-9695-45e1-9a63-a4006944a02d                                                                                                                                                                                                           
 spec:                                                                                                                                                                                                                                                 
   clusterIP: None                                                                                                                                                                                                                                     
   ports:                                                                                                                                                                                                                                              
   - name: http-metrics                                                                                                                                                                                                                                
     port: 10252                                                                                                                                                                                                                                       
     protocol: TCP                                                                                                                                                                                                                                     
     targetPort: 10252                                                                                                                                                                                                                                 
   selector:                                                                                                                                                                                                                                           
     component: kube-controller-manager                                                                                                                                                                                                                
   sessionAffinity: None                                                                                                                                                                                                                               
   type: ClusterIP                                                                                                                                                                                                                                     
 status:                                                                                                                                                                                                                                               
   loadBalancer: {} 

and kube-scheduler service:
 apiVersion: v1                                                                                                                                                                                                                                        
 kind: Service                                                                                                                                                                                                                                         
 metadata:                                                                                                                                                                                                                                             
   annotations:                                                                                                                                                                                                                                        
     kubectl.kubernetes.io/last-applied-configuration: |                                                                                                                                                                                               
       {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kube-prometheus-stack-kube-scheduler","argocd.argoproj.io/instance":"kube-prometheus-stack","chart":"kube-prometheus-stack-13.13.0","heritage":"Helm","jobLab 
 el":"kube-scheduler","release":"kube-prometheus-stack"},"name":"kube-prometheus-stack-kube-scheduler","namespace":"kube-system"},"spec":{"clusterIP":"None","ports":[{"name":"http-metrics","port":10251,"protocol":"TCP","targetPort":10251}],"selec 
 tor":{"component":"kube-scheduler"},"type":"ClusterIP"}}                                                                                                                                                                                              
   creationTimestamp: "2021-02-26T10:24:28Z"                                                                                                                                                                                                           
   labels:                                                                                                                                                                                                                                             
     app: kube-prometheus-stack-kube-scheduler                                                                                                                                                                                                         
     argocd.argoproj.io/instance: kube-prometheus-stack                                                                                                                                                                                                
     chart: kube-prometheus-stack-13.13.0                                                                                                                                                                                                              
     heritage: Helm                                                                                                                                                                                                                                    
     jobLabel: kube-scheduler                                                                                                                                                                                                                          
     release: kube-prometheus-stack                                                                                                                                                                                                                    
   name: kube-prometheus-stack-kube-scheduler                                                                                                                                                                                                          
   namespace: kube-system                                                                                                                                                                                                                              
   resourceVersion: "1106659"                                                                                                                                                                                                                          
   selfLink: /api/v1/namespaces/kube-system/services/kube-prometheus-stack-kube-scheduler                                                                                                                                                              
   uid: 7dd3e985-d407-4e1f-8e1b-628cadababf4                                                                                                                                                                                                           
 spec:                                                                                                                                                                                                                                                 
   clusterIP: None                                                                                                                                                                                                                                     
   ports:                                                                                                                                                                                                                                              
   - name: http-metrics                                                                                                                                                                                                                                
     port: 10251                                                                                                                                                                                                                                       
     protocol: TCP                                                                                                                                                                                                                                     
     targetPort: 10251                                                                                                                                                                                                                                 
   selector:                                                                                                                                                                                                                                           
     component: kube-scheduler                                                                                                                                                                                                                         
   sessionAffinity: None                                                                                                                                                                                                                               
   type: ClusterIP                                                                                                                                                                                                                                     
 status:                                                                                                                                                                                                                                               
   loadBalancer: {}

Closing comment (2021-03-05T13:54:28Z): Possibly related to #718