Change:
Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.
Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion. This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.

Discussion:
"k8s.io/kubernetes/pkg/kubelet/util/format" ) <del> // ProberResults stores the results of a probe as prometheus metrics. var ProberResults = prometheus.NewGaugeVec( prometheus.GaugeOpts{ </del> <ins> // ProberResults stores the cumulative number of a probe by result as prometheus metrics. var ProberResults = prometheus.NewCounterVec( prometheus.CounterOpts{ </ins> Subsystem: "prober", <del> Name:      "probe_result", Help:      "The result of a liveness or readiness probe for a container.", </del> <ins> Name:      "probe_total", Help:      "Cumulative number of a liveness or readiness probe for a container by result.", </ins> }, []string{"probe_type", <del> "container_name", </del> <ins> "result", </ins> "container", <del> "pod_name", </del> "pod", "namespace", "pod_uid"},

const ( liveness probeType = iota readiness <ins> probeResultSuccessful string = "successful" probeResultFailed     string = "failed" probeResultUnknown    string = "unknown" </ins> ) // For debugging.

onHold bool // proberResultsMetricLabels holds the labels attached to this worker <del> // for the ProberResults metric. proberResultsMetricLabels prometheus.Labels </del> <ins> // for the ProberResults metric by result. proberResultsSuccessfulMetricLabels prometheus.Labels proberResultsFailedMetricLabels     prometheus.Labels proberResultsUnknownMetricLabels    prometheus.Labels </ins> } // Creates and starts a new probe worker.

w.initialValue = results.Success } <del> w.proberResultsMetricLabels = prometheus.Labels{ "probe_type":     w.probeType.String(), "container_name": w.container.Name, "container":      w.container.Name, "pod_name":       w.pod.Name, "pod":            w.pod.Name, "namespace":      w.pod.Namespace, "pod_uid":        string(w.pod.UID), </del> <ins> basicMetricLabels := prometheus.Labels{ "probe_type": w.probeType.String(), "container":  w.container.Name, "pod":        w.pod.Name, "namespace":  w.pod.Namespace, "pod_uid":    string(w.pod.UID), </ins> } <ins> w.proberResultsSuccessfulMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsSuccessfulMetricLabels["result"] = probeResultSuccessful w.proberResultsFailedMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsFailedMetricLabels["result"] = probeResultFailed w.proberResultsUnknownMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsUnknownMetricLabels["result"] = probeResultUnknown </ins> return w }

} w.probeManager.removeWorker(w.pod.UID, w.container.Name, w.probeType) <del> ProberResults.Delete(w.proberResultsMetricLabels) </del> <ins> ProberResults.Delete(w.proberResultsSuccessfulMetricLabels) ProberResults.Delete(w.proberResultsFailedMetricLabels) ProberResults.Delete(w.proberResultsUnknownMetricLabels) </ins> }() probeLoop:

return true } <ins> switch result { case results.Success: ProberResults.With(w.proberResultsSuccessfulMetricLabels).Inc() case results.Failure: ProberResults.With(w.proberResultsFailedMetricLabels).Inc() default: ProberResults.With(w.proberResultsUnknownMetricLabels).Inc() } </ins> if w.lastResult == result { w.resultRun++ } else {

} w.resultsManager.Set(w.containerID, result, w.pod) <del> ProberResults.With(w.proberResultsMetricLabels).Set(result.ToPrometheusType()) </del> if w.probeType == liveness && result == results.Failure { // The container fails a liveness check, it will need to be restarted.

return true } <ins> func deepCopyPrometheusLabels(m prometheus.Labels) prometheus.Labels { ret := make(prometheus.Labels, len(m)) for k, v := range m { ret[k] = v } return ret } </ins>