Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Prometheus graphs with metrics coming from cAdvisor are full of holes. Graphs with metrics coming from Prometheus node exporter are fine. ! What you expected to happen: cAdvisor should return all metrics, all the time. How to reproduce it (as minimally and precisely as possible): Here the diff between the time it returned 81 containermemoryusage metrics and 14: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.3 Cloud provider or hardware configuration**: Azure OS (e.g. from /etc/os-release): 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-master--0 4.4.0-89-generic -Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
could you please take a look? cc
Related:
We run a kubespray installation, currently at k8s 1.7.2, and we noticed the exact behavior from cAdvisor metrics. Kubespray runs kubelet in a container on each host, exposing port 4194. We found that direct repeated queries to this port will return an inconsistent number of metrics. Sometimes, a metric will be present. Sometimes, not. It appears to be completely random. This causes prometheus to miss scrape values which, in turn, causes grafana do plot graphs with holes in them. ! For reference, I myself do run another unrelated instance of cAdvisor-v0.26.1 in a different Docker cluster, and it seems to work fine. This seems related to the metrics exposed by Kubernetes.
cc
When should we expect a release containing this fix?
8 Will definitely have it. It will land in 1.7.5 or 1.7.6, depending on how quickly the cherrypick makes it in. IIRC, 1.7.5 should be cut any day now.
Thank you for the quick reply . Would be great to see it in 1.7.5, monitoring is completely broken for us due to this bug
I didn't find anywhere the cherry-pick of this issue. Is it in progress? If not, what can we do to move it forward?

Discussion:
} func containerLabels(c *cadvisorapi.ContainerInfo) map[string]string { <del> set := map[string]string{metrics.LabelID: c.Name} </del> <ins> // Prometheus requires that all metrics in the same family have the same labels, // so we arrange to supply blank strings for missing labels var name, image, podName, namespace, containerName string </ins> if len(c.Aliases) > 0 { <del> set[metrics.LabelName] = c.Aliases[0] } if image := c.Spec.Image; len(image) > 0 { set[metrics.LabelImage] = image </del> <ins> name = c.Aliases[0] </ins> } <ins> image = c.Spec.Image </ins> if v, ok := c.Spec.Labels[types.KubernetesPodNameLabel]; ok { <del> set["pod_name"] = v </del> <ins> podName = v </ins> } if v, ok := c.Spec.Labels[types.KubernetesPodNamespaceLabel]; ok { <del> set["namespace"] = v </del> <ins> namespace = v </ins> } if v, ok := c.Spec.Labels[types.KubernetesContainerNameLabel]; ok { <del> set["container_name"] = v </del> <ins> containerName = v } set := map[string]string{ metrics.LabelID:    c.Name, metrics.LabelName:  name, metrics.LabelImage: image, "pod_name":         podName, "namespace":        namespace, "container_name":   containerName, </ins> } return set }

// containerPrometheusLabels maps cAdvisor labels to prometheus labels. func containerPrometheusLabels(c *cadvisorapi.ContainerInfo) map[string]string { <del> set := map[string]string{metrics.LabelID: c.Name} </del> <ins> // Prometheus requires that all metrics in the same family have the same labels, // so we arrange to supply blank strings for missing labels var name, image, podName, namespace, containerName string </ins> if len(c.Aliases) > 0 { <del> set[metrics.LabelName] = c.Aliases[0] } if image := c.Spec.Image; len(image) > 0 { set[metrics.LabelImage] = image </del> <ins> name = c.Aliases[0] </ins> } <ins> image = c.Spec.Image </ins> if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNameLabel]; ok { <del> set["pod_name"] = v </del> <ins> podName = v </ins> } if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNamespaceLabel]; ok { <del> set["namespace"] = v </del> <ins> namespace = v </ins> } if v, ok := c.Spec.Labels[kubelettypes.KubernetesContainerNameLabel]; ok { <del> set["container_name"] = v </del> <ins> containerName = v } set := map[string]string{ metrics.LabelID:    c.Name, metrics.LabelName:  name, metrics.LabelImage: image, "pod_name":         podName, "namespace":        namespace, "container_name":   containerName, </ins> } return set }