Change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ): 4.9.24-coreos Install tools: Others: What happened: After bringing a cluster up, there are no container metrics in the kubelet at /metrics What you expected to happen: I expect to see the metrics, but there are none How to reproduce it (as minimally and precisely as possible): Anything else we need to know: If I try and access the metrics in cAdvisor directly on port 4194 instead of on port then I see the following type of error for all of the metrics:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
This appears to be a v1.7 issue. If I run 1.6.6 I get these metrics. Seems like something has happened in 1.7. I'm running the 1.7 beta because I need nodePort for cni, so now I'm stuck.
Can you triage the issue since?
cc/
Can you help verifying is this is a new regression introduced in 1.7 cadvisor through node-e2e? Thanks!
I've tried this on both cos and cvm in 1.7 cluster, and no container metrics show up. However, I created a 1.6.4 GKE cluster, and container metrics are there. This is a regression in 1.7. Will figure out what is wrong here.
Here is a corresponding cadvisor issue
Even the user still can get the container metrics / stats from kubelet /summary api (recommended way), but since this is a regression introduced in 1.7 release, and we aren't dropping prometheus support, let's fix this for 1.7. Thanks!
Fixed in Will close this issue after building new cadvisor release and cherrypick into 1.7
Thanks for the quick find everyone! I'll be happy to give this a go on the next release!
I'm still facing this same issue. My kubernetes cluster is deployed with kubeadm on Centos 7. I run a k8S v1.8.1 cluster. When I try to fetch containers metrics this way: http://localhost:8001/api/v1/proxy/nodes/node1:0/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics I find no metrics. kubelet is started with . If I start kubelet with . metrics are available through .

Discussion:
}, { "ImportPath": "github.com/codegangsta/negroni", <del> "Comment": "v0.1.0-62-g8d75e11", </del> <ins> "Comment": "v0.1-62-g8d75e11", </ins> "Rev": "8d75e11374a1928608c906fe745b538483e7aeb2" }, {

}, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/cache/memory", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/client/v2", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/collector", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/common", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/docker", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/libcontainer", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/raw", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/rkt", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/systemd", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/devicemapper", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/http/mux", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/machine", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/raw", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/rkt", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages/static", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cloudinfo", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload/netlink", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/docker", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/oomparser", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysfs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysinfo", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/tail", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/zfs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/certificate-transparency/go",

}, { "ImportPath": "github.com/opencontainers/runc/libcontainer", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/apparmor", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups/fs", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups/systemd", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/configs", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/configs/validate", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/criurpc", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/keys", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/label", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/seccomp", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/selinux", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/stacktrace", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/system", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/user", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/utils", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, {

}, { "ImportPath": "github.com/pelletier/go-buffruneio", <del> "Comment": "v0.1.0", </del> "Rev": "df1e16fde7fc330a0ca68167c23bf7ed6ac31d6d" }, {

}, { "ImportPath": "gopkg.in/gcfg.v1", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/scanner", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/token", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/types", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/inf.v0", <del> "Comment": "v0.9.0", </del> "Rev": "3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4" }, {

prometheus.NewGoCollector(), prometheus.NewProcessCollector(os.Getpid(), ""), ) <del> mux.Handle(prometheusEndpoint, promhttp.HandlerFor(r, promhttp.HandlerOpts{})) </del> <ins> mux.Handle(prometheusEndpoint, promhttp.HandlerFor(r, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError})) </ins> } func staticHandlerNoAuth(w http.ResponseWriter, r *http.Request) {