Code change:
This is a Feature Request Port this: to the migration guide documentation: as a known issue. Note, CRI-O behavior may be different.
/sig node
/language en
Is the ask here to port the Google doc into the Kubernetes docs or just link to the Google documentation (which isn't necessarily allowed)?
We can't really do either, unless Google licenses to the CNCF. We could write new docs to highlight the same known issue (what's the issue? - I deliberated haven't looked).
Here is the GH issue: These metrics are missing in endpoint with Containerd (they were present with Dockershim): Also the format of container name dimension of many metrics was with dockershim, but the format on the containerd node is . This is coming from this file: that even has an internesting note that this is implementaiton details, but changing it will likely break somethings (btw, maybe this may be noted as well in the list of known issues). So workaround for both issues (missing metrics and the name of metric dimension) is to use a standalone cAdvisor, and use one of the releases with the suffix "-containerd-cri"
That sounds quite involved - should we open a related issue to track explaining how folks would do that?
Do we have a document that can be followed for using a standalone cadvisor? I checked the tags in cadvisor and turns out 2 versions have containerd as suffix but none have the "-containerd-cri" suffix. Ex: v0.41.0-containerd-fix-rc.2 Does the latest version v0.44.0 contain the fix?
can you please help here?
The fix was in I think but not sure which releases contain it
So what do we need to do next, Align versions?
/assign
Yeah I'm not sure how to tell which version has the changes. It's for sure in 0.43.x but the current "Latest" tag is 0.39.x and doesn't seem to have the fix. I think we should wait for or to verify.
It looks like updates the page's docker-specific information related to cAdvisor to be more generic to container runtimes. Would that be enough to address this issue or does more runtime-specific information need to be ?
what's your opinion in respect of ?
It's due to it being reverted in which is a real bummer for those of us using GKE and the default for new node pools is containerd.
/triage accepted
What should the next steps on this issue look like?
my thoughts: This doesn't belong on the suggested page because it's specific to containerd. I'd say let's update as follows: In Before you begin, add a line to "Read the known issues section" Add a section at the end, before What's Next, called "Known issues after migrating" Add "Versions of containerd prior to 0.45.0 didn't collect some metrics in the endpoint. For a list of missing metrics, refer to . To collect these metrics, install a containerd package that is version 0.45.0 or later that has the suffix." In , mention "To avoid the missing metrics in [Known issues after migrating](#known-issues-after-migrating), install a containerd package that is version 0.45.0 or later that has the suffix."
According to the comment at the only current work-around for containerd users is to deploy and scrape a standalone cadvisor daemonset, built from the branch. The stats exposed by the kubelet will be broken until 1.25 or 1.26 when is complete. Since it is the kubelet's cadvisor that is broken (and I would argue that it is broken/regressed by this issue), it should be covered in the kubernetes docs.
Is there a more involved process then than just installing the package?
it's not a package, it's a branch of the cadvisor project: I'm not sure if they're publishing binaries or images from that branch. A sample DaemonSet is available, which could be modified to use :
Could we say: "To collect these metrics, . Set the $VERSION variable to any cAdvisor version that has the suffix, such as " ?
I think that needs more work, for example the default DaemonSet only mounts the docker socket path; it does not mount the containerd socket, and even when I modify it to do so the metrics appear to be missing most of the important labels. Perhaps has some suggestions.
This feels like it would need to be its own task page, possibly linked from our existing Metrics doc as well
I've put a ready to run temporary solution here
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

Related discussion:
text: Kubernetes. They needed to be pushed to some registry to allow them to be used by Kubernetes. <ins> ## Known issues ### Some filesystem metrics are missing and the metrics format is different The Kubelet `/metrics/cadvisor` endpoint provides Prometheus metrics, as documented in [Metrics for Kubernetes system components](/docs/concepts/cluster-administration/system-metrics/). If you install a metrics collector that depends on that endpoint, you might see the following issues: - The metrics format on the Docker node is `k8s_<container-name>_<pod-name>_<namespace>_<pod-uid>_<restart-count>` but the format on other runtime is different. For example, on containerd node it is `<container-id>`. - Some filesystem metrics are missing, as follows: ``` container_fs_inodes_free container_fs_inodes_total container_fs_io_current container_fs_io_time_seconds_total container_fs_io_time_weighted_seconds_total container_fs_limit_bytes container_fs_read_seconds_total container_fs_reads_merged_total container_fs_sector_reads_total container_fs_sector_writes_total container_fs_usage_bytes container_fs_write_seconds_total container_fs_writes_merged_total ``` #### Workaround You can mitigate this issue by using [cAdvisor](https://github.com/google/cadvisor) as a standalone daemonset. 1. Find the latest [cAdvisor release](https://github.com/google/cadvisor/releases) with the name pattern `vX.Y.Z-containerd-cri` (for example, `v0.42.0-containerd-cri`). 2. Follow the steps in [cAdvisor Kubernetes Daemonset](https://github.com/google/cadvisor/tree/master/deploy/kubernetes) to create the daemonset. 3. Point the installed metrics collector to use the cAdvisor `/metrics` endpoint which provides the full set of [Prometheus container metrics](https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md). Alternatives: - Use alternative third party metrics collection solution. - Collect metrics from the Kubelet summary API that is served at `/stats/summary`. </ins> ## {{% heading "whatsnext" %}} - Read [Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) to understand your next steps