Change:
Kubernetes. They needed to be pushed to some registry to allow them to be used by Kubernetes. <ins> ## Known issues ### Some filesystem metrics are missing and the metrics format is different The Kubelet `/metrics/cadvisor` endpoint provides Prometheus metrics, as documented in [Metrics for Kubernetes system components](/docs/concepts/cluster-administration/system-metrics/). If you install a metrics collector that depends on that endpoint, you might see the following issues: - The metrics format on the Docker node is `k8s_<container-name>_<pod-name>_<namespace>_<pod-uid>_<restart-count>` but the format on other runtime is different. For example, on containerd node it is `<container-id>`. - Some filesystem metrics are missing, as follows: ``` container_fs_inodes_free container_fs_inodes_total container_fs_io_current container_fs_io_time_seconds_total container_fs_io_time_weighted_seconds_total container_fs_limit_bytes container_fs_read_seconds_total container_fs_reads_merged_total container_fs_sector_reads_total container_fs_sector_writes_total container_fs_usage_bytes container_fs_write_seconds_total container_fs_writes_merged_total ``` #### Workaround You can mitigate this issue by using [cAdvisor](https://github.com/google/cadvisor) as a standalone daemonset. 1. Find the latest [cAdvisor release](https://github.com/google/cadvisor/releases) with the name pattern `vX.Y.Z-containerd-cri` (for example, `v0.42.0-containerd-cri`). 2. Follow the steps in [cAdvisor Kubernetes Daemonset](https://github.com/google/cadvisor/tree/master/deploy/kubernetes) to create the daemonset. 3. Point the installed metrics collector to use the cAdvisor `/metrics` endpoint which provides the full set of [Prometheus container metrics](https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md). Alternatives: - Use alternative third party metrics collection solution. - Collect metrics from the Kubelet summary API that is served at `/stats/summary`. </ins> ## {{% heading "whatsnext" %}} - Read [Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) to understand your next steps

Discussion:
/kind feature /sig node docs for the flag pod-infra-container-image as dockershim is removed (also tracked here: ) how to transition from using this flag to Contained setting when migrating off dockershim Also see
/sig docs
/triage accepted
I'm not quite sure what needs to happen with this issue specifically. Is the umbrella issue going to cover this? What needs to change in relation to this issue?
Can you please describe more?
similar to this: The flag is deprecated. On migration - users needs to stop using this flag and instead configure the container runtime with the proper configuration setting. For containerd it is a
/milestone 1.24
Users should stop using the flag only when using a container runtime that supports image pinning. At the moment there's work being done on the containerd side. Other CRIs only have issues created for implementing image pining:
The current situation is that you end up with two (or more) versions of pod-infra-container: Depending on the actual versions of kubernetes and the container runtime being involved.
The pod-infra-container-image flag is only mentioned in two files currently. The one in reference docs will go away when reference docs are rebuilt for 1.24. The mention in will be removed in PR to be named later. If there are new feature to replace image pining for other runtimes, please create a new issue.
removes the option. So we should close this issue once that PR merges.
The dockershim work is ongoing, but as 1.24 is releasing today, i'm clearing the milestone. /milestone clear
The final instance of --pod-infra-container-image was removed with . /close
Closing this issue. <detailsIn response to : The final instance of --pod-infra-container-image was removed with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen this bug was suggesting to add a note to dockershim migration document to suggest customers what to replace this flag with on migration off dockershim. See comment This document will benefit from this instruction: This is one of the flags, other flags not documented in the migration document and causing confusion (see report: ):
Reopened this issue.

<detailsIn response to : /reopen This is one of the flags, other flags not documented in the migration document and causing confusion (see report: ): Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For cri-dockerd, it is called :-) Eventually it will also be possible to set using configuration files: But I believe the current status (editing flags in systemd unit) is documented properly, on the container runtime page.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Would it be appropriate to link right before starts? We could add a line asking our audience to refer to the linked guide to understand how to get Kubelet working with containerd. If that doesn't work, where else would you recommend placing it?

This is a Feature Request Port this: to the migration guide documentation: as a known issue. Note, CRI-O behavior may be different.
/sig node
/language en
Is the ask here to port the Google doc into the Kubernetes docs or just link to the Google documentation (which isn't necessarily allowed)?
We can't really do either, unless Google licenses to the CNCF. We could write new docs to highlight the same known issue (what's the issue? - I deliberated haven't looked).
Here is the GH issue: These metrics are missing in endpoint with Containerd (they were present with Dockershim): Also the format of container name dimension of many metrics was with dockershim, but the format on the containerd node is . This is coming from this file: that even has an internesting note that this is implementaiton details, but changing it will likely break somethings (btw, maybe this may be noted as well in the list of known issues). So workaround for both issues (missing metrics and the name of metric dimension) is to use a standalone cAdvisor, and use one of the releases with the suffix "-containerd-cri"
That sounds quite involved - should we open a related issue to track explaining how folks would do that?
Do we have a document that can be followed for using a standalone cadvisor? I checked the tags in cadvisor and turns out 2 versions have containerd as suffix but none have the "-containerd-cri" suffix. Ex: v0.41.0-containerd-fix-rc.2 Does the latest version v0.44.0 contain the fix?
can you please help here?
The fix was in I think but not sure which releases contain it
So what do we need to do next, Align versions?
/assign
Yeah I'm not sure how to tell which version has the changes. It's for sure in 0.43.x but the current "Latest" tag is 0.39.x and doesn't seem to have the fix. I think we should wait for or to verify.
It looks like updates the page's docker-specific information related to cAdvisor to be more generic to container runtimes. Would that be enough to address this issue or does more runtime-specific information need to be ?
what's your opinion in respect of ?

It's due to it being reverted in which is a real bummer for those of us using GKE and the default for new node pools is containerd.
/triage accepted
What should the next steps on this issue look like?
my thoughts: This doesn't belong on the suggested page because it's specific to containerd. I'd say let's update as follows: In Before you begin, add a line to "Read the known issues section" Add a section at the end, before What's Next, called "Known issues after migrating" Add "Versions of containerd prior to 0.45.0 didn't collect some metrics in the endpoint. For a list of missing metrics, refer to . To collect these metrics, install a containerd package that is version 0.45.0 or later that has the suffix." In , mention "To avoid the missing metrics in [Known issues after migrating](#known-issues-after-migrating), install a containerd package that is version 0.45.0 or later that has the suffix."
According to the comment at the only current work-around for containerd users is to deploy and scrape a standalone cadvisor daemonset, built from the branch. The stats exposed by the kubelet will be broken until 1.25 or 1.26 when is complete. Since it is the kubelet's cadvisor that is broken (and I would argue that it is broken/regressed by this issue), it should be covered in the kubernetes docs.
Is there a more involved process then than just installing the package?
it's not a package, it's a branch of the cadvisor project: I'm not sure if they're publishing binaries or images from that branch. A sample DaemonSet is available, which could be modified to use :
Could we say: "To collect these metrics, . Set the $VERSION variable to any cAdvisor version that has the suffix, such as " ?
I think that needs more work, for example the default DaemonSet only mounts the docker socket path; it does not mount the containerd socket, and even when I modify it to do so the metrics appear to be missing most of the important labels. Perhaps has some suggestions.

This feels like it would need to be its own task page, possibly linked from our existing Metrics doc as well
I've put a ready to run temporary solution here
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale