Change:
What happened: New overlay filesystems created for Pods after kubelet start trigger repeated error messages. For example: Error messages are logged every 10 seconds for every filesystem created after kubelet start. For large kubelet nodes, this issue creates an excessive amount of error logs. What you expected to happen: New overlay filesystems created for Pods after kubelet start should not log error messages. How to reproduce it (as minimally and precisely as possible): Use cristatsprovider with cri-o and configure overlay storage. UsingLegacyCadvisorStats() needs to return false per for cristatsprovider to be used. Start with an empty /var/lib/containers/storage and schedule Pods on the node. Anything else we need to know?: As part of , cadvisor.GetFsInfoByFsUUID() was changed to cadvisor.GetDirFsInfo() in getFsInfo(); however, the test for cadvisorfs.ErrNoSuchDevice was not modified, changing how getFsInfo() handles a cache miss from cadvisor. cadvisor caches UUIDs/partitions/mounts on kubelet start and the cache is never refreshed. Given that it's expected that getFsInfo() will fail to retrieve data from cadvisor (see comments in ImageFsStats() and ), this condition should not be logged as an error. Alternatively, cadvisor fs could be modified to refresh on cache miss enabling getFsInfo() to return data on these filesystems. Environment: Kubernetes version (use ): 1.18.6 CRI: CRI-O
/sig node cc
ptal
thanks for reporting the issue. I will take a look at in upcoming week.
/assign
Sorry this is taking slightly longer. Currently held up in something little urgent, but as soon as that's over I will look into this one.
Do you have the time to look at this issue ? I can help with testing / debugging if required. Environment Kubernetes 1.19.3 Cri-o 1.19.0
Sorry I haven't gotten a chance to look into it yet, I would certainly appreciate any help in debugging. Thanks.
Right now I am thinking of either handling the cache miss better, or returning the even if the fs type is not in cadvisor, or both.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
kindly ping - has this already been implemented ?
/assign
Fix for k/k in and cadvisor in
/remove-lifecycle rotten
hi do you know when the fixes will be released ? Thanks!
I don't, unfortunately.
the fix has made it into , so I imagine the change will come in by 1.22
/triage accepted /priority important-soon
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
So v1.22 is out for a while now. Is there someone who has already upgraded to v1.22 and can confirm that the getFsInfo errors are fixed?
haircommander mentioned that the fix made it into cadvisor 0.40.0-rc.1. But cadvisor version bump did not happen in k8s yet, probably due to issues mentioned here: iirc kind did not work with image kindest/node v1.22.1, I did not look into logs however. Will update post once I have access to the affected device. Edit:
I still have these logs with version 1.22.2
I have got the same error log about kubelet kubernetes version 1.22.2 containerd: 1.4.11 Is this problem have been resolve?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I still have these logs with version 1.23.1
turns out we ended up being blocked on , so the fix has yet to be merged. let's push for it in 1.24
I'll wait for 1.24. With 1.22.5 My cluster has some of these messages but is stable. But with 1.23.x I get disk pressure because my sys logs get spammed quickly. I hoped going away fomr cadvisor to CRI stats woudl help, but instead of stat errors I now have these FS errors.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant and requires a fix with
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale Related PR still not merged.
SnoozeThis is useful if we don't want to keep polling until that PR is closed.
() I will wait until is merged and then add a comment.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Hi folks, I still have this issue in most recent Kubernetes 1.24.6. cAdvisor 0.44.0 that fix went into that version, is now a long time ago already in 1.24.0. Dep bump to runc 1.1.0, cadvisor 0.44.0 Automated cherry pick of : Bump cAdvisor to v0.44.1 28.4.2022 So is there possibly any chance to progress with a review of this PR please?
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
Can we have any news on that topic please?
/remove-lifecycle rotten
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Resolved.

Discussion:
"sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)