Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-node-crio-cgrpv2-e2e E2eNode Suite: [] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api E2eNode Suite: [] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resource metrics api From what I have observed these tests have always failed on cgroup v2 host. cgroup v2 has a different set of files compared to cgroup v1 to report the metrics. A lot of places in cadvisor, where the metrics end up getting read from, still assume cgroup v1 paths and files. Some of the following PRs end up addressing the issue partially, but not completely.
/sig node
/cc
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
still working my way towards a reproducer.
I've opened to fix one issue I've seen. I think we will also need to modify libcontainer to return memory stats from for the root cgroup /cc
another fix for cadvisor:
for completeness, the patch for runc/libcontainer:
this should be fixed with
can we close now?
Test is still failing. See testgrid:
hm.. it is failing on: not sure how to deal with this test. Should we handle it differently for cgroup v2? Or just enable it for cgroup v1? what do you think?
I don't know how changes in conformance test should be done. The issue in this case is that for cgroup v1, and will always be on due to how the hierarchy stuff works there. Not sure why the conformance test is doing instead of just .... For v2 these value will sum all the page faults in all descendant processes below it. The same applies to pods as well, but it looks like there are no test catching that. Updating the value in the test is the "simplest" fix, but it doesn't solve the underlying difference in metrics when switching to cgroup v2 (eg. if this cause someones cluster to go wilde du to the "extreme" increase in this value). Who is responsible for these metrics/stats, and who should decide what to do? sig-node? I don't really know what to do.. Maybe has some thoughts?
I am not sure how that metric was used on cgroup v1. We could hardcode it to 0 for cgroup v2, but I don't think it is a good idea as it finally seems to make sense :-) Is it just something that should be documented?
No idea myself. A quick test on my rpi using cgroup v1 I get this: Where the first non-null value is and the other one is the root control group; and that is pretty much expected based on the fact that the value is not hierarchical for cgroup v2. Not sure who "own" this.. :thinking:
Thanks for trying that. Opened a PR for comments: If that is not the right approach, the only alternative I see is to hardcode the value to 0.
The git blame shows that this was introduced here: Do you have any thoughts about this
Also, the structs were introduced here: /cc
I think changing the test is the right thing to do. It is a breaking change to change the metric, but it wasn't particularly useful before. Also, we won't be able to hide the change to the metric for pod cgroup metrics, so we might as well change it for system containers as well.
Ahh, sounds good. Thanks!
I'm trying to port the PR to 1.20 version which we're using with cgroups v2. Could you please give more information about the failing tests? such as detailed failure messages. Thanks very much! Actually, I got the same error when running the test against upstream master with this PR and 1.20:
I see no failures in and jobs with the stats summary test with master.

Discussion:
"UserDefinedMetrics": gomega.BeEmpty(), }) } <ins> expectedMajorPageFaultsUpperBound := 10 if IsCgroup2UnifiedMode() { expectedMajorPageFaultsUpperBound = 1000 } </ins> podsContExpectations := sysContExpectations().(*gstruct.FieldsMatcher) podsContExpectations.Fields["Memory"] = ptrMatchAllFields(gstruct.Fields{ "Time": recent(maxStatsAge),

"WorkingSetBytes": bounded(10*e2evolume.Kb, memoryLimit), "RSSBytes":        bounded(1*e2evolume.Kb, memoryLimit), "PageFaults":      bounded(0, 1000000), <del> "MajorPageFaults": bounded(0, 10), </del> <ins> "MajorPageFaults": bounded(0, expectedMajorPageFaultsUpperBound), </ins> }) runtimeContExpectations := sysContExpectations().(*gstruct.FieldsMatcher) if systemdutil.IsRunningSystemd() && framework.TestContext.ContainerRuntime == "docker" {