Change:
Test is failing on crio conformance (cgroup v1 / cgroup v2) Test doesn't fail. You can reproduce it on a PR with /test pull-kubernetes-node-crio-e2e. I reverted go bump and that was not the case. master CI <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority critical </details
failed to get imageFS data
Ah. Yea this confirms it.
looks like the KEP-4191 bump to beta is causing the failure.
/priority critical-urgent /sig node
/assign kwilczynski
CRI-O version and runtime configuration from the test run: Version string (from the log)Configuration
CRI-O is not configured to enable the split image file system per: (an excerpt from the runtime configuration)
The code that is returning the error:
Other logs I see:
Are we planning to get this fixed for v1.31 or revert the promotion of KEP 4191 to beta with Since code freeze is in effect now, we'd need approval from the release team either way. If we choose to revert the graduation to beta, the KEP won't be part of v1.31.
I think we have a bug where we have the feature enabled but the container runtime is not configured.
I am looking at fixing this as soon as possible...
That PR was mostly to find out the issue. I'd like to fix it and we will file a release exception as we are past the freeze.
Since you're planning to file an exception request, I'm removing the KEP from the milestone so that the release team can better track KEPs. If the issue is not fixed and the exception request is not raised, we would still need to revert the KEP back into alpha (). We should either fix the issue and merge any additional code changes with an exception request or revert the feature gate to alpha as done in the above PR.
/cc
this is actually going to be quite a common use case, as users would most likely not have their runtimes configured initially or even at all (if someone does not care about things like a split or separate filesystem). I wonder how we can reliably detect whether a runtime has support enabled or not. At the moment, the error originates from cAdvisor, where we ask for stats under a specific label, which does not exist. Since the feature is enabled in Kubernetes, we assume that the runtime also has it enabled and send the label that pertains to ContainreFs. There is a features discovery bit to the CRI API, which we already use to divulge details about support for Recursive Read-only Mounts, Supplemental Groups, etc. I wonder if we should also convey support about split filesystem support that way?
/milestone v1.31 Not only stat summary, we can see other similar failures.
So this only fails in the case of crio with feature on but system is not configured. I used the ImgaeFsInFo to detect if the system is split so I am curious if something went wrong there. Containerd is not failing at all with this feature on so we are detecting containers lack of support without issue.
this is easily reproducible if you have CRI-O running without the option set. So, is this what you mean by "system not configured"?
Yes.
I see , is this going to be reverted or are you working on a fix?
/assign kannon92
We have open to solve this. Waiting on exception approval and this to the milestone. Confirmed that these tests work now.

Discussion:
} return imageFs, imageFs, nil } <del> containerFsInfo, err := p.cadvisor.ContainerFsInfo() if err != nil { return nil, nil, fmt.Errorf("failed to get container fs info: %v", err) } </del> imageStats, err := p.imageService.ImageFsInfo(ctx) if err != nil || imageStats == nil { return nil, nil, fmt.Errorf("failed to get image stats: %v", err)

Inodes:         imageFsInfo.Inodes, InodesUsed:     imageFsInodesUsed, } <ins> // We rely on cadvisor to have the crio-containers label for split filesystem case. // We return to avoid checking ContainerFsInfo. </ins> if !splitFileSystem { return fsStats, fsStats, nil } <ins> containerFsInfo, err := p.cadvisor.ContainerFsInfo() if err != nil { return nil, nil, fmt.Errorf("failed to get container fs info: %w", err) } </ins> containerFs := imageStats.ContainerFilesystems[0] var containerFsInodesUsed *uint64 if containerFsInfo.Inodes != nil && containerFsInfo.InodesFree != nil {

featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.KubeletSeparateDiskGC, true) mockCadvisor.EXPECT().ImagesFsInfo().Return(imageFsInfo, nil) <del> mockCadvisor.EXPECT().ContainerFsInfo().Return(imageFsInfo, nil) </del> mockRuntime.EXPECT().ImageFsInfo(ctx).Return(imageFsInfoResponse, nil) provider := newCadvisorStatsProvider(mockCadvisor, &fakeResourceAnalyzer{}, mockRuntime, nil, NewFakeHostStatsProvider())