Change:
assert.EqualValues(t, seed+offsetFsInodesFree, *stats.InodesFree, label+".InodesFree") } <del> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, stats *statsapi.FsStats) { </del> <ins> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, containerLogStats []*volume.Metrics, stats *statsapi.FsStats) { </ins> var usedBytes, inodeUsage int for _, cseed := range containerSeeds { <del> usedBytes = usedBytes + cseed + offsetFsTotalUsageBytes </del> <ins> usedBytes += cseed + offsetFsBaseUsageBytes </ins> inodeUsage += cseed + offsetFsInodeUsage <ins> // If containerLogStats is nil, then the log stats calculated from cAdvisor // information is used. Since it's Total - Base, and these values are // set to the offset, we can use the calculated difference in the offset // to account for this. if containerLogStats == nil { usedBytes += offsetFsTotalUsageBytes - offsetFsBaseUsageBytes } </ins> } for _, vseed := range volumeSeeds { <del> usedBytes = usedBytes + vseed + offsetFsUsage </del> <ins> usedBytes += vseed + offsetFsUsage </ins> inodeUsage += vseed + offsetFsInodeUsage } <ins> for _, logStats := range containerLogStats { usedBytes += int(logStats.Used.Value()) inodeUsage += int(logStats.InodesUsed.Value()) } </ins> assert.EqualValues(t, usedBytes, int(*stats.UsedBytes), label+".UsedBytes") assert.EqualValues(t, inodeUsage, int(*stats.InodesUsed), label+".InodesUsed") }

Discussion:
Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!