Change:
Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2. or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!

Discussion:
// the user and has network stats. podStats.Network = cadvisorInfoToNetworkStats(&cinfo) } else { <del> podStats.Containers = append(podStats.Containers, *cadvisorInfoToContainerStats(containerName, &cinfo, &rootFsInfo, &imageFsInfo)) </del> <ins> containerStat := cadvisorInfoToContainerStats(containerName, &cinfo, &rootFsInfo, &imageFsInfo) // NOTE: This doesn't support the old pod log path, `/var/log/pods/UID`. For containers // using old log path, they will be populated by cadvisorInfoToContainerStats. podUID := types.UID(podStats.PodRef.UID) logs, err := p.hostStatsProvider.getPodContainerLogStats(podStats.PodRef.Namespace, podStats.PodRef.Name, podUID, containerName, &rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch container log stats", "containerName", containerName) } else { containerStat.Logs = logs } podStats.Containers = append(podStats.Containers, *containerStat) </ins> } }

cadvisortest "k8s.io/kubernetes/pkg/kubelet/cadvisor/testing" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" containertest "k8s.io/kubernetes/pkg/kubelet/container/testing" <ins> kubecontainertest "k8s.io/kubernetes/pkg/kubelet/container/testing" "k8s.io/kubernetes/pkg/kubelet/kuberuntime" </ins> "k8s.io/kubernetes/pkg/kubelet/leaky" serverstats "k8s.io/kubernetes/pkg/kubelet/server/stats" statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" <ins> "k8s.io/kubernetes/pkg/volume" </ins> ) func TestFilterTerminatedContainerInfoAndAssembleByPodCgroupKey(t *testing.T) {

assert.EqualValues(t, p0Time.Unix(), ps.StartTime.Time.Unix()) checkNetworkStats(t, "Pod0", seedPod0Infra, ps.Network) <del> checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, []int{seedEphemeralVolume1, seedEphemeralVolume2}, ps.EphemeralStorage) </del> <ins> checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, []int{seedEphemeralVolume1, seedEphemeralVolume2}, nil, ps.EphemeralStorage) </ins> if ps.CPU != nil { checkCPUStats(t, "Pod0", seedPod0Infra, ps.CPU) }

assert.Equal(imageFsInfo.Inodes, stats.Inodes) assert.Equal(*imageFsInfo.Inodes-*imageFsInfo.InodesFree, *stats.InodesUsed) } <ins> func TestCadvisorListPodStatsWhenContainerLogFound(t *testing.T) { const ( namespace0 = "test0" ) const ( seedRoot           = 0 seedRuntime        = 100 seedKubelet        = 200 seedMisc           = 300 seedPod0Infra      = 1000 seedPod0Container0 = 0 seedPod0Container1 = 0 ) const ( pName0 = "pod0" ) const ( cName00 = "c0" cName01 = "c1" ) const ( rootfsCapacity    = uint64(10000000) rootfsAvailable   = uint64(5000000) rootfsInodesFree  = uint64(1000) rootfsInodes      = uint64(2000) imagefsCapacity   = uint64(20000000) imagefsAvailable  = uint64(8000000) imagefsInodesFree = uint64(2000) imagefsInodes     = uint64(4000) ) prf0 := statsapi.PodReference{Name: pName0, Namespace: namespace0, UID: "UID" + pName0} infos := map[string]cadvisorapiv2.ContainerInfo{ "/":              getTestContainerInfo(seedRoot, "", "", ""), "/docker-daemon": getTestContainerInfo(seedRuntime, "", "", ""), "/kubelet":       getTestContainerInfo(seedKubelet, "", "", ""), "/system":        getTestContainerInfo(seedMisc, "", "", ""), // Pod0 - Namespace0 "/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace0, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace0, cName00), "/pod0-c1": getTestContainerInfo(seedPod0Container1, pName0, namespace0, cName01), } containerLogStats0 := makeFakeLogStats(0) containerLogStats1 := makeFakeLogStats(0) fakeStats := map[string]*volume.Metrics{ kuberuntime.BuildContainerLogsDirectory(prf0.Namespace, prf0.Name, types.UID(prf0.UID), cName00): containerLogStats0, kuberuntime.BuildContainerLogsDirectory(prf0.Namespace, prf0.Name, types.UID(prf0.UID), cName01): containerLogStats1, } fakeStatsSlice := []*volume.Metrics{containerLogStats0, containerLogStats1} fakeOS := &kubecontainertest.FakeOS{} freeRootfsInodes := rootfsInodesFree totalRootfsInodes := rootfsInodes rootfs := cadvisorapiv2.FsInfo{ Capacity:   rootfsCapacity, Available:  rootfsAvailable, InodesFree: &freeRootfsInodes, Inodes:     &totalRootfsInodes, } freeImagefsInodes := imagefsInodesFree totalImagefsInodes := imagefsInodes imagefs := cadvisorapiv2.FsInfo{ Capacity:   imagefsCapacity, Available:  imagefsAvailable, InodesFree: &freeImagefsInodes, Inodes:     &totalImagefsInodes, } options := cadvisorapiv2.RequestOptions{ IdType:    cadvisorapiv2.TypeName, Count:     2, Recursive: true, } mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) mockCadvisor.EXPECT().ContainerInfoV2("/", options).Return(infos, nil) mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisor.EXPECT().ImagesFsInfo().Return(imagefs, nil) mockRuntime := containertest.NewMockRuntime(mockCtrl) mockRuntime.EXPECT().ImageStats().Return(&kubecontainer.ImageStats{TotalStorageBytes: 123}, nil).AnyTimes() volumeStats := serverstats.PodVolumeStats{} p0Time := metav1.Now() mockStatus := statustest.NewMockPodStatusProvider(mockCtrl) mockStatus.EXPECT().GetPodStatus(types.UID("UID"+pName0)).Return(v1.PodStatus{StartTime: &p0Time}, true) resourceAnalyzer := &fakeResourceAnalyzer{podVolumeStats: volumeStats} p := NewCadvisorStatsProvider(mockCadvisor, resourceAnalyzer, nil, nil, mockRuntime, mockStatus, NewFakeHostStatsProviderWithData(fakeStats, fakeOS)) pods, err := p.ListPodStats() assert.NoError(t, err) assert.Equal(t, 1, len(pods)) // Validate Pod0 Results checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, nil, fakeStatsSlice, pods[0].EphemeralStorage) } </ins>

result.CPU = cpu result.Memory = memory <ins> // NOTE: if they can be found, log stats will be overwritten // by the caller, as it knows more information about the pod, // which is needed to determine log size. </ins> if rootFs != nil { // The container logs live on the node rootfs device result.Logs = buildLogsStats(cstat, rootFs)

kubepodtest "k8s.io/kubernetes/pkg/kubelet/pod/testing" serverstats "k8s.io/kubernetes/pkg/kubelet/server/stats" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" <ins> "k8s.io/kubernetes/pkg/volume" </ins> ) const (

assert.EqualValues(t, seed+offsetFsInodesFree, *stats.InodesFree, label+".InodesFree") } <del> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, stats *statsapi.FsStats) { </del> <ins> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, containerLogStats []*volume.Metrics, stats *statsapi.FsStats) { </ins> var usedBytes, inodeUsage int for _, cseed := range containerSeeds { <del> usedBytes = usedBytes + cseed + offsetFsTotalUsageBytes </del> <ins> usedBytes += cseed + offsetFsBaseUsageBytes </ins> inodeUsage += cseed + offsetFsInodeUsage <ins> // If containerLogStats is nil, then the log stats calculated from cAdvisor // information is used. Since it's Total - Base, and these values are // set to the offset, we can use the calculated difference in the offset // to account for this. if containerLogStats == nil { usedBytes += offsetFsTotalUsageBytes - offsetFsBaseUsageBytes } </ins> } for _, vseed := range volumeSeeds { <del> usedBytes = usedBytes + vseed + offsetFsUsage </del> <ins> usedBytes += vseed + offsetFsUsage </ins> inodeUsage += vseed + offsetFsInodeUsage } <ins> for _, logStats := range containerLogStats { usedBytes += int(logStats.Used.Value()) inodeUsage += int(logStats.InodesUsed.Value()) } </ins> assert.EqualValues(t, usedBytes, int(*stats.UsedBytes), label+".UsedBytes") assert.EqualValues(t, inodeUsage, int(*stats.InodesUsed), label+".InodesUsed") }