Change:
<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --as discussed in issue and , we get the metrics of terminated container. metric information about terminated pods is expected? If so, that has changed recently? Kubelet endpoints and CRI metrics protos should report metrics for terminated container should be standardized
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/cc
/assign
We only provide metrics for non-terminal pods: We only provide metrics for running containers:
ok, so issue reported to Metrics Server looks is a bug in container runtime CRI metrics implementation.
/cc
also cc as both of those examples are from cadvisor (as cri-o uses the cadvisor stats provider)
/retitle Kubelet should not report metrics from init containers
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen
I believe this is only a bug in the cadvisor stats provider, as there is in the cri stats provider code to handle this. Between the dockershim deprecation and the (hopeful) promotion of PodAndContainerStatsFromCRI to beta in 1.24, I am wondering if it is worth the time to fix this in the kubelet. dockershim and CRI-O are the last users of the cadvisor stats provider, and CRI-O will use CRI stats when PodAndContainerStatsFromCRI is true by default. If one were to fix this, I think the spot would be in which according to (which was lost in but the code seems to be the same) should filter out terminated containers, but doesn't seem to be doing so.
This seems to be fixed for me in 1.23 (using , so as per , is still using legacy stats), and the only "recent" PR touching related to pod stats is , so I guess that refactor actually fixed filtering out terminated containers, ? Either this, or it was fixed in in the meanwhile.
Doing a bit more digging (I just can't help it!), here (which in turn calls directly here ), returned doesn't contain any exited init containers data, so the filtering is done by . was bumped in from v0.39.2 to v0.43.0, so I'm guessing the fix should be somewhere .
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc As discussed in pr , we need to consider other ways to fix this issue

Discussion:
result := make(map[string]cadvisorapiv2.ContainerInfo) for _, refs := range cinfoMap { if len(refs) == 1 { <del> result[refs[0].cgroup] = refs[0].cinfo </del> <ins> // ContainerInfo with no CPU/memory usage for uncleaned cgroups of // already terminated containers, which should not be shown in the results. if !isContainerTerminated(&refs[0].cinfo) { result[refs[0].cgroup] = refs[0].cinfo } </ins> continue } sort.Sort(ByCreationTime(refs))

return cstat.CpuInst.Usage.Total != 0 && cstat.Memory.RSS != 0 } <ins> // isContainerTerminated returns true if the specified container info has // both zero CPU instantaneous usage and zero memory RSS usage, and // false otherwise. func isContainerTerminated(info *cadvisorapiv2.ContainerInfo) bool { if !info.Spec.HasCpu && !info.Spec.HasMemory { return true } cstat, found := latestContainerStats(info) if !found { return true } if cstat.CpuInst == nil || cstat.Memory == nil { return true } return cstat.CpuInst.Usage.Total == 0 && cstat.Memory.RSS == 0 } </ins> func getCadvisorContainerInfo(ca cadvisor.Interface) (map[string]cadvisorapiv2.ContainerInfo, error) { infos, err := ca.ContainerInfoV2("/", cadvisorapiv2.RequestOptions{ IdType:    cadvisorapiv2.TypeName,

namespace = "test" pName0    = "pod0" cName00   = "c0" <ins> pName1    = "pod1" cName11   = "c1" pName2    = "pod2" cName22   = "c2" </ins> ) infos := map[string]cadvisorapiv2.ContainerInfo{ // ContainerInfo with past creation time and no CPU/memory usage for

// The latest containers, which should be in the results. "/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace, cName00), <ins> "/pod1-i.slice":  getTestContainerInfo(seedPod0Infra, pName1, namespace, leaky.PodInfraContainerName), "/pod1-c1.slice": getTestContainerInfo(seedPod0Container0, pName1, namespace, cName11), "/pod2-i-terminated-1": getTerminatedContainerInfo(seedPastPod0Infra, pName2, namespace, leaky.PodInfraContainerName), // ContainerInfo with past creation time and no CPU/memory usage for // simulating uncleaned cgroups of already terminated containers, which // should not be shown in the results. "/pod2-c0-terminated-1": getTerminatedContainerInfo(seedPastPod0Container0, pName2, namespace, cName22), </ins> } filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) <del> assert.Len(t, filteredInfos, 2) assert.Len(t, allInfos, 6) </del> <ins> assert.Len(t, filteredInfos, 4) assert.Len(t, allInfos, 10) </ins> for _, c := range []string{"/pod0-i", "/pod0-c0"} { if _, found := filteredInfos[c]; !found { t.Errorf("%q is expected to be in the outputn", c) } } <del> for _, c := range []string{"pod0-i-terminated-1", "pod0-c0-terminated-1", "pod0-i-terminated-2", "pod0-c0-terminated-2", "pod0-i", "pod0-c0"} { </del> <ins> for _, c := range []string{"pod0-i-terminated-1", "pod0-c0-terminated-1", "pod0-i-terminated-2", "pod0-c0-terminated-2", "pod0-i", "pod0-c0", "c1"} { </ins> if _, found := allInfos[c]; !found { t.Errorf("%q is expected to be in the outputn", c) }

ps, found = indexPods[prf3] assert.True(t, found) <del> assert.Len(t, ps.Containers, 2) </del> <ins> // /pod3-c0-init has no stats should be filtered assert.Len(t, ps.Containers, 1) </ins> indexCon = make(map[string]statsapi.ContainerStats, len(ps.Containers)) for _, con := range ps.Containers { indexCon[con.Name] = con

assert.Equal(t, cName31, con.Name) checkCPUStats(t, "Pod3Container1", seedPod3Container1, con.CPU) checkMemoryStats(t, "Pod3Container1", seedPod3Container1, infos["/pod3-c1"], con.Memory) <del> con = indexCon[cName30] assert.Equal(t, cName30, con.Name) checkEmptyCPUStats(t, "Pod3Container0", seedPod3Container0, con.CPU) checkEmptyMemoryStats(t, "Pod3Container0", seedPod3Container0, infos["/pod3-c0-init"], con.Memory) </del> } func TestCadvisorListPodCPUAndMemoryStats(t *testing.T) {

} <del> // container which had no stats should have zero-valued CPU usage func checkEmptyCPUStats(t *testing.T, label string, seed int, stats *statsapi.CPUStats) { require.NotNil(t, stats.Time, label+".CPU.Time") require.NotNil(t, stats.UsageNanoCores, label+".CPU.UsageNanoCores") require.NotNil(t, stats.UsageNanoCores, label+".CPU.UsageCoreSeconds") assert.EqualValues(t, testTime(timestamp, seed).Unix(), stats.Time.Time.Unix(), label+".CPU.Time") assert.EqualValues(t, 0, *stats.UsageNanoCores, label+".CPU.UsageCores") assert.EqualValues(t, 0, *stats.UsageCoreNanoSeconds, label+".CPU.UsageCoreSeconds") } // container which had no stats should have zero-valued Memory usage func checkEmptyMemoryStats(t *testing.T, label string, seed int, info cadvisorapiv2.ContainerInfo, stats *statsapi.MemoryStats) { assert.EqualValues(t, testTime(timestamp, seed).Unix(), stats.Time.Time.Unix(), label+".Mem.Time") require.NotNil(t, stats.WorkingSetBytes, label+".Mem.WorkingSetBytes") assert.EqualValues(t, 0, *stats.WorkingSetBytes, label+".Mem.WorkingSetBytes") assert.Nil(t, stats.UsageBytes, label+".Mem.UsageBytes") assert.Nil(t, stats.RSSBytes, label+".Mem.RSSBytes") assert.Nil(t, stats.PageFaults, label+".Mem.PageFaults") assert.Nil(t, stats.MajorPageFaults, label+".Mem.MajorPageFaults") assert.Nil(t, stats.AvailableBytes, label+".Mem.AvailableBytes") } </del> func checkCPUStats(t *testing.T, label string, seed int, stats *statsapi.CPUStats) { require.NotNil(t, stats.Time, label+".CPU.Time") require.NotNil(t, stats.UsageNanoCores, label+".CPU.UsageNanoCores")