Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- 03-27 Two different failure reasons /assign /milestone 1.20 /priority critical-urgent /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , ] Use to clear the milestone. <detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- #### Which test(s) are failing: 03-27 #### Testgrid link: Two different failure reasons #### Anything else we need to know: /milestone 1.20 /priority critical-urgent /cc </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.21
/sig scalability cc
In both cases we have no master logs which makes it nearly impossible to debug this. should fix the issue with missing logs. In the meantime, I'm running a manual run of the same config as the last failing run
one of the reasons for failure is: We also see quite a few OOMs on ~10 nodes. The processes that are oom killed looks quite random: pause pods, system-logind, systemd-journal. It's quite unlikely that those are the processes that eats all the memory
I checked serial console logs for one of nodes where OOM has been reported and most likely metrics-server is a component that consumes a lot of memory and triggers OOM killer on nodes. In my manual run that is currently happening we see that it fails and jumps from node to node triggering OOMs on another node:
From successful run: There are multiple instances of metrics-server, but all of them in a short period of time (when the cluster starts). They are recreated by podnanny. It looks like in failed runs podnanny for metrics-server failed to resize metrics-server deployment to a size appropriate for 5k node cluster.
From pod_nanny logs (in the manual run):
Indeed, 'etcdobjectcounts' disappeared: ~ kubectl get --raw /metrics | grep etcdobjectcounts ~
It looks like etcdobjectcounts has been renamed () and this makes podnanny not able to resize metrics-server. For that reason metrics-server is not setting any resources (should be set by podnanny based on the value of the metric) which allows it to schedule on small nodes (other than the "heapster" one) and OOM continuously breaking other components running on that node (e.g. daemonset pods). The questions here: how is it possible that we discover this now as the has been merged ~20 days ago. Is it intentional that etcdobjectcounts disappears in the same version as the new metric appears? It may be hard to adjust addon-resizer if there is no grace period when both are available.
/cc Is it intentional that etcdobjectcounts disappears in 1.21 version? Shouldn't we move this to at least 1.22 to give some grace period when both metrics are available?
I can't remember, did we delete the original? We should probably retain it for a release or two prior for the reason above.
cc
Are you sure they are gone?
We are double writing at the moment.
Ah actually I think the deprecated version was incorrect, so the metric was hidden a release too soon.
/reopen Thanks for a fix! I would like to keep it open until we see at least a single green run to double check that this was the only issue in the test
Reopened this issue. <detailsIn response to : /reopen I would like to keep it open until we see at least a single green run to double check that this was the only issue in the test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
https://gke-scalability- has passed In kube-scheduler logs I see that metrics-server was correctly upsized by addon-manager (resizes happens on the very beginning of the test + hash in the name changes which means that deployment's spec has changed).
The issue seems to be resolved. I triggered another manual run in the internal project to make sure that it's not flake. We expect result in ~4h.
The manual test has passed. I believe this is resolved.

Discussion:
etcdObjectCounts = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ Name:              "etcd_object_counts", <del> DeprecatedVersion: "1.21.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> Help:              "Number of stored objects at the time of last check split by kind. This metric is replaced by apiserver_storage_object_counts.", StabilityLevel:    compbasemetrics.ALPHA, },