Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- (gce-ubuntu-master-default) Kubernetes e2e suite: BeforeSuite (There are other flakes in this job, but they are failing less often and different test each time, so let's say this issue is only for BeforeSuite) is not starting: Spyglass: Triage: Mentioned in /cc <!-- Please see the deflaking doc () for more guidance! --
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc Since it is related to the metrics server. the metrics server keeps dying as the log shown in I cannot get more logs of metrics servers from the CI artifacts. Currently, we are using metric server 0.4.4 and the latest 0.5.0 is released. Are there some known bugs that may be related to this test failure?
I don't think so. /cc
The metrics-server fails to bind ~because there is another process listening~ on port 443 are some of the konnectivity pods listening on the port :443?
In the dockerfile of metrics-server if has ? Maybe it has permission problem?
ahh, right, nice catch, I didn't pay attention to the error and blindly assumed collision :sweat_smile: ... who can take an action on that? has this changed recently?
If the metrics-server version used after , I think it is correct. has no change recently.
I vaguely remember some changes recently to reduce privileges on pods on CI, tthis seems likely a permission problem, does this ring a bell? EDIT This PR introduced the change , any special reason why we want to run it in a privileged port if this is accessed using a Service? EDIT 2 let see if this is the problem
Yeah, Maybe you can refer to this, the release notes about v0.4.4 metrics-server . I think you have resolved this problem.
Release v0.3.X was already using privileged port 443, as v0.4.X has migrated to distroless image we needed to introduce to give it needed privileges. What's weird for me is that the test is flaking and not failing. Could it be related to node where MS is scheduled?
You mean, there may be other reasons? If so, may require in-depth analysis.
As far as I can tell, is never starting properly, even when the jobs are successful. If we were to take this job, that has succeeded: From the serial logs, we can see that the metrics-server pod is crash looping: But from the build logs, we can see that at some point the pod was deemed ready: To me it seems that there are 2 issues, one being that metrics-server doesn't have the capabilities it needs to run and another one that still needs to be investigated where metrics-server is sometimes marked ready even though it is in a crash looping state.
This is a problem with the metrics-server deployment on the CI jobs and the metrics tests I don't know which sig owns these components but it will be better to find first an owner, what is is clear is that the metrics tests are not reliable since once you set a readiness probe on the metrics-server it fails consistently , , however the tests pass cc
it seems the last try was the good one :smile:

Discussion:
apiVersion: apps/v1 kind: Deployment metadata: <del> name: metrics-server-v0.4.4 </del> <ins> name: metrics-server-v0.5.0 </ins> namespace: kube-system labels: k8s-app: metrics-server addonmanager.kubernetes.io/mode: Reconcile <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> spec: selector: matchLabels: k8s-app: metrics-server <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> template: metadata: name: metrics-server labels: k8s-app: metrics-server <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> spec: securityContext: seccompProfile:

kubernetes.io/os: linux containers: - name: metrics-server <del> image: k8s.gcr.io/metrics-server/metrics-server:v0.4.4 </del> <ins> image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0 </ins> command: - /metrics-server - --metric-resolution=30s <del> # These are needed for GKE, which doesn't support secure communication yet. # Remove these lines for non-GKE clusters, and when GKE supports token-based auth. - --kubelet-port=10255 - --deprecated-kubelet-completely-insecure=true </del> <ins> - --kubelet-use-node-status-port - --kubelet-insecure-tls </ins> - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP - --cert-dir=/tmp - --secure-port=443

- containerPort: 443 name: https protocol: TCP <ins> readinessProbe: httpGet: path: /readyz port: https scheme: HTTPS periodSeconds: 10 failureThreshold: 3 livenessProbe: httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 failureThreshold: 3 </ins> volumeMounts: - mountPath: /tmp name: tmp-dir

- --memory={{ base_metrics_server_memory }} - --extra-memory={{ metrics_server_memory_per_node }}Mi - --threshold=5 <del> - --deployment=metrics-server-v0.4.4 </del> <ins> - --deployment=metrics-server-v0.5.0 </ins> - --container=metrics-server - --poll-period=30000 - --estimator=exponential