Code change:
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
failed in https://k8s-
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Any updates on this issue If you could you change the milestone to 1.5 and add a label for the responsible team that would be a great help as we triage flakes for the 1.5 release. Thanks!
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
There is a problem with fetching images on GKE. It seems that pod is unable to fetch image for more than 2 minutes. It is strange, as AFAIK the image should be pre-fetched by image-puller:
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
, all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. If you are not the correct owner please identify the correct owner and add them to this thread for assignment. Thanks for your help. cc:
WRT https://k8s- There was a problem w/removing namespace. This is a duplicate of and is not related to HPA.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Could you please help triage this issue as blocking or non-blocking for 1.5 (or help find someone who can). If there is a real issue here, we need to find out ASAP, or else we risk the 1.5 release date.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
WRT https://k8s- build: kubelet logs for pod rc-light-qyowb : Kubelet is not starting pod for 2 minutes, reason unknown. The same problem is in issue Can you take a look or assign someone from kubelet team?
I suspect that the logging level is set to in GKE tests and so most of the useful logs are not available. Let me figure out a means to bump the logging level to further debug this issue. you might still have to triage the non v1.4 release failures to address concern .
how can we update the environment variables for GKE test clusters? For OSS, we up the kubelet logging for test clusters
I still don't agree with test environmental changes, which is one of the reasons we've never made this a priority. A GKE test cluster looks the same as a GKE prod cluster. How would you debug this in production? Always ask that, and you come back to test == prod.
cc
In my opinion it is release blocker: sometimes pods are not started
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
The pod startup failure you posted in is for release 1.4 branch. Why do you think it is a release blocker? We need to triage other failures on HEAD and/or 1.5 branch which could be release blockers. If the intent is to keep prod and test in sync, then let's up the logging level in prod as well. When issues happen in prod, we might need those missing log lines anyways. I suspect that we might have to have GCI rotate logs more aggressively as a side effect.
Why is the argument to increase the log level rather than making sure that all logs necessary for debugging production issues are at level 2 or above? Levels 3/4 shouldn't be necessary to figure out why a key control loop isn't behaving correctly.
What you suggest is a good fix for the future. We need to reexamine logging levels across kubelet. However, that might not happen in the next few days. How do we make progress now?
Would isolating logging level changes to specific test suites be acceptable? That way, most of our tests would match production setup.
That's what I'd suggested on the other (internal) code review.
Do we have consensus on how to get past this as a release-blocker?
We are waiting on to give a go ahead from the release perspective. The kubelet failures that and I have been discussing about have not been on v1.5 AFAIK.
Sorry, I didn't notice that this problem is on 1.4 branch. If so, the issue is not release-blocker. So, I'm removing the label.
Ack. Marking as non-release-blocker
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
PR should be cherry-picked to 1.5 This is not a release-blocker as it affects e2e tests only.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}

Related discussion:
text: } framework.ExpectNoError(framework.RunRC(controllerRcConfig)) <del> // Make sure endpoints are propagated. // TODO(piosz): replace sleep with endpoints watch. time.Sleep(10 * time.Second) </del> <ins> // Wait for endpoints to propagate for the controller service. framework.ExpectNoError(framework.WaitForServiceEndpointsNum( c, ns, controllerName, 1, startServiceInterval, startServiceTimeout)) </ins> }