Change:
currentLine++ } if currentLine == len(outputLines) { <del> framework.Failf("Failed to find %s in %s", requirement[0], output) </del> <ins> return fmt.Errorf("failed to find %s in %s", requirement[0], output) </ins> } for _, item := range requirement[1:] { if !strings.Contains(outputLines[currentLine], item) { <del> framework.Failf("Failed to find %s in %s", item, outputLines[currentLine]) </del> <ins> return fmt.Errorf("failed to find %s in %s", item, outputLines[currentLine]) </ins> } } } <ins> return nil } func checkOutput(output string, required [][]string) { err := checkOutputReturnError(output, required) if err != nil { framework.Failf("%v", err) } } func checkKubectlOutputWithRetry(required [][]string, args ...string) { var pollErr error wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie(args...) err := checkOutputReturnError(output, required) if err != nil { pollErr = err return false, nil } pollErr = nil return true, nil }) if pollErr != nil { framework.Failf("%v", pollErr) } return </ins> } func getAPIVersions(apiEndpoint string) (*metav1.APIVersions, error) {

Discussion:
https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing.

I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close