Change:
Job should run a job to completion when tasks sometimes fail and are locally restarted Expected error: <*errors.errorString | 0xc20807c990: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred
Couple of failures: from
Another failure:
Another failure:
This test was introduced by and in and . It's currently failing at a rate of about 7%. When it passes, it typically does so in 30-90 seconds. Occasionally it times out after 15 minutes. I think that there's a real bug in the system here. The timeout occurs while waiting for the Job to terminate, which it never does.
I'm going to label it as flaky until the bug has been fixed.
Another portion of flakes:
/sub
any progress on this? Is it being actively worked on?
I've been working on this today. Saw 1 e2e failure in 50 runs. Logs suggest the problem is that half of kubelet's brain thinks it started a pod, and the other half of it thinks it didn't. In fact it didn't. So the test times out waiting for the not-started pod to complete.
Update: the kubelet did start all the pods, and they all terminated, and then the job controller somehow didn't notice that they were done until 40+ minutes later.
This is interesting. There were 3 apiserver 500's around the time of the test failure, and no 500's any other time in the 8 hours the cluster was up, and many other test runs occurred. 500s:
The code that is failing is this in at line 271 the gets an error, and then decides the error is a 500.
can you say anything about the above mentioned apiserver internal error which occurs immediately on receiving a watch request?
Look in the relevant kubelet logs to see how they interpreted the error. It's a bit silly that we printed a stack trace but not the error text :( , Eric Tune wrote:
E1022 04:13:24. 3622 ] Failed to watch *api.Pod: Internal error occurred: too old resource version: 171 ()
This is happening at line 269 of Is this an unexpected condition? If not, why is it a 500?
Oh, I thought that was fixed. It shouldn't be an internal error, you're out of the watch window and need to re-list. , Eric Tune wrote:
OK I reopened . did the kubelets involved re-list soon after getting these 500s? If so, then this is coincidence and not your problem. , Daniel Smith wrote:
In all 3 cases, they relist immediately, but then they don't re-watch for about an hour. That seems wrong, no? Relist at 3:56 and rewatch at 4:53 Relist at 4:07 and rewatch at 5:00 relist at 4:13 and rewatch at 4:55
The time reported is the end of the watch, not the beginning. The last one it seems you actually didn't get the log line for the watch, actually. But the first two look WAI. , Eric Tune wrote:
Oh, duh.
Working on which I suspect may fix this.
Okay, looking at a new test failure I see that there are expected failures, and the test just times out due to backoff by the kubelet. The following things happen: runs and fails (1) runs and fails (2) backs off for 10s runs and fails (3rd failure) backoff 20s runs and fails (4th failure) backoff 40s runs and fails (5th failure) backoff 1m20s runs and fails (6th failure) backoff 2m40s runs and fails (6th failure) kubelet backoff 5m runs and fails (6th failure) backoff 5m timeout now that at least 15m10s has elapsed in backoffs. Failures have a probability of 50%. Chance of getting 8 fails in a row is 1 in 256. Four pods per test run is 4 in 256 or 1 in 64. I see failures about 1 in 50, so that seems about right. So, I need to rewrite the test so that it writes to the emptydir after the first failure, and succeeds after that.
Ugh. My fix for local breaks non-local restarts.
Happened again.

Discussion:
GCE_FLAKY_TESTS=( "DaemonRestartsControllersManager" "Daemonssetsshouldsrunsandsstopscomplexsdaemon" <del> "Jobssareslocallysrestarted" </del> "Resourcesusagesofssystemscontainers" "allowssschedulingsofspodssonsasminionsaftersitsrejoinssthescluster" # file: resize_nodes.go, issue: #13258 "podsw/twosRWsPDssbothsmountedstosonescontainer,swritestosPD" # file: pd.go, issue: #15382

It("should run a job to completion when tasks sometimes fail and are locally restarted", func() { SkipIfProviderIs("gke") By("Creating a job") <del> // 50% chance of container success, local restarts. job := newTestJob("randomlySucceedOrFail", "rand-local", api.RestartPolicyOnFailure, parallelism, completions) </del> <ins> // One failure, then a success, local restarts. // We can't use the random failure approach used by the // non-local test below, because kubelet will throttle // frequently failing containers in a given pod, ramping // up to 5 minutes between restarts, making test timeouts // due to successive failures too likely with a reasonable // test timeout. job := newTestJob("failOnce", "fail-once-local", api.RestartPolicyOnFailure, parallelism, completions) </ins> job, err := createJob(f.Client, f.Namespace.Name, job) Expect(err).NotTo(HaveOccurred())

SkipIfProviderIs("gke") By("Creating a job") // 50% chance of container success, local restarts. <ins> // Can't use the failOnce approach because that relies // on an emptyDir, which is not preserved across new pods. // Worst case analysis: 15 failures, each taking 1 minute to // run due to some slowness, 1 in 2^15 chance of happening, // causing test flake.  Should be very rare. </ins> job := newTestJob("randomlySucceedOrFail", "rand-non-local", api.RestartPolicyNever, parallelism, completions) job, err := createJob(f.Client, f.Namespace.Name, job) Expect(err).NotTo(HaveOccurred())

}, Spec: api.PodSpec{ RestartPolicy: rPol, <ins> Volumes: []api.Volume{ { Name: "data", VolumeSource: api.VolumeSource{ EmptyDir: &api.EmptyDirVolumeSource{}, }, }, }, </ins> Containers: []api.Container{ { Name:    "c", Image:   "gcr.io/google_containers/busybox", Command: []string{}, <ins> VolumeMounts: []api.VolumeMount{ { MountPath: "/data", Name:      "data", }, }, </ins> }, }, },

// Bash's $RANDOM generates pseudorandom int in range 0 - 32767. // Dividing by 16384 gives roughly 50/50 chance of success. job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "exit $(( $RANDOM / 16384 ))"} <ins> case "failOnce": // Fail the first the container of the pod is run, and // succeed the second time. Checks for file on emptydir. // If present, succeed.  If not, create but fail. // Note that this cannot be used with RestartNever because // it always fails the first time for a pod. job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "if [[ -r /data/foo ]] ; then exit 0 ; else touch /data/foo ; exit 1 ; fi"} </ins> } return job }