Change:
// setup host path directory to pass information between pod restarts func setupHostPathDirectory(job *batchv1.Job) { <del> if len(job.Spec.Template.Spec.NodeName) > 0 { </del> <ins> if _, nodeNameSpecified := job.Spec.Template.Spec.NodeSelector["kubernetes.io/hostname"]; nodeNameSpecified { </ins> randomDir := "/tmp/job-e2e/" + rand.String(10) hostPathType := v1.HostPathDirectoryOrCreate job.Spec.Template.Spec.Volumes[0].VolumeSource = v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: randomDir, Type: &hostPathType}}

Discussion:
pull-kubernetes-e2e-kind example: Example, but does not seem to be specific to the test: "Using a pod failure policy to not count some failures towards the backoffLimit". First time noticed, but potentially for long if this is very rare. The 3rd pod for the job could not get assigned resources by kubelet to start. See kubelet logs from kind-worker2: Relevant lines: Due to hitting the limit the pod is marked as by kubelet which is unexpected by the test. No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suppose the reason is that we preassigning the instead of asking scheduler to do so. This might be related to
FYI
we should not preassign, is that required by this test? Do you have a link to those tests?
What is required is that all pods run on the same node, because we use HostPath to communicate some information between recreations of a pod, and between replicas for indexed job. All tests in this file: , which use the function are affected I think. Here is the problematic place: I suppose using a instead of setting directly might already help, because it the decision will go via scheduler which will delay the scheduling until node has enough capacity.
FYI for input on the proposed solution to use or another alternative
hmm ,I don't like why this is this way , if the test need to be serialized it must be tagged as Serial, but making this assumptions on e2e tests is not great
Yeah, I would very much want to avoid making the tests serial. I can see two options: term - replace the with . IIUC, this will already keep the pods unscheduled until there is capacity. With the timeout of 15min as we have for job it is very unlikely the pod would wait that long. Currently, it gets assigned but fails. the tests so that the state is kept on a mounted volume obtained by PVC.

This should eliminate the need for the pods to run on the same node. Since 2. is slightly more involving, it might be good to have 1. merged.
you can not assume the e2e test will have dynamic PV working, if you want to assume that you have to tag the test so jobs can filter based on the tag
Let's start with the NodeSelector
Actually, that wouldn't help, because currently the Job controller just recreates the Pod. With a node selector, it would just be scheduler that is now "busy-waiting" and we would still hit a timeout. Can the implementation be changed to save the information in a ConfigMap? It probably requires a specialized container image. Unless a ConfigMap can be mounted as readwrite?
Interesting, so I greped the logs for and only found this pod in this line: Oct 24 07:42:50 kind-worker2 kubelet[248]: I1024 07:42:50. 248 ] "Event occurred" object="job-712/pod-disruption-failure-ignore-3-mpvmm" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="OutOfpods" message="Node didn't have enough resource: pods, requested: 1, used: 110, capacity: 110" Which should not be that bad, because we have backoffLimit=6, yet the job failed with . So, maybe some other failures were incrementing the counter wrongly. I don't know at the moment.
My mistake here. The job has actually : This explains why the job failed with BackoffLimitExceeded after seeing failing with admission error by kubelet. Meaning 137 exit code, but without DisruptionTarget condition. So it was counted as failed and killed the job. So, I think in this case adding the would actually help. It would not help, if was for a prolonged amount of time to cause test timeout. However, the test timeout is 15min. Also, there is no indication that kubelet was in the time for long. A single event was enough to cause the pod fail with 137, and this single failure caused the job failure.

As for long term solution using config maps - yes, it could work, but I think it's better to do a quick fix to minimize the flakes before release.
I agree. Thanks for investigating!