Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete. First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1..4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs .... However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway... Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me? if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).

Discussion:
if IsJobFinished(&job) { return nil } <ins> var manageJobErr error </ins> if pastActiveDeadline(&job) { // TODO: below code should be replaced with pod termination resulting in // pod failures, rather than killing pods. Unfortunately none such solution

// some sort of solution to above problem. // kill remaining active pods wait := sync.WaitGroup{} <ins> errCh := make(chan error, int(active)) </ins> wait.Add(int(active)) for i := int32(0); i < active; i++ { go func(ix int32) { defer wait.Done() if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name, &job); err != nil { defer utilruntime.HandleError(err) <ins> glog.V(2).Infof("Failed to delete %v, job %q/%q deadline exceeded", activePods[ix].Name, job.Namespace, job.Name) errCh <- err </ins> } }(i) } wait.Wait() <ins> select { case manageJobErr = <-errCh: if manageJobErr != nil { break } default: } </ins> // update status values accordingly failed += active active = 0

jm.recorder.Event(&job, v1.EventTypeNormal, "DeadlineExceeded", "Job was active longer than specified deadline") } else { if jobNeedsSync && job.DeletionTimestamp == nil { <del> active = jm.manageJob(activePods, succeeded, &job) </del> <ins> active, manageJobErr = jm.manageJob(activePods, succeeded, &job) </ins> } completions := succeeded complete := false

return err } } <del> return nil </del> <ins> return manageJobErr </ins> } // pastActiveDeadline checks if job has ActiveDeadlineSeconds field set and if it is exceeded.

// manageJob is the core method responsible for managing the number of running // pods according to what is specified in the job.Spec. // Does NOT modify <activePods>. <del> func (jm *JobController) manageJob(activePods []*v1.Pod, succeeded int32, job *batch.Job) int32 { </del> <ins> func (jm *JobController) manageJob(activePods []*v1.Pod, succeeded int32, job *batch.Job) (int32, error) { </ins> var activeLock sync.Mutex active := int32(len(activePods)) parallelism := *job.Spec.Parallelism jobKey, err := controller.KeyFunc(job) if err != nil { utilruntime.HandleError(fmt.Errorf("Couldn't get key for job %#v: %v", job, err)) <del> return 0 </del> <ins> return 0, nil </ins> } <ins> var errCh chan error </ins> if active > parallelism { diff := active - parallelism <ins> errCh = make(chan error, diff) </ins> jm.expectations.ExpectDeletions(jobKey, int(diff)) glog.V(4).Infof("Too many pods running job %q, need %d, deleting %d", jobKey, parallelism, diff) // Sort the pods in the order such that not-ready < ready, unscheduled

if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name, job); err != nil { defer utilruntime.HandleError(err) // Decrement the expected number of deletes because the informer won't observe this deletion <ins> glog.V(2).Infof("Failed to delete %v, decrementing expectations for job %q/%q", activePods[ix].Name, job.Namespace, job.Name) </ins> jm.expectations.DeletionObserved(jobKey) activeLock.Lock() active++ activeLock.Unlock() <ins> errCh <- err </ins> } }(i) }

diff = 0 } jm.expectations.ExpectCreations(jobKey, int(diff)) <ins> errCh = make(chan error, diff) </ins> glog.V(4).Infof("Too few pods running job %q, need %d, creating %d", jobKey, wantActive, diff) active += diff

if err := jm.podControl.CreatePodsWithControllerRef(job.Namespace, &job.Spec.Template, job, newControllerRef(job)); err != nil { defer utilruntime.HandleError(err) // Decrement the expected number of creates because the informer won't observe this pod <ins> glog.V(2).Infof("Failed creation, decrementing expectations for job %q/%q", job.Namespace, job.Name) </ins> jm.expectations.CreationObserved(jobKey) activeLock.Lock() active-- activeLock.Unlock() <ins> errCh <- err </ins> } }() } wait.Wait() } <del> return active </del> <ins> select { case err := <-errCh: // all errors have been reported before, we only need to inform the controller that there was an error and it should re-try this job once more next time. if err != nil { return active, err } default: } return active, nil </ins> } func (jm *JobController) updateJobStatus(job *batch.Job) error {

// run err := manager.syncJob(getKey(job, t)) <del> if err != nil { t.Errorf("%s: unexpected error when syncing jobs %v", name, err) </del> <ins> // We need requeue syncJob task if podController error if tc.podControllerError != nil { if err == nil { t.Errorf("%s: Syncing jobs would return error when podController exception", name) } } else { if err != nil { t.Errorf("%s: unexpected error when syncing jobs %v", name, err) } </ins> } // validate created/deleted pods