openlava/openlava — Issue #27: Jobs pending with "slot limit reached" when slots are free
Author: cansmith
Created: 2012-02-22 20:50:18+00:00
State: open

I saw this issue today when testing out some rusage scenarios. Somebody else mentioned this to me recently as well, but I hadn't observed it until today. I've added lsid output to show the version, but it's somewhere on the master branch. There is nothing in the mbatchd log file that indicates any kind of error or issue. After I do 'badmin reconfig' (and change nothing else) the problem goes away and the jobs get dispatched as expected.

plato:~> lsid
openlava project 2.0, Jan 5 2012

My cluster name is openlava
My master name is plato
plato:~> bqueues -l

QUEUE: normal
-- For normal low priority jobs, running only if hosts are lightly loaded. This is the default queue.

PARAMETERS/STATISTICS
PRIO NICE STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SSUSP USUSP RSV
30 20 Open:Active - - - - 2 2 0 0 0 0
Interval for a host to accept two jobs is 0 seconds

SCHEDULING PARAMETERS
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -

USERS: all users
HOSTS: all hosts used by the LSF Batch system

plato:~> bhosts
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
node1 ok - 1 0 0 0 0 0
node2 ok - 1 0 0 0 0 0
node3 ok - 1 0 0 0 0 0
node4 ok - 1 0 0 0 0 0
plato ok - 2 0 0 0 0 0
plato:~> bhosts -l node1
HOST node1
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
ok 100.00 - 1 0 0 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
Total 0.0 0.0 0.0 0% 0.0 12 0 28144 16G 451M 410M
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 0M

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -

plato:~> bjobs -l

Job <203>, User , Project , Status , Queue , Com
mand
Wed Feb 22 12:34:40: Submitted from host , CWD <$HOME>, Output File , Requested Resources , Specified
Hosts ;
PENDING REASONS:
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;

SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -
------------------------------------------------------------------------------

Job <204>, User , Project , Status , Queue , Com
mand
Wed Feb 22 12:34:47: Submitted from host , CWD <$HOME>, Output File , Requested Resources , Specified
Hosts ;
PENDING REASONS:
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;
Job slot limit reached;

SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -

plato:~> bparams -l

System default queues for automatic queue selection:
DEFAULT_QUEUE = normal

The interval for dispatching jobs by master batch daemon:
MBD_SLEEP_TIME = 10 (seconds)

The interval for checking jobs by slave batch daemon:
SBD_SLEEP_TIME = 7 (seconds)

The interval for a host to accept two batch jobs:
JOB_ACCEPT_INTERVAL = 0 (* MBD_SLEEP_TIME)

The idle time of a host for resuming pg suspended jobs:
PG_SUSP_IT = 180 (seconds)

The amount of time during which finished jobs are kept in core:
CLEAN_PERIOD = 3600 (seconds)

The maximum number of finished jobs that can be stored in current events file:
MAX_JOB_NUM = 1000

The maximum number of retries for reaching a slave batch daemon:
MAX_SBD_FAIL = 3

The default project assigned to jobs.
DEFAULT_PROJECT = default

The interval to terminate a job.
JOB_TERMINATE_INTERVAL = 10

The maximum number of jobs in a job array.
MAX_JOB_ARRAY_SIZE = 1000

User level account mapping for remote jobs is disabled.

The batch jobs' temporary output directory:
JOB_SPOOL_DIR = /cfs/data/openlava_spool

Static shared resource update interval for the cluster:
SHARED_RESOURCE_UPDATE_FACTOR = 2147483647

The Maximum JobId defined in the system:
MAX_JOBID = 999999