adaptivecomputing/torque — pull request #288 by itkovian (2014-11-28 00:04:00+00:00)
Title: Use pgrep to make sure the correct pbs_mom process is identified

The problem addressed is the following. Occasionally, we are seeing service pbs_mom restart hanging with momctl unable to connect to a running pbs_mom. We are on EL6/EL7, and the offending code is located in contrib/init.d/pbs_mom.in +36.

The problem can be reproduced (on a RHEL based system):

~~~~
# service pbs_mom start
Starting TORQUE Mom: [ OK ]
# echo $$ > /var/spool/pbs/mom_priv/mom.lock
# pkill pbs_mom
# service pbs_mom stop
Shutting down TORQUE Mom: cannot connect to MOM on node 'localhost', errno=111 (Connection refused)
cannot connect to MOM on node 'localhost', errno=111 (Connection refused)
cannot connect to MOM on node 'localhost', errno=111 (Connection refused)
cannot connect to MOM on node 'localhost', errno=111 (Connection refused)
cannot connect to MOM on node 'localhost', errno=111 (Connection refused)

kill -0 $pid thinks there is a process with the given $pid (true), but it also assumes this is the pbs_mom process (false), and momctl cannot connect to it:

# momctl -d0
cannot connect to MOM on node 'localhost', errno=111 (Connection refused)
~~~~

So basically any sanity check that wants to restart the pbs_mom service is causing it to hang.