LP1204123 Set SIGCHLD handler to DEFAULT for children (perl)

Remove the special SIGCHLD handler applied to the parent/listener
process from spawned child processes (drones) by setting child handler to
DEFAULT.  Otherwise, if an OpenSRF drone process spawns a sub-process,
the drone will execute the SIGCHLD handler meant for the Listener when
the sub-process dies, wherein hijinks ensue.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 SIGTERM causes graceful shutdown (Perl)

When sent the SIGTERM signal, listener processes will wait for all child
processes to complete their currently running tasks before killing all
child processes and exiting.

SIGQUIT and SIGINT can still be used for semi-graceful shutdowns, where
the listener de-registers and cleans up child processes, but does not
wait for child processes to finish their tasks.

To kill with fire, SIGKILL is still your friend.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 SIGUSR1 causes router unregister (Perl/C)

$ kill -s USR1 <listener_pid>

The listener process will unregister all router connections.  This
allows services to remove themselves from consideration for new
inbound requests while allowing their drone processes to complete
any currently running tasks.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 SIGTERM graceful shutdown (C)

When a TERM signal is received, the listener parent process will wait
for all child processes to complete any active tasks before killing the
child processes and shutting down.

Receiving a SIGINT or SIGQUIT results in a non-graceful shutdown.

In all cases, the system first un-registers with all routers.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 opensrf.settings reload file option

opensrf.settings API calls

opensrf.settings.host_config.get
opensrf.settings.default_config.get

now sport an additional "reload" boolean parameter.  When set, the settings
file (opensrf.xml) will be reloaded from the file system to pick up any
changes which have occurred since the last load/reload.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 SIGUSR2 causes router re-register (Perl/C)

When a listener process receives the USR2 signal, the 'register' command
is sent to all known routers.  This allows for the addition of new
routers to running opensrf services.

If a register command is sent to a router for which the process is
already successfully register, the router will safely discard it, so
multiple SIGUSR2 calls are safe.

Minor int => bool clarification repair to existing C shutdown
(prefork_clear) code.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 Don't treat interrupts as errors

Interrupted calls to select() are generally not errors, but more likely
signals received by the process.  Avoid setting the transport_client
error flag to true in these cases.  Otherwise, after one signal is
received, no other outbound communication can occur on the client
connection.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 osrf_ctl.sh supports sending signals

Using the following new options to osrf_ctl.sh, the script can now send
signals to any/all OpenSRF Listener processes:

-a signal / signal_all
-k <signal> [value passed to 'kill -s']
-s <service>

With an action of 'signal' and service provided by -s, the specified
signal is sent only to the listener process of the service.

With an action of 'signal_all', the specified signal is sent to all
listener prcesses.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 Improved per-service control (C/Perl)

C opensrf no longer mantains a single master process.  Instead, like
Perl, each Listener process writes its own PID file and can be managed
individually.  Related to this, much code was cleaned up in
osrf_system.c.

osrf_ctl.sh now has the ability to stop/start individual services for
Perl and C, using the new -s <service> option.  Examples:

osrf_ctl.sh -l -a restart_c -s opensrf.math
osrf_ctl.sh -l -a restart_perl -s opensrf.settings

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 Remove osrf_ctl debug echos

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 opensrf-perl.pl expansion / replace osrf_ctl

opensrf-perl.pl can now stop/start/signal/etc. all opensrf services,
regardless of language.  It is in effect a replacement for osrf_ctl.sh
with considerably broader powers.

To ease transition, osrf_ctl.sh has been taught to pass
start/stop/restart_all commands through to opensrf-perl.pl.

The layout of commands has changed some (e.g. --start-all instead of
--action start_all) and a host of new commands have been added.

Below are the full set of commands.  See --help for full descriptions:

--config
--pid-dir
--settings-startup-pause
--localhost
--service
--verbose
--no-daemon
--help
--start-all
--start
--restart-all
--restart
--stop-all
--stop
--graceful-shutdown-all
--graceful-shutdown
--fast-shutdown-all
--fast-shutdown
--immediate-shutdown-all
--immediate-shutdown
--kill-with-fire
--signal-all
--signal
--signal-timeout

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 opensrf-perl.pl expansion continued

* improved cleanup of stale pid files during startup
* added --force-clean-process option to kill orphan (no pidfile)
  processes during startup
* other minor cleanup

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 opensrf-perl.pl improve logging

Log messages by default.  Replace --verbose flag with --quiet flag.

Also treat all messages the same, instead of separating them into info
and debug.  They're all pretty useful.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 install opensrf-perl.pl as osrf_control

Symlink osrf_control to opensrf-perl.pl to act as the replacement for
osrf_ctl.sh.  I chose "osrf_control" since it matches more closely to
"osrf_config" -- also it's probably less confusing than calling it
osrf_ctl, which would likely get confused w/ osrf_ctl.sh

TODO: documentation

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 osrf_control improved sample usage

Use a more realistic default usage message.  --config and --pid-dir have
sane defaults, so people will not normally need to specify those.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 more osrf_ctl/control script cleanup

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 more osrf_ctl/control script cleanup

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
C SIGHUP handling and config reloading

Sending the HUP signal to a C Listener process now results in the
following:

 * Reload the opensrf_core config
 * update the log file level
 * kill idle child processes
 * child processes that are active when the signal is received are
   tracked and killed once they become idle.
 * New children are spawned per the min child settings

The primary use case for these changes is temporarily changing the log
level for a given service for debug purposes.  It may also
be used, for example, to gracefully recover excess RAM consumed by a
child process.

opensrf.xml (opensrf.settings config) is not reloaded, so min/max child
settings will not be affected.

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 osrf_control doc updates

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
LP1204123 osrf_ctl.sh improvement

Removing no-longer-supported --verbose flag from osrf_ctl.sh stub

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>