xdata/xodtemplate: Fix processing of servicelists for serviceextinfo

This patch fixes two issues with service lookups for serviceextinfo
objects.

Firstly, we fix parameters to xodtemplate_create_service_list(),
which were previously warped so that we shipped the hostgroup_name
as the host_name and vice versa.

Secondly, we reset the master_service_list and service_map for each
iteration to make sure the serviceextinfo isn't set for more objects
than it should be. This second part only matters if there's more
than one serviceextinfo object (which is usually the case if one
uses such deprecated constructs).

Even though serviceextinfo objects are deprecated, we should still
handle them properly if we encounter them.

Original-patch-by: Ricardo Maraschini <ricardo.maraschini@gmail.com>
Commit-message-by: Andreas Ericsson <ae@op5.se>
Signed-off-by: Ricardo Maraschini <ricardo.maraschini@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
xdata/xodtemplate: Clear the service bitmap between each serviceextinfo

Without this patch it would be impossible to assign multiple extinfo
objects to the same service, which was possible (although never
officially supported, since it wasn't documented) in Nagios 3.x.

Since it's (marginally) useful and the patch is literally a oneliner
that makes the extinfo code consistent with how escalations and
dependencies work, we may as well put it in (and if "That's what she
said", you need to find yourself a somewhat more passionate partner).

Signed-off-by: Andreas Ericsson <ae@op5.se>
xdata/xodtemplate: Fix the "not enough data" check for serviceextinfo

Since serviceextinfo objects need service_description and either
hostgroup_name or host_name, we should check for just that when
determining if we have enough data or not.

I'm not exactly sure what that check is doing there, since we will
fail with a hard error a few lines later if we can't find the extinfo
targets anyway, but I guess there's some perverted sense of continuity
in perpetuating insane legacies. Besides, people will probably whine
if their configs suddenly stop working, and I really don't want to
deal with that particular grief.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/runcmd: Create a separate process group when spawning jobs

This has no major impact on performance, but it does mean that a
worker can control and wait() for any and all programs spawned by
whatever processes it fires up to do the actual work.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Utilize process groups when controlling and reaping jobs

When a job exits nicely (as should be the most common thing), we
don't do any extra work at all. When we're forced to kill one
though, we'll signal all processes in the child's process group
and then attempt to reap all those children in order to avoid a
buildup of zombies.

We use a 5ms busy-loop to reap the children, where each failure to
wait() that doesn't trigger an error (such as ECHILD) is considered
worthy of a 1ms sleep and a retry.

Note that this is a best-effort thing however. Processes stuck in
uninterruptable I/O will still, in all likelihood, become zombies.

Signed-off-by: Andreas Ericsson <ae@op5.se>
xdata/xodtemplate: Allow serviceescalations on servicegroups

It used to be allowed (I think), but the check that made us continue
on "not enough data" didn't check if servicegroup_name was set, so
every escalation had to have service_description and either host_name,
hostgroup_name or both.

Signed-off-by: Andreas Ericsson <ae@op5.se>
worker result parsing: Fix typo in gperf input file

s/error_message/error_msg/

Signed-off-by: Andreas Ericsson <ae@op5.se>
daemon: Make query socket path main-config-relative

This just brings it in line with all the other path-variables and
makes testing a little bit easier.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Close input filedescriptors more aggressively

Previously it was possible to end up in the infinite loop of doom
between check_completion() and gather_output() (since the former
calls the latter and vice versa), leading to a coredump when we run
out of stack space. With this patch, that really shouldn't happen,
since we'll now close the offending filedescriptor before we call
out to check_completion() from gather_output(), which means we'll
at most be called upon once to get the final output from each
filedescriptor.

We also avoid calling wait4() without WNOHANG, as that could otherwise
lead us to wait forever in case plugins end up in uninterruptible sleep.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Maintain a kill-queue of processes we fail to reap

Previously it was possible for us to send SIGKILL to our children
and then attempt to reap the child before it had received the signal,
which caused the process to become a zombie.

If we indeed do fail to reap the process after killing it, we'll try
again a few seconds later, and keep on trying every few seconds
until we've either successfully reaped the child, it goes away by
some other means (such as being traced and having its parent
altered), or we get shut down, in which case it's init's problem.

This covers scenarios where the child is stuck in uninterruptable
sleep (such as after a vfork() call, f.e.), but not those where the
plugin itself alters its own process group after its inception.

Signed-off-by: Andreas Ericsson <ae@op5.se>
Avoid crashing when freeing objects after parsing faulty configuration

When we get bad serviceescalations or other entries where we had
already counted the host and servicedependencies but not sorted
them into their list yet, we would go ahead and shut down, calling
the free_object_data() function which would happily try to access
entries in hostdependency_ary and servicedependency_ary, which
caused the crashes mentioned in the subject line.

This patch fixes that particular corner case.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Fix typo in scheduling reaping for jobs

Not quite sure what I was thinking there, but mea culpa for sure.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Fix typo in worker handling code

When a helper is terminated by a signal, we must use WTERMSIG to
find out which signal it was. Previously we used WSTOPSIG, which
is really only useful if the child received SIGSTOP and is as
such nowhere near as useful.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Reduce logging when reaping dormant jobs

We were quite spammy earlier. Since logmessages go through the socket
we don't want to be quite so spammy.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Revamp reap timing somewhat for stale jobs

It's easy to artificially create a scenario where our immediate child
goes into uninterruptible sleep with some simple but clearly bananas
programming. Our immediate child, hereafter called "helper", must issue
the vfork() system call and its child may spawn a ton of offspring of
its own if it so wishes, and then go into sleep. If the child's
offspring (the helper's grandchildren) exit, they will become zombies
since the child doesn't wait() for them. The helper on the other hand
is stuck in uninterruptible sleep until its child has called either
exit() or execve(). If it instead calls sleep() the helper can get
stuck in uninterruptible sleep (ps state 'D') for quite some time.

The first time we attempt to reap such a job, the child and all the
childs' children will get the SIGKILL signal and, (unless they too
are stuck in uninterruptible sleep), exit promptly, but not get
wait()ed for, turning them into zombies. The helper on the other hand
will either get its signal queued for when it wakes up (on Linux) or
get its signal discarded (possibly other unices).

The helper will wake up when its immediate child dies, so apparently
that counts as "calling exit", but it will normally not have time to
wake up and smell the signal before the CPU is handed back to the
signalling process (ie, our worker). In such cases the helper will
be ready for reaping very, very soon, but not immediately. When that's
the case, we schedule a 0.25 second sleep before the first attempt at
reaping the check, and then try again every 5 seconds until we've
managed it.

In case the signal is lost instead of queued, that means we'll at
least have killed the helper 0.25 seconds after we get the code back,
and the worst case scenario is that the signal doesn't register until
after we go past the state where we check if the child has exited. In
such cases, we'll leave zombies hanging around for 5 seconds.

The scenario really should be very rare though, and especially on linux
where signals are delivered atomically.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Add gcc printf attribute to wlog() function

It helps to get the warning if you miss an argument or some such.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: check for completion before we send SIGKILL

Sometimes helpers complete their work just before we attempt to reap
them, in which case the SIGKILL goes to a zombie and we're missing out
on the last chance to reap the actual result of the job instead.

This patch corrects that by forcing one last attempt at properly
reaping the child in case we're about to kill it.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Try reaping processes when either I/O channel shuts down

It makes sense to do this, since some systems (notably Linux 3.8.0-25
from the Ubuntu repos with some wildly inappropriate sysctl settings)
fail to notify the parent process of stdout and stderr closing for
the child in case they've either had no output or use _exit() instead
of the more appropriate exit().

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Catch SIGCHLD and schedule a "general" reaping when we do

This means we need a quick way to access the job we just reaped, so
we also add and maintain a fanout table for that precise purpose.

Since we don't expect to run more than a few hundred jobs at the same
time (at worst, that is), we use a relatively small one of 4096 entries
which seems to suffice just fine for our needs.

This appears to eliminate the "checks sometimes reach timeout without
being reaped" more or less completely, unless those checks actually
*do* reach their timeout.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Unify job-state logging somewhat

We still only log exceptional states, but now we do it so that grep'ing
becomes a bit easier and the job id lines up properly with the pid
when viewing lists of the error messages.

Signed-off-by: Andreas Ericsson <ae@op5.se>