Cherry-pick nagios 7a5c874, which makes sure we don't forget output

Original commit message:

Really fixed bug #484 - Output from checks not getting back into Core

There was a race condition in the code that gathered the output from
checks where if the stderr was read first, stdout would never be read.
This commit fixes that condition.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
Change-Id: I292564157dbe9bfa822d7439d905f596ac25a413
test_checks: disable test case for service state type transition

Rationale (also as inline comment):

Since we now (as per a466b8f75d19819ca329069b555ea8c3fb872717) use only
asynchronous checks, we cannot get the routing result necessary to
determine the real state of the host before we're done evaluating the
service check result. This means that we'll always run max_attempts
checks for a service before we set it in a hard state.

The solution proposed to mitigate this issue is to add some kind of
trigger to checks, such that (in this case) the scheduled host check,
when finished would update the state type of the service appropriately.

Change-Id: I3bd7208118c1ddb353793d3f3e2e950d61e2ec3b
Signed-off-by: Anton Lofgren <alofgren@op5.com>
checks: don't require neb modules to redo service alert calculations

It's pretty unlikely that they want to do something different (although
that's still possible.) Nebmods that want to report/record host and
service alerts can now do so, rather effortlessly.

This patch adds NEBATTR_CHECK_ALERT to signify alert status of a check.

This allows us to avoid duplicating alert controls for modules which
just want to record alert information on their own.

Unfortunately, this requires some ABI breakage (in checks:handle_host_status)
which for some unknown reason is not declared static, despite not being
used anywhere outside the compilation unit, in order to gain access to the
alert_recorded information. It would be _possible_ to work around this
by recording the information to a global instead, but I really feel we
have enough of those.

Change-Id: I2667d093611eb1e8a91953d4713cc68c6138076d
Signed-off-by: Anton Lofgren <alofgren@op5.com>
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
Allow contactgroups without members but having contactgroup_members

This doesn't make any sense actually. First if the members-string
isn't set, skip. If it still isn't set, and another string isn't set,
skip. Later, do some stuff, but not with the members-string, and if
it still isn't set, skip...

A contact group can have members in three ways:
- contacts, through the members attribute in the contactgroup
  definition
- contacts, through the contactgroup attribute in the contact
  definition
- contactgroups, through the contactgroup_members definition in the
  contactgroup definition.

Bailing out, just by not having "members", in the case where
contactgroup_members is resolved doesn't make much sense at all...

This solves the issue of nested contact groups doesn't work for
contactgroups not having members set.

Resolves the nagios part of op5#7958

Change-Id: I7a1d6615a98cef2024b8dc1df68fb4ad10884397
Signed-off-by: Max Sikstrom <msikstrom@op5.com>
test_checks: NULL-initialize check result structure

Previously, engine was left uninitialized, which seems to work out fine
in practice when building on centos6, but x86_32 on centos5 crashed a
lot.

Valgrind exposed a bunch of "Conditional jump or move depends on
uninitialised value(s)", a value which this small patch simply
initializes.

Change-Id: I0c050d1ec1d6a97e06a3884c7b4e2f7123805f65
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
test_macros: Verify escaping of HOSTCHECKCOMMAND

We need to be able to STRIP_ILLEGAL_MACRO_CHARS for check command
macros. This patch verifies that we are, as implemented in f3a7f6d2

This verifies op5#7762.

Change-Id: I230dd230271744f32eb0556871390402e69b94dd
Signed-off-by: Anton Lofgren <alofgren@op5.com>
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
worker: De-nest worker killing code

There's only really one system call that needs to loop - no need to put
a return in the middle of an if statement inside a loop that covers half
of the rest of the function.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
nsutils: Stop with timediff nonsense

Someone, who'd painted themselves into a corner, decided that the proper
solution was to paint more brightly, very, very close to the corner.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
downtime.c: Fix a couple of malloc argument errors

What should have been allocated was an unsigned long, as that's what the
returned value was cast to. What was allocated was a pointer to an
unsigned long. This might be the same number of bytes, or it might not -
it's either way not the same thing.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
downtime: Handle malloc errors correctly

We did take a return value, but then we proceeded to overwrite it in
case of problems. Oh well.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
worker: Get event runtime from the event runtime

Rather than trying to reconstruct it based on incomplete sources,
actually read the correct information at once.

"Incomplete" in this case means we never realized we'd tried to move the
event a few seconds forward, and just spammed the user over and over
with "failed to reap" messages.

This fixes bug op5#8112. Actually.

Reproducing said bug is a matter of using the following plugin:
  if [ $1 -lt 5 ]; then
          bash $0 $(($1 + 1))
  fi
  nc -l 8000
for any check, and simply wait 60 seconds for it to run.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
objects: Fix object cache for host flap, stalk

The character for the UP state in both flap_detection_options and
stalking_options is 'o' - 'u' means unreachable. Nagios reads this
correctly, but saves the object.cache incorrectly. This makes CGIs
look wrong, but is even more serious in combination with merlin, which
uses the object.cache, not the config, to push config to pollers. This
makes flap detection and stalking completely broken for poller systems.

Hence this fixes bug op5#8049.

Sidenote: I'm getting sick of these "we generated config we couldn't
read" bugs.

Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
commands: Disconnect command worker during restart

The command worker lives on when nagios restarts. This is quite nice -
your code can keep spewing in passive checkresults and whathaveyou,
even when nagios isn't actively monitoring.

"But what happens with those check results?", one might ask. One would,
in fact, be asking a very relevant question. Before this patch, they
would be submitted while nagios waits for workers to connect, but before
completely loading the configuration, leading to bug op5#8169 where nagios
would segfault. After this, the core process simply ignores the command
worker until the time comes when it would, during startup, be starting
the new command worker, after which it reads any incoming commands and
processes them ASAP. No more segfaults for you, fixing bug op5#8169.

This could be enhanced further, by letting the command worker buffer
incoming commands explicitly - instead, it simply lets the kernel buffer
them in it's tiny-but-for-a-few-seconds-hopefully-sufficient buffer.

As this is a race, it's hard to reproduce - I added a sleep(10) at the
bottom of the main loop in nagios.c while testing.

Change-Id: I5c0e6fe936c76ec44c7891190e8c5048c5319ee4
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
workers: Log wproc debug info as debug info

When checks or other commands result in misc errors, this used to be
logged extremely verbosely by the worker, and then with reasonable
granularity by each of the job types themselves.

Before:

[1386751372] Warning: Check of service 'spam' on host 'monitor' timed
out after 60.009s!
[1386751672] wproc: Core Worker 5257: job 462 (pid=24856) timed out.
Killing it
[1386751672] wproc: CHECK job 462 from worker Core Worker 5257 timed out
after 60.01s
[1386751672] wproc:   command: /opt/plugins/check_spam.sh
[1386751672] wproc:   host=monitor; service=spam;
[1386751672] wproc:   early_timeout=1; exited_ok=0; wait_status=0;
error_code=62;
[1386751672] wproc:   stderr line 01: /opt/plugins/check_spam.sh: line
3: [: -lt: unary operator expected

After:

[1386777436] Warning: Check of host 'monitor' timed out after 60.03
seconds

The removed information goes to the debug log with the IPC debug level,
except for the completely redundant "job $number (pid=$number) timed
out" line (repeating the information from both the line above /and/
below), which is simply removed.

Change-Id: I2fe9769beeb961a0f6659054573175e0d910322e
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>
runcmd: Don't crash when exposed to ampersands

An off-by-one error caused us to copy both a null-byte and the job
control character when exposed to commands containing job control
characters. If there are more than 10 of those, we'd run out of memory
and write to uninitialized memory. If we were unlucky, the uninitialized
memory was "badly" located, which means worker crash and bug op5#8182.

The fun thing is, the way the "embedded shell" thingie works, every time
you call add_ret(), you might as well just return - alas, we don't, for
reasons that are hard to see clearly but I assume involve having
something to speed up in a future version. Oh well.

So, back to the bug: we discovered a case where knew we would never,
ever read the output, and proceeded to handle that by... Writing too
much to the output to the point where the worker crashed.

Proving, with a test, that we didn't really do anything is hard. The
extra test case added to test-runcmd doesn't fail or generate a crash
with the old code, but at least it generates memory errors when run
through valgrind, and those errors are now gone.

So, this should fix the crash in bug op5#8182, when using the initial,
incorrect quoting - obviously, the check result still won't make sense
as it's wrong, but now you get to see that rather than infinite crashes.

Change-Id: I0b7ebd9a782125bc060ea9eb785d70de7bb69228
Signed-off-by: Robin Sonefors <robin.sonefors@op5.com>