core: Send notifications through workers

Notifications take a significant amount of time, most of which used to
stall Nagios completely. With this change, we get rid of that latency-
inducing behaviour and paw off notifications to be run by one of our
command execution workers.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/iocache: Add iocache_reset() and use it where sensible

It's nifty to be able to reset an I/O cache structure without having
to release and re-create all its memory. This patch allows us to do
just that, and also implements it where sensible.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/iocache: Add functions iocache_{add,write,send,sendto}()

It's nifty to be able to cache output in memory before flushing it,
in order to avoid tons of small writes. This patch does just that,
although testing is currently lacking.

We're not really using these functions anywhere right now, but they've
been sitting in my working tree far too long not to be committed now.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Replace worker response code parsing with a perfect hash

Courtesy of gperf, we now have a very small, very fast perfect hash
function that we can paw off the individual keys from the worker
responses to, allowing us to get a numerical value for them in constant
time, which lets us do response code parsing about 15 times faster
than we could earlier.

Since this happens once for every variable returned from the worker
for every check run, that means we'll most likely run this function
several hundred million times per hour, so the saving will end up
being significant in the long run.

The downside is that altering the response variable keys requires
fiddling with gperf.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Make DEBUGL_IPC and DEBUGL_SCHEDULING proper debug levels

They used to share an ID with checks, which generate a lot of
irrelevant data. Since the ipc we're doing now is a lot more important
we need it to have its own debug level in order to make sense of any
problems related to it.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Remove embedded perl

For several years it's been the most complex, the buggiest and the
least tested part of Nagios, and people experiencing problems with
the Nagios core have always gotten the suggestion to disable the
embedded perl interpreter and its broken caching, and it's been
all but impossible to test its functionality with all the different
versions of Perl available.

Besides; Since we run checks through workers now instead of forking
directly and larger processes cause slower fork()'s, it's likely we
already have better check performance than adding X MiB's of heap
memory that we'd have to earmark for caching pre-compiled perl
plugins if we were to support it again.

A much more feasible way forward is to use a module to distribute
certain checks to a specialized worker. Using the newly introduced
libnagios, it shouldn't take very long to write such a one.

Signed-off-by: Andreas Ericsson <ae@op5.se>
configure: Re-generate with embedded perl dropped

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: fsync() files before we close them

Otherwise we just setup the errno flag to contain EBADFS, meaning
future debugging calls to strerror() or the %m allocator in strings
become erroneous.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Remove weird switch() statement when scanning check result queue

First of all, we use stat(2) instead of lstat(2), so the supposed check
for symlinks is bogus. If it was a symlink, we'd either get ENOENT or
the stat info of the actual file.

It's also wrong to check st.st_mode & S_IFMT against S_IFREG and then
checking it again with the macro S_ISREG(), since they'll both always
evaluate to true.

Instead, we remove the case statement and only use a simple S_ISREG()
and ignore everything but ordinary files.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Fix deleting too old check result files

Even under pretty normal circumstances, the check result spool dir
can fill up with a tremendous amount of check result files, which kills
Nagios' performance completely.

The problem is reloads, where old checks may be abandoned in case
they take too long to finish. In that case, half the check result file
is stashed in the spool directory (the other half is only written as
the check returns). With a huge amount of checks and semi-frequent
restarts, the checks will start to accumulate and Nagios will spend
more and more time scanning a huge directory of files where very few of
the check result files have ".ok" files accompanying them, leading to
a ton of cache-misses when we try to stat() the ".ok" file.

This patch fixes it by using the mtime from the stat call earlier in
the chain so even check results without an ".ok" file can be deleted.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Process check results ad-hoc

First relegating them to an in-memory list is just an unnecessary
extra step that we can avoid entirely by just passing the parsed
check results to their respective handlers as soon as we encounter
them. While we might encounter occasional long timeouts while reaping
checks (in case they cause notifications or other helper programs to
be spawned), we will save some cpu time overall on this change, and
some extra memory as well.

Also, we avoid the whole problem of potentially reaping more checks
than we have time to process later, which could cause a buildup of
check result data in memory which was possible (although very unlikely)
with the old code.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/iobroker: Add iobroker_is_registered(fd)

As per the questioning name, it returns 1 iff fd is registered and
0 in all other cases.

This is sort of a debug function, since callers should really keep
track of which descriptors they've registered with the broker, but
since I keep tip-toeing around it when committing changes it might
as well be a permanent fixture of the broker.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/iobroker: Make iobroker_poll() return number of triggered fd's

This makes it possible to do some simple verification that we're
getting some sort of input from something registered with the broker.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Make send_kvvec() return what send() returned

Doing so makes it possible to keep track of and report write errors,
which is naturally crucial, but also helps us keep track of how much
data we've sent, which is really nifty to know when debugging.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/kvvec: Add possibility to avoid multiple alloc()/free() calls

It's quite simple to allow callers to provide a pointer to their
own previously allocated key/value vector, and this allows them to
avoid a large amount of needless memory munging, so it proves to be
quite a useful performance improvement as well.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/kvvec: Use assignment rather than copying in buf2kvvec()

This saves an enormous amount of work, since we no longer have to
get fresh memory for each check result reported in by the workers.
Since new memory would inevitably be located in a different page
than the I/O cache buffer we've got allocated for the worker, that
means we can now avoid at least one cache-line fill and a (relatively)
huge constant overhead of check result reception.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Avoid leaking filedescriptors on worker creation errors

This amounts to adding a few close() calls and re-arranging some code.
With the patch it's obviously correct.

Signed-off-by: Andreas Ericsson <ae@op5.se>
lib/worker: Add set_socket_options()

It's handy to disallow sockets to spill over into plugins, and it's
also nifty to beef up the send and receive buffers somewhat, so we
do just that.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Retain workers between reloads

Respawning them for reloads makes no sense, since it just means we'll
lose a lot of half-completed checks for no good reason.

While we're at it, we fix worker process memory management so they
can safely release each other's memory.

Signed-off-by: Andreas Ericsson <ae@op5.se>
core: Use a command-file worker

The only thread remaining (since a looong time back) is that for
artificially increasing the size of the external command buffer.
By replacing it with a process, we can get rid of threading (still
on the todo-list) entirely.

On sane systems, we should be able to add the command pipe directly
to the iobroker_set to be polled by the main process, but it appears
OSX (and thus FreeBSD) have crap support for polling filesystem pipes
along with sockets in a single. Solaris and Irix appear to have some
similar issues, according to some basic google-fu, so in the best
spirit of mathematicians we remove a difficult problem (lots of
exception-handling) in the master process by reducing it to one we've
already solved and put command file polling, reading and forwarding
into a separate process.

Signed-off-by: Andreas Ericsson <ae@op5.se>