afmongodb: Include libmongo-client/configure.gnu in the tarball

In order to install cleanly with an internal libmongo-client, we need
that library to be compiled in statically, which is what the
configure.gnu script (installed via autogen.sh from git) enforces.

However, since configure.gnu is not part of libmongo-client, its make
dist will not include it, so its parent must do that. For this reason,
and similarly to how the same file in lib/ivykis is handled, include
libmongo-client/configure.gnu in the EXTRA_DIST of
afmongodb/Makefile.am.

Reported-by: Peter Czanik <czanik@balabit.hu>
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
logreader: When following a file, treat non-regular files specially

When following a file with follow-freq(), and it is not a regular
file, but something for which stat(2) always returns 0 size (such as
the /dev/klog character device on FreeBSD),
log_reader_io_follow_file() never went into the process input branch,
because the current position (0) was always at EOF (also 0).

To break out of this unfortunate situation, if the file to read from
is not a regular file, go into the process input loop anyway: at
worst, it will do an expensive noop.

Signed-off-by: Gergely Nagy <algernon@balabit.hu>
ivykis: Update to 0.30.4.

Update the bundled ivykis to 0.30.4, which contains an important
workaround that affects syslog-ng users on FreeBSD.

Along with the previous commit, these two appear to fix all problems
related to /dev/klog on FreeBSD.

Signed-off-by: Gergely Nagy <algernon@madhouse-project.org>
afstreams: unref the reader at deinit time

In order to make sure we don't leak the /dev/log fd, unref the reader
at deinit time, and set it to NULL. This way, it will be reopened
shortly after in the reload cycle, and we won't leak the fd.

Reported-by: Michael Hocke <michael.hocke@nyu.edu>
Signed-off-by: Gergely Nagy <algernon@balabit.hu>
Preparations for 3.3.6.91

NEWS & VERSION updated for 3.3.6.91 (aka 3.3.7 RC2)

Signed-off-by: Gergely Nagy <algernon@balabit.hu>
Update the travis control file to apt-get update too

To fix issues with uninstallable packages, update the travis-ci
control file so that it calls apt-get update too.

Signed-off-by: Gergely Nagy <algernon@balabit.hu>
pdbtool dump: fixed -p option parsing

And also added a warning if neither -T or -P was specified.

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
travis: Install under $HOME, to avoid permission issues

Instead of using sudo to install, install to a directory under $HOME,
so we don't run into permission problems when running the test suite.

Signed-off-by: Gergely Nagy <algernon@balabit.hu>
fixed invalid declarations for bison generated yyparse() functions

The new bison finally declares the yyparse() function in its header, which
revealed a couple of invalid declarations in a couple of parsers.

This patch fixes that.

Reported-By: Conrad Hoffmann <ch@bitfehler.net>
Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
Signed-off-by: Gergely Nagy <algernon@madhouse-project.org>
LogParser: pass length argument to log_parser_process()

Since a number of parsers require an explicit length (json-parser,
and the future syslog-parser), it's useful to pass this information along,
especially as we have it in log_parser_queue(). This can potentially
save an strlen() on a potentially large chunk of data in the fastpath.

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
cfg-grammar: reorganize rules so that msg_format related rules can be reused from plugins

This patch introduces msg_format_option rule that can be referenced from
plugin grammars to parse format specific options. This is going
to be needed by the syslog-parser() plugin.

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
cfg-grammar: move common parser rules from csvparser plugin

Earlier the only plugin that used these were the csvparser plugin, but
now they make sense for syslog-parser() too

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
LogReader: separate msg-format related flags to a separate function

Earlier msg_format related flags were parsed by log_reader_options_process_flag().
Separate that to another function, so that it can be used independently
from LogReader.

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
syslog-parser: introduce the new syslog-parser() parser

This patch creates a new parser, to explicitly parse messages as syslog
messages. This can be used to selectively parse these messages based on
some filters.

Imagine this use-case:

log {
        source {
                udp(port(2000) flags(no-parse));
        };
        parser {
                log {
                        junction {
                                log {
                                        filter { netmask("127.0.0.1/32"); };
                                        parser { syslog-parser(); };
                                        flags(final);
                                };
                                log {
                                        filter { netmask("127.0.0.2/32"); };
                                        parser { csv-parser(columns("C1", "C2", "C3")); };
                                        flags(final);
                                };
                        };
                };

        };
        destination {
                file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n"));
        };
};

Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2
as a csv-parser() style messages.

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>
Merge remote-tracking branch '3.3/master'

Conflicts:
	.travis.yml
	NEWS
	VERSION
	lib/parser-expr-parser.c
	lib/rewrite-expr-parser.c
	modules/afsocket/afinet-source.h