Unix & Linux: Understand logging in Linux (2 Solutions!!)
Channel: Roel Van de Paar
Date: 2020-08-16

Unix & Linux: Understand logging in Linux


The Question: As I understand, Linux kernel logs to /proc/kmsg file(mostly hardware-related
messages) and /dev/log socket? Anywhere else? Are other applications also able
to send messages to /proc/kmsg or /dev/log? Last but not least, am I correct
that it is the syslog daemon(rsyslog, syslog-ng) which checks messages from
those two places and then distributes those to various files like /var/log/
messages or /var/log/kern.log or even central syslog server?

Solutions: Please watch the whole video to see all solutions, in order of how many people found them helpful

== This solution helped 87 people ==
Simplified, it goes more or less like this:
The kernel logs messages (using the printk() function) to a ring buffer in
kernel space. These messages are made available to user-space applications in
two ways: via the /proc/kmsg file (provided that /proc is mounted), and via the
sys_syslog syscall.
There are two main applications that read (and, to some extent, can control)
the kernel's ring buffer: dmesg(1) and klogd(8). The former is intended to be
run on demand by users, to print the contents of the ring buffer. The latter is
a daemon that reads the messages from /proc/kmsg (or calls sys_syslog, if /proc
is not mounted) and sends them to syslogd(8), or to the console. That covers
the kernel side.
In user space, there's syslogd(8). This is a daemon that listens on a number of
UNIX domain sockets (mainly /dev/log, but others can be configured too), and
optionally to the UDP port 514 for messages. It also receives messages from
klogd(8) (syslogd(8) doesn't care about /proc/kmsg). It then writes these
messages to some files in /log, or to named pipes, or sends them to some remote
hosts (via the syslog protocol, on UDP port 514), as configured in /etc/
syslog.conf.
User-space applications normally use the libc function syslog(3) to log
messages. libc sends these messages to the UNIX domain socket /dev/log (where
they are read by syslogd(8)), but if an application is chroot(2)-ed the
messages might end up being written to other sockets, f.i. to /var/named/dev/
log. It is, of course, essential for the applications sending these logs and
syslogd(8) to agree on the location of these sockets. For these reason syslogd
(8) can be configured to listen to additional sockets aside from the standard /
dev/log.
Finally, the syslog protocol is just a datagram protocol. Nothing stops an
application from sending syslog datagrams to any UNIX domain socket (provided
that its credentials allows it to open the socket), bypassing the syslog(3)
function in libc completely. If the datagrams are correctly formatted syslogd
(8) can use them as if the messages were sent through syslog(3).
Of course, the above covers only the "classic" logging theory. Other daemons
(such as rsyslog and syslog-ng, as you mention) can replace the plain syslogd
(8), and do all sorts of nifty things, like send messages to remote hosts via
encrypted TCP connections, provide high resolution timestamps, and so on. And
there's also systemd, that is slowly phagocytosing the UNIX part of Linux.
systemd has its own logging mechanisms, but that story would have to be told by
somebody else. :)
Differences with the *BSD world:
On *BSD there is no klogd(8), and /proc either doesn't exist (on OpenBSD) or is
mostly obsolete (on FreeBSD and NetBSD). syslogd(8) reads kernel messages from
the character device /dev/klog, and dmesg(1) uses /dev/kmem to decode kernel
names. Only OpenBSD has a /dev/log. FreeBSD uses two UNIX domain sockets /var/
run/log and var/rub/logpriv instead, and NetBSD has a /var/run/log.

With thanks & praise to God, and with thanks to the many people who have made this project possible! | Content (except music & images) licensed under cc by-sa 3.0 | Music: https://www.bensound.com/royalty-free-music | Images: https://stocksnap.io/license & others | With thanks to user Martin (https://unix.stackexchange.com/users/33060), user lcd047 (https://unix.stackexchange.com/users/111878), user JdeBP (https://unix.stackexchange.com/users/5132), and the Stack Exchange Network (http://unix.stackexchange.com/questions/205883). Trademarks are property of their respective owners. Disclaimer: All information is provided "AS IS" without warranty of any kind. You are responsible for your own actions. Please contact me if anything is amiss at Roel D.OT VandePaar A.T gmail.com.