Monitoring IP-based networks
By: Florian Lehner
Tags: Linux, Monitoring, Ebpf, Network, Golang

Monitoring is an essential part of operating an IP-based network. Either to get more insights while debugging and tracing issues or simply for accounting reasons. This article will discuss various techniques, that can be used on Linux systems, to increase the observability of IP packets in networks. One question, that needs to be answered quite often, is “How many bytes have been transferred between two IP addresses and which upper layer protocol type has been used?”. This article will show different ways to answer these questions.

An opportunistic approach answering these questions is to use a library like libpcap and inspect every IP packet in user space. For this proposal, every IP packet will be copied to user space to inspect it and extract the needed information. There is no impact of the processing of the IP packet in kernel space, as the user space receives only a copy. Nevertheless, copying all IP packets to user space produces a high I/O load on the system and can have a negative impact on the overall performance of the system.

A different way to answer the initial question comes with iptables. Over the years, a feature-rich ecosystem has been built around iptables. This ecosystem can not only be used to filter IP packets but also to get information about processed packets with various extensions. These extensions are widely used and add even more requirements to successors, like nftables. One of the components that can be used along with iptables is libnetfilter_log. With the libnetfilter_log application programming interface (API) user space tools can receive metadata and copies of IP packets. But similar to the approach with libpcap, information, like source and destination IP from our initial question, needs to be extracted by hand. In addition, filtering rules need to be adapted to send data via the libnetfilter_log API. A similar technique is provided by libnetfilter_queue. This API not only copies data to user space as libnetfilter_log does. Furthermore, the kernel puts the processing of the IP packet on hold and waits for instruction on how to handle it.



A slightly different strategy can be implemented by using the libnetfilter_conntrack API. It grants direct access to the in-kernel connection tracking system. As a bonus, it provides all the information to answer the initial question without the need to copy the packet to user space, parse and extract its information. Unfortunately, ENOBUFS¹ errors need to be handled and the information about sessions can get lost because of this.

All of the above-mentioned solutions have one thing in common: All of them have their origin in the netfilter project. This project does not only continually improve the in-kernel part of the networking subsystem but also provides APIs for user space implementations.

A fundamentally different approach was introduced back in 1992 under the name BSD Packet Filter (BPF). BPF was developed to increase the performance of packet filtering by directly applying the filter on the kernel data link layer instead of copying the IP packet across the kernel/user space protection boundary. The popularity of BPF increased over the last years and it is no longer limited to socket based operations. The iovisor project makes use of the BPF infrastructure in the Linux kernel and builds powerful tools for every kind of tracing inside the Linux kernel. And all that without the need to recompile the Linux kernel or load custom kernel modules. In addition, with the most recent extended BPF (eBPF), you can directly make use of exclusive in-kernel data structures.

eBPF provides various ways, called program types, to monitor the in-kernel packet processing. One of these program types is BPF_PROG_TYPE_TRACEPOINT. Tracepoints are predefined hooks where you can attach your eBPF program in the execution path of the kernel. Depending on the chosen tracepoint, you can directly access and extract information to answer the initial question and improve your monitoring.

A quite new and special program type is BPF_PROG_TYPE_XDP. Except for the generic parts, this program type has to be supported by the underlying network hardware. With the eXpress Data Path (XDP) network packets can be modified just before they will be pushed to the network stack inside the kernel. By rejecting a network packet on the network card, the kernel will not know about it and save on resources for it. Depending on the use case, this technique can reduce the I/O load dramatically. So far, XDP is limited to ingress traffic.

This article covers various ways how to extract information, needed to monitor an IP-based network. As network throughput permanently increases, the processing speed of this data needs to improve and extracting related information is only the first step of the challenge to monitor an IP-based network. Every approach has its advantages and disadvantages. It’s your responsibility to decide which one solves your problem best.