Demystification of TC
By: Frederic Boismenu
Tags: Containers, Tc, Mesos, Isolation, Infrastructure

TC Object Model

Overview

The TC objects have been thought as a way to model your policies as a decision graph.

Filters and Actions essentially allows you to handle the classification problem, while the definition of the actual policies is done with QDiscs and Classes objects.

QDisc and Classes

QDisc stands for queuing discipline, it is the high level object for shaping and scheduling algorithms (both activities are referred collectively as scheduling in Linux kernel terminology).

For example, the simplest qdisc would be a FIFO, as a matter of fact the default qdisc in Linux is pfifo_fast.

Every network device (physical or virtual!) have two entry points, one for each data direction:

one for outgoing packets ( egress path),

path), and one for incoming packets (ingress path).

On these entry points, we must have a qdisc attached to it, the kernel is always providing default ones if not explicitly setup.

On the egress entry point, a pfifo_fast qdisc is setup by default.

The ingress qdisc is a bit special, as it forbids to attach any qdisc but allows to attach filters to it. We cannot shape ingress traffic, but with filter we can police it.

Here is the default setup of your machine if you don’t change anything explicitly:

There are two types of qdiscs: classful and classless.

You can think of classless qdiscs simply as queues, sometimes with fancy algorithms ruling their basic operations such as enqueue, dequeue, etc. All the packets in the classless qdiscs are managed on the same basis, without any requirement (and capability) for classification. The pfifo_fast is the perfect example of a classless qdiscs. While it’s doing fancy operations such as honoring traffic priority, we cannot define policies for different network flows.

Classful qdics allows to have a different policy for each Class of traffic.

For example, HTB is a traffic shaping classful qdisc. Using it, we could define three classes: one for HTTP traffic, one for BitTorrent traffic, and a third catch all one, assigning them respectively rates of 500Mbps, 200Mbps, and 300Mbps.

Classful qdisc also have the capability to define policies as a hierarchy. In the previous example we could further refine the HTTP policy, and split its 500Mbps into sub-classes, allocating 200Mbps to Youtube traffic, and 300MBps to all other HTTP traffic.

This is possible because a Class is a wrapper of a qdisc, which can be classless or classful.

To summarize the relationship between qdisc and class objects:

Classless qdisc are not allowed to own any Class or Filter

Classful qdisc contains one or several Class(es) and Filter(s)

Classes wraps qdiscs (of any type)

Here is a schematic view of the relationship between qdiscs and classes:

The whole point of classes is to add an identifier (the classid) for qdiscs so they can be targeted by Filters. This is effectively allowing to classify traffic flows.

There is an important constraint with qdisc hierarchies: the leaf(s) have to be classless qdiscs. This is enforced by the kernel which is terminating any newly added class by a pfifo_fast qdics.

Here is the example of a complex, but more realistic setup:

A policy tree like above could help us to describe a something like:

Let’s reserve 10 Gbps of data of the systems (classid=1)

and 1Gbps for Mesos containers (you probably do want to use these numbers…) — classid=2

… then among the Mesos containers, we further divide them into 3 priority classes

all leafs here are using the default pfifo_fast scheduling (but we could use another one such as fq_codel)

Notice that from the network card perspective, it is only talking to the root qdisc, which is actually polling packets from its children with it’s own algorithms, and this recursively until the leaf nodes are polled.

Handles and Classids

Handles and Classids have the same underlying type: a 32bits unsigned integer. The 32bits are split in two 16bits parts <major>:<minor>.

By convention, QDiscs have a handle, with a minor always set to 0 (or simply omitted, in which case we use the notation <major>:). The major number is arbitrary (can be chosen by the end user or randomly assigned by system).

And Classes have a classid, you already knew it. There is a constraint: the major number of the classid must match the handle of the parent qdisc. The minor number of the classid is the class instance number.

There are several special and reserved handle numbers, the most important one is FFFF: being the root qdisc attached to the device.

Practically speaking…

Let’s start to build some isolation for our Mesos containers!

We are going to use:

for traffic shaping: HTB a classful qdisc

for fairness (within the container flows):FQ_CODEL, a classless qdisc

HTB will also gives us fairness among the containers.

Let’s focus on the egress (outgoing) path for now:

Filters & Actions

Filters and actions subsystem of TC is extremely powerful and flexible. It can be viewed as a DSL to program packet processing.

Detailing its super powers is outside the scope of this article, if you want to dig the subject I’d recommend following starting points:

That being said, let’s have a look at what we need from filters to assign packets to relevant queuing discipline.

Filters are attached to classful qdisc (it is not possible to attach filters to classless qdiscs). They remotely look a bit like iptables rules, where you have a part matching packets and another part describing what you want to do with it, it’s called action.

The most common action allows to push it to qdisc class:

The priority defines in which order your filters are evaluated, the lower the priority the higher in the chain.

Our main goal is to apply a different policy per container. So we need to identify to which container a packet belongs to. For outgoing packets it is straightforward, the kernel is giving us the information. But for incoming packets we choose to leverage netfilter’s connection tracking system to identify to which container they are it going to.

Tracking connections means that we can tie back all packets together to their applicative conversation. With CONNMARK, packets belonging to a same conversation will be tagged with the same identifier we chose. This is called “marking” packets.

As a mark we are going to set a number uniquely identifying the associated container.

This identifier is technically the net_cls cgroup classid. In our specific case of Mesos, it is allocated for us by the net_cls isolator.

Once we have this marking in place, tc filters can use this mark to apply the policy corresponding to the container.

Here is the setup we want:

Except that it would not work! Remember, we cannot shape on ingress!

To make it possible, we need to use an IFB device.

Flow redirection to IFB

IFB is a virtual network device with egress and ingress path inverted allowing to shape incoming traffic by attaching any type of qdisc to it.

First, we need to create the device by loading a kernel module.

Then, we redirect ingress traffic our physical NIC with TC incoming traffic to the newly created IFB device. While we do so, we’ll also call CONNMARK capabilities to restore the connection mark.

At this point packets going out of the IFB device can be shaped before reaching the application in the end.

Indeed TC filters have the capabilities to “steal” packets from an interface and to push them to another one.

So the ingress data path now looks like the following:

Now, let’s see how we can set this up!

CONNMARK setup

Nothing super fancy here, just using documented boiler plate:

That’s it, we are done! Now, we are able to shape our traffic going in and out with a different rate per Mesos container.

Going further down the rabbit hole

Now that we have a working system, let’s get a bit more familiar with it to see how we can maintain and troubleshoot it.

Where am I?

Let’s have a look to where all this TC and netfilter things are happening:

Note that this is a simplified 10000 feet view, especially for netfilter which has been grossly over simplified here if you want a zoom of the subject please refer to this map on wikipedia.

As you can see at the lowest of the stack, besides the driver code, you have the hooks used by tools such as tcpdump. Then comes the queuing disciplines, and above that all netfilter logic mixed into the rest of the IP stack.

Queuing disciplines are very low in the stack, which also further explains why handling incoming traffic is difficult since we have little context on the packets.

Let’s review the practical implications of this layering

Packet capture

The traffic you capture with tools such as tcpdump is very very low in the stack, so:

for incoming packets, you see packets which may later on be dropped or redirected by TC or IPTables.

for outgoing packets, you don’t see packets sent by the application but dropped by TC or IPTables

various optimization are happening in the driver: GRO, GSO, TSO, etc, so you don’t see real packets, which makes network analysis complex, especially combined with TC shaping

Network traffic you may capture via NFLOG facility suffers from the same problems of regular captures, but in mirror:

for incoming packets, you don’t see packets dropped by TC

for outgoing packets, you see packets sent by the application but that will never be sent on the wire because of TC

Network analysis could be done by capture data with both regular taps and NFLOG facility and by comparing the capture… You can try to look at tools diffing packet capture files, or export them as text and use your favorite diff tool!

NFLOG example:

There is yet another way to capture traffic, from within TC using the mirred action:

Troubleshooting TC

Ok, you have put in place your qdiscs, classes, filters and … it does not work! what now?

Well, that’s a tough one, here is two tools that can help you.

To debug your filters there’s an action called ‘simple’ which is as the name indicates it to log messages and exposes statics. It can be very helpful to check if your matching expression are actually matched, if the rules are checked, etc.

Another approach is to rely on counters from TC and potentially from iptables:

To setup iptables based counters:

Conntrack for fun and packet loss

Our setup is using conntrack, this is not neutral and needs to be managed in larger systems.

Conntrack is basically managing a connection table, which is bounded in number of entries. If the table gets full on new connections your packets are going to be dropped. You can see if it is happening in dmesg.

Tuning conntrack is subject by itself, but you have to especially pay attention to:

Memory taken by each entry, this is varying with your architecture, on my server it’s 376 bytes big

UDP traffic which can really blow up your table size

different timeout values (see table below)

Conntrack table size can be changed dynamically via sysctl:

Conclusion

We have seen how we can isolate network usage containers from each other on a Mesos. To get there we understood network traffic notions such as packet scheduling, traffic shaping, packet marking, connection tracking, flow classification, and integrate them altogether with Mesos.

The proposed solution has many advantages over existing systems, e.g. the port mapping isolator, such as providing granular policies per container combined with being very agnostic to the underlying network configuration. Also, the setup is a lot simpler as it relies only on cgroups and no namespace.

It’s not without limitation, for example network traffic on the loopback is not taken into account, UDP network traffic is globally not very well handled.

We have demonstrated everything with tc shell commands, a real world Mesos isolator would probably use libnl to implement it.

As usual with network related stuff, the hardest part when you put in place such kind of system will be to help your users, explain the system to them, and troubleshoot it.

Hopefully this article will help you with these matters.

If you want to know more on how we actually implemented network isolation with our Mesos deployment, you can check out our talk at MesosCon18:

https://mesoscon18.sched.com/event/HYTq/moving-toward-network-isolated-containers-in-mesos

https://www.slideshare.net/FredericBoismenu/mesos-nnetwork-isolation-at-criteo

And stay tuned for second post diving deeper into how we integrated into Mesos and let our users defined their network bandwidth requirements!