Porting dummyet to Linux andWindows (and userland)
Luigi Rizzo, Università di Pisa
May 14, 2010

SummaryIn this talk we will describe the issues and lessons learned in
porting a network-related kernel module from FreeBSD todifferent operating systems.
In detail:◮ motivation and objectives;◮ description of the system being ported;◮ porting strategy;◮ identification of the subsystems involved;◮ system-specific issues;◮ lessons learned.

2 / 25

Motivation for this work
As part of the ONELAB2 project www.onelab.eu we needed to
implement in-node emulation for PlanetLab.

We opted for a port of ipfw and dummynet because:◮ existing Linux solutions (tc+netem; NISTnet; netpath)
were not as flexible as dummynet;◮ a non-negligible integration and porting work was stillneeded even with the above systems;◮ a Linux port was desirable in itself.
3 / 25

Objectives
During the work, we decided to address the following issues:◮ add scheduler support (direct requirement of the Onelab
project);◮ improve scalability (fixes a performance issue in Onelab);◮ provide some user-level testing tools (ease development,improve the quality of the software);◮ create a generic Linux port, because Planetlab nodes use
different Linux versions;◮ develop OpenWRT and Windows versions, as it only
required a limited additional effort, and would make thetool available to a much larger user base.

4 / 25

The systems to be ported
Ipfw and Dummynet info.iet.unipi.it/∼luigi/dummynet/ are afirewall and traffic shaper/network emulator, made of:◮ a user interface, /sbin/ipfw, running in user space andcommunicating with the kernel through a control socket;◮ several kernel modules (ipfw.ko, dummynet.ko,schedulers...) attached to the pfil hooks to intercept
packets.
The original code was not specifically designed for portability,
so it uses several FreeBSD-specific structures and subsystems:◮ mbufs, pfil hooks, memory allocator, locking, timerservices;◮ ip output() and netisr dispatch();◮ routing table, module management, control sockets.
5 / 25

Porting approach
Our approach was to port the code to Linux with as little
modifications as possible to the original code:◮ faster, less error prone;◮ easier to keep the software up to date;◮ small performance loss is not a concern.
Workplan:◮ identify differences among platforms;◮ provide replacements for headers;◮ provide wrappers for similar functions/subsystems;◮ develop glue code to map FreeBSD kernel APIs tounderlying OS APIs.
We do not require nor use any GPL code.
6 / 25

Porting the userspace code
Porting the userspace code to Linux/Cygwin was almost
straightforward:◮ language and APIs are relatively portable across platforms(BSD, Linux, Cygwin);◮ no strange linker tricks in the code;
Main points:◮ header adaptation – discussed next;◮ missing library functions (humanize number(), ...)
obtained from the original, BSD-licensed source code;◮ Windows: remap setsockopt() to DeviceIoControl()(similar in principle, device handle instead of a socket);◮ sysctl emulated over the control interface;All extensions are in one file, glue.c – 800 lines (mostly forlibrary functions and sysctl emulation).
7 / 25

Building userspace code with Windows tools
Windows porting with native tools (MSVC, tcc) is slightly
more difficult:◮ useful because it produces a GPL-free binary;◮ larger differences in headers, APIs and basic data types(WORD DWORD FAR ...);◮ missing functionalities (fork, process control, printfformats ...);◮ missing compiler features (e.g. C99 initializers in MSVC).
Problems solved with some headers tricks, minor rewrites orremoving some functions: only two small "#ifdef TCC ..."
sections in ∼7000 lines of code.
8 / 25

Porting the kernel side

Porting the kernel code is much more challenging andinteresting:◮ lack of cross-platform standards, both for header names
and content, and kernel APIs;◮ many more subsystems involved.
Header remapping and large use of macros go a long way in
reducing differences.

9 / 25

Step 0: userspace version of the kernel code
We started by building a userspace version of the kernel code:◮ quickly identify missing headers and libraries;◮ experiment with various porting approaches.
Not a wasted effort:◮ eventually, we had a daemon that could talk to/sbin/ipfw through emulated *sockopt();◮ useful to test rule injection and listing;◮ opened the way to develop the scheduler testing code;◮ we plan to add packet handling (e.g. from a PCAP file)
to test packet matching functionality and performance.

10 / 25

Step 1: Header remapping
There are significant differences in kernel headers:◮ some BSD headers are missing on other systems;◮ some have the same name but different content;◮ some have different names for a given content;
From many headers we need only a handful of lines, so:◮ -include ... to import common definitions (2 files,∼1000 lines);◮ a subtree -Iinclude/ contains ∼30 headers copied
almost verbatim from FreeBSD;◮ a subtree -Iinclude e/ is populated with ∼50 empty
headers, for files with no (remaining) content.
Kernel compile flags start with-nostdinc -include ../glue.h -include missing.h-Iinclude -Iinclude e ...
11 / 25

Header contents
The -include’d headers do a variety of remapping tricks:# define ifnet

net_devic e/* remap */# define printf ( fmt , arg ...) printk ( KERN_ERR fmt , ## arg )# define bcopy ( _s , _d , _l )memcpy ( _d , _s , _l )# define I P _ F W _ S E T S O C K O P T \CTL_CODE ( FILE_DEVICE_ I PF W , I P _ F W _ B A S E _ C T L + 1 , \METHOD_BUFFER ED , F I L E _ W R I T E _ D A T A )# define _ S Y S C T L _ B A S E ( _name , _var , _ty , _perm )\m o d u l e _ p a r a m _ n a m e d ( _name , *( _var ) , _ty ,\( ( _perm ) == CTLFLAG_RD ) ? 0444: 0644 )Most of these macros are the result of a comparison of how
the various subsystems are implemented on different platforms.
12 / 25

Step 2..N: handle kernel subsystems
An interesting part of the work has been identifying thedifferences in various subsystems:◮ packet representation and packet hooks;◮ memory allocation;◮ locking;◮ timers (API and resolution);◮ module support;◮ userland/kernel communication;◮ OS-specific issues.
These will be described in the next slides.

13 / 25

Packet representation
In-kernel packet representation always uses a descriptor tostore metadata and a linked lists of buffers:◮ mbufs on FreeBSD;◮ skbufs on Linux;◮ NDIS PACKETs on Windows.
Our code uses mbufs, so we do the following:◮ create mbuf lookalikes on entry;◮ copy metadata from native
representation;◮ reference or copy data;◮ destroy the wrapper on exit.
Exact details depend on packet hooks behaviour.
14 / 25

Packet filtering hooks
Dummynet must sometimes hold/delay/drop packets. Slightly
different semantics among systems:
FreeBSD pfil hooks allow a hook function to free or hold a packet;
Linux netfilter hooks require all packets to be marked andreturned. Packets can be held on a subsequent QUEUEcall;
Windows NDIS miniport modules do not allow modifications topackets. A module must replicate the packet tohold/modify it.
Some unnecessary data copies could be saved if FreeBSD had
a clear separation between classification and action on thepacket.
15 / 25

Memory allocation
malloc() remapped to OS-specific allocators:◮ kmalloc()/ kfree() on Linux;◮ ExAllocatePoolWithTag()/ExFreePool() onWindows;
UMA allocators are replaced by a much simpler version:
typedef# define# define# define# define

int uma_zone_t ;/* the zone size */uma_zcrea t e ( name , len , _3 , _4 , _5 , _6 , _7 , _8 ) ( len )uma_zallo c ( zone , flags ) malloc ( zone , M_IPFW , flags )uma_zfree ( zone , item )free ( item , M_IPFW )u m a _ z d e s t r o y ( zone )do {} while (0)16 / 25

Locking

Fortunately we use a very simple locking mechanisms (rwlocks,rmlocks, mtx).
◮ define/declare/lock/unlock/destroy wrapped in macros;◮ map to spinlock t on Linux;◮ map to FAST MUTEX on Windows.

17 / 25

Timers (and callouts)
Used for two purposes:◮ Return the time of day, with < 10µs resolution andprecision.
getmicrouptime() or microuptime() on FreeBSD;
do gettimeofday() on Linux;◮ custom replacement (TSC-based) on Windows as we
could not find a function with less than 10ms resolution.
◮◮◮ Wake me up after time T:

callout init/callout reset on FreeBSD.
mapped onto init timer()/add timer() on linux;◮ Deferred Procedure Calls (DPC) on Windows:
KeInitializeDpc()/KeSetTimer()◮ hardest part was locating the right API to set the kerneltick on Windows (ExSetTimerResolution() ).
◮◮18 / 25

Module support
Modules have descriptors to indicate constructors, destructors
and dependencies:
D E C L A R E _ M O D U L E ( dummynet , dummynet_mod ,S I _ S U B _ P R O T O _ I F A T T A C H D O M A I N , SI_ORDER_ANY -1);
M O D U L E _ D E P E N D ( dummynet , ipfw , 2 , 2 , 2);
D E C L A R E _ M O D U L E ( ipfw_nat , ipfw_nat_mod ,S I _ S U B _ P R O T O _ I F A T T A C H D O M A I N , S I _ O R D E R _ A N Y );M O D U L E _ D E P E N D ( ipfw_nat , libalias , 1 , 1 , 1);
M O D U L E _ D E P E N D ( ipfw_nat , ipfw , 2 , 2 , 2);◮ heavily based on linker sets;◮ potential portability issues with different toolchains (e.g.

we use MSVC and possibly TCC).
Possible workarounds:◮ make the descriptors globally visible;◮ manually (or automatically) build the list of module
descriptors.

19 / 25

Kernel – userland communication◮ getsockopt()/setsockopt() on a raw socket.
◮ Linux has a similar mechanism, slightly different API;◮ Windows uses DeviceIoControl(), which operates on adevice descriptor;◮ in both cases, ported using wrappers to adapt the API;
◮ the interface has been extended to emulate sysctl for
platforms missing them.

20 / 25

Linux specific issues and features◮ sysctl mapped to /sys/module/ entries on 2.6.x,implemented via sockopt on 2.4.x (openwrt);◮ jail-id replaced by vserver id;◮ IPV6 and in-kernel NAT not implemented yet.
Only one major complaint: very unstable kernel APIs.
The code is cluttered by many (∼ 30) conditional sections for
specific kernel versions;

21 / 25

Windows specific issues and features◮ sysctl implemented via sockopt;◮ no jail/uid/gid matching;◮ no matching on interface names;◮ IPV6 and in-kernel NAT not implemented yet;◮ loopback traffic does not go through NDIS;◮ NDIS glue mostly coming from the miniport driver;◮ installer files available;◮ signed kernel modules for 64-bit systems in the works;

22 / 25

Overall porting effort> wc glue . h tcc_glue . h ipfw / glue . c ...
543
232
841

2187
884
3051

16385 glue . h7141 tcc_glue . h23538 ipfw / glue . c // sysctl and libraries

627
547
906

2480
1802
3681

18627 dummynet2 / missing . h12914 dummynet2 / bsd_compat . c
25957 dummynet2 / ipfw2_mod . c

630
225

2624
934

20104 dummynet2 / md_win . c
7100 dummynet2 / winmissing . h4551

17643

131766 total

23 / 25

Lessons learned
The original code was reasonably portable, despite the lack ofany specific effort. Some things could and should be improved:◮ need better split between classification and emulation;◮ confusion on the endianness of certain fields (ip len,etc.) obfuscates the code and requires writable buffers;◮ nested #include would have made header mapping a lot
simpler;◮ when it comes to locking and other architecture-specificfunctions, hiding details behind macros is a big advantage.

24 / 25

Availability and Credits
Latest code at http://info.iet.unipi.it/∼luigi/dummynet/
The new code is available for◮ FreeBSD HEAD and stable/8◮ Linux/OpenWRT◮ Windows XP, Windows 7 (32 and 64 bit)
◮ OSX ? (currently older version. Ask Apple...)Credits:◮ Marta Carbone (Linux port)◮ Fabio Checconi (QFQ, KPS)
◮ Riccardo Panicucci (scheduler API)
◮ Francesco Magno (Windows port)
25 / 25