Update on Network Team Driver project
What’s that, what’s done, what’s planned
Red HatJiřı́ Pı́rko (jpirko@redhat.com)

Jiřı́ Pı́rko (jpirko@redhat.com)

1Section 1
Lightning talk partJiřı́ Pı́rko (jpirko@redhat.com)

2

Lightning talk partlink aggregation setup examples

desktops

desktops

user switch 2

user switch 1

core switch 1

core switch 2

serverserver A

Jiřı́ Pı́rko (jpirko@redhat.com)

server B3

Lightning talk part

Team driver project overview
http://www.libteam.org
Provides an alternative to existing bonding driverImplements various kinds of link aggregation
One interface to user-space - Team Netlink
Minimum of the code runs in kernel-space
Introduces NO performance penalty (fast TX/RX paths is in kernel)

Control logic is implemented in user-space daemon (teamd)
StatusPackage libteam-1.0 in stable repository of F17/F18
Working on package for Debiankdump support (Rawhide, F19)

Jiřı́ Pı́rko (jpirko@redhat.com)

4Lightning talk part

Team device architectureuser-spacerunnerrunner

teamd

teamd

libteamlibteam
Team Netlink

team0(team driver)eth0

team1(team driver)

eth1

eth2

eth3

kernelJiřı́ Pı́rko (jpirko@redhat.com)

5Lightning talk part

Features comparison (selected ones)
featurebroadcast,round-robin TX policy
active-backup TX policy
LACP (802.3ad) support
Hash-based TX policy
User can set hash function
TX load-balancing support (TLB)
TX load-balancing support for LACP
Ethtool link monitoring
ARP link monitoring
NS/NA (IPV6) link monitoring
RCU locking on TX/RX paths
port prio and stickinessseparate per-port link monitoring setup
multiple link monitoring setupVLAN support
multiple device stacking
Jiřı́ Pı́rko (jpirko@redhat.com)

bonding
Yes
Yes
Yes
Yes
NoYes
NoYes
Yes
NoNoNoNoLimitedYes
Yes

teamYes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
6

Lightning talk part

What’s planned
Network Manager basic support (F19 or F20)
Network Manager full GUI support (F20 or F21)
RX balancing support (ala bonding ALB)
bridge/openvswitch support
Connection tracking

Autoconfiguration support using LLDP (link layer discovery protocol)
Extended custom TX hash function setup
Pre-defined protocols (eth, vlan, ipv4, ipv6, udp, tcp, sctp, ...)
User-defined custom protocols

Extended teamd API support
Full featured teamd control API - wrapped by libteamd
Support teamd runners over teamd API (standalone processes)

Jiřı́ Pı́rko (jpirko@redhat.com)

7Section 2
AddendumJiřı́ Pı́rko (jpirko@redhat.com)

8Addendum

Team Netlink

Uses Generic Netlink
Setting options and getting events
Port list message
Read-only (kernel-space to user-space only)
Contains interface indexes, link info (ethtool)

Option list message
Both directions
User-space sets options and that changes behaviour
Various data types

Accesible directly by wrap-up teamnl utility (debug)

Jiřı́ Pı́rko (jpirko@redhat.com)

9Addendum

team driver
Present in upstream Linux kernel since 3.3
One instance - one network interface (e.g. team0)
Code partition:
Netdev API code
Necessary fast-path code (transmit and receive packets)
Netlink communication (Team Netlink)
Modes code

Team ”modes”:
One mode - one kernel module
Each mode determines specific low-level behaviour
Well defined API between team core and mode code
round-robin, broadcast, active-backup, loadbalance
Easy to add moreJiřı́ Pı́rko (jpirko@redhat.com)

10Addendum

libteam

Does user-space wrapping of Team Netlink communication
Wraps-up Route Netlink messages (newlink, dellink etc.)
Uses libnl (genl, rtnl)
Exports API to user for controlling kernel team driver instance(setting options)
Allows to register ”handlers” for watching team driver instanceevents (port and option changes)
Python binding availableJiřı́ Pı́rko (jpirko@redhat.com)

11Addendum

teamd

Uses libteam, libdbus, jansson, libdaemon
One instance controls one team driver instanceOn startup it creates team driver instanceConfig file in JSON format
Higher logic is implemented in Runners
Link monitoring is implemented in Link-watches
Run-time control via D-Bus API
Easy to use:
# teamd -f team0.confJiřı́ Pı́rko (jpirko@redhat.com)

12Addendum

teamd Runners + Link-watches
Runners
Only one has to be selected
Determine behaviour
Set-up team driver instance (mode, options)Monitor team driver instance behaviour and react to thatKinds:
Round-robin / Broadcast
Active-backup
LACP (802.3ad)
Load-Balancing

Link-watches
Link monitoring
Kinds:ethtool
ARP ping
IPv6 NS/NA ping (Neighbor Discovery Protocol)

Can be set as global or per-port
Possible to set multiple link-watches
Jiřı́ Pı́rko (jpirko@redhat.com)

13

Addendum

teamd D-Bus API

Service name: org.libteam.teamd.[teamdevname]Methods:
ConfigDump - Get teamd JSON config
StateDump - Get teamd state
PortAdd, PortRemove - Adds/removes port
PortConfigUpdate - Updates config for a given port

Wrapped up by teamdctl utility
Might be eventually extended for possibility of runner
implementation (idea)

Jiřı́ Pı́rko (jpirko@redhat.com)

14

Addendum

teamd config example 1

team0.conf{"device": "team0","runner": {"name": "roundrobin"},"ports": {"eth1": {}, "eth2": {}}}

Jiřı́ Pı́rko (jpirko@redhat.com)

15

Addendum

teamd config example 2
team0 ab ethtool.conf{"device": "team0","runner": {"name": "activebackup"},"link_watch": {"name": "ethtool"},"ports": {"eth1": {"prio": -10,
"sticky": true},"eth2": {"prio": 100}}}

Jiřı́ Pı́rko (jpirko@redhat.com)

16

Addendum

Advantages comparing to bonding

Extensibility. Anyone can easily add features/change behaviour
Better system stability (daemon crash is always better than kernel
panic/memory corruption etc.)
Better debugging posibilities
Minimum kernel changes required

Jiřı́ Pı́rko (jpirko@redhat.com)

17

Addendum

team driver implementation

RCU-locking (multiple incoming/outgoing packets can go in parallelwith setting up the device)
Exploits rx handler on RX path to intercept packets
Uses dev queue xmit() to pass packets to NIC driver on TX path
netdevice notifier events are passed via Netlink
Option infrastructure - for easy add options to modes

Jiřı́ Pı́rko (jpirko@redhat.com)

18

Addendum

Extension possibilities

Designed to be easily extendable by any advanced user
Kernel-space extension
Add mode (should be rarely needed)

User-space extension
Make libteam based application
Make libteam python binding based application
Add teamd runner (preferred)

Jiřı́ Pı́rko (jpirko@redhat.com)

19

The end.
Thanks for listening.

Jiřı́ Pı́rko (jpirko@redhat.com)

20