Quality of Service on Linuxfor the Atlas TDAQ Event Building Network
Y.Yasu1 , A. Manabe1 , Y. Nagasaka2 , Y. Hasegawa3 , M. Shimojima4 ,M. Nomachi5 , H.Fujii1 and Y.Watase1 on behalf of the Atlas Trigger/DAQ group
1 (High Energy Accelerator Research Organization, Japan)
2 (Hiroshima Institute of Technology, Japan)
3 (Shinshu University, Japan)
4 (Nagasaki Institute of Applied Science, Japan)
5 (Osaka University, Japan)

Abstract

Congestion control for packets sent on a network is important for DAQ systems that
contain an event builder using switching network technologies. Quality of Service (QoS)
is a technique for congestion control. Recent Linux releases provide QoS in the kernel
to manage network trac. We have analyzed the packet-loss and packet distribution
for the event builder prototype of the Atlas TDAQ system. We used PC/Linux with
Gigabit Ethernet network as the testbed. The result showed that QoS using CBQ andTBF eliminated packet loss on UDP/IP transfer while the UDP/IP transfer in best e ort
made lots of packet loss. The result also showed that the QoS overhead was small. We
concluded that QoS on Linux performed eciently in TCP/IP and UDP/IP and will havean important role of the Atlas TDAQ system.

Keywords: QoS, Linux, TCP, UDP, IP Multicast, CBQ, TBF, DAQ

1 Introduction
Trac management of the data ow is an essential point to avoid congestion. QoS is one of thesolutions to avoid the congestion, because it provides the bandwidth allocation in the network.
Gigabit Ethernet technology, which is a major in the high performance network, however, has
no functionality of the QoS while ATM has the functionality on a hardware level. Recently,link-sharing and resource management for packet network is intensively studied in computerscience [3]. On Gigabit Ethernet with QoS, it will be expected to avoid the congestion of theevent builder trac.
From the view point of scalability for the event builder, IP multicast is attractive. However, TCP does not support multicast while UDP is a non-reliable protocol. There are manystudies of IP multicast. A solution is to make protocols reliable in the application level. Another solution is to make it reliable in lower levels such as Data link layer. QoS on Linux is
implemented below the IP layer and woven into the network layer.
We studied the DAQ/EF-1 Event Builder using Linux/Gigabit Ethernet [1]. We alreadyhad a preliminary result of QoS for the event builder [2].

2 Congestion Control and Bandwidth Allocation
The essential point of trac management is packet scheduling, known as queueing method.
Packet queueing disciplines are studied in techniques related to ATM in computer science.
They are Class-Based Queueing (CBQ), Token Bucket Filter (TBF) and so on.

2.1 Trac Management in LinuxFor Linux, kernel 2.2.x and later support many kinds of QoS. The developer toolkit also supplies
some commands for manipulating the parameters in the kernel. The typical command is calledtc. CBQ queueing discipline in Linux kernel supports hierarchical allocation of bandwidth forlink sharing, limiting the packet rate, prioritizing and so on. The TBF queueing discipline inLinux kernel can also limit packet rate by using a token bucket algorithm. The trac control in
TCP/IP for Linux is done only for output queueing. For an example of a queueing discipline,
one can de ne CBQ as root with a bandwidth of 1000Mbit/s. When a packet comes into
the queueing discipline, the packet will be sent to a classi er called u32, which classi es thepacket. The packet will come into a class called CBQ, which also assigns 1000Mbit/s as thebandwidth. After the packet passes through the class, a queueing discipline called TBF willlimit the transfer speed. The transfer speed will be limited to 100Mbit/s according to the rateof the token even if packets arrive over the rate.

3 Performance Evaluation

CPU
PentiumIII/500MHz
Linux kernel version
2.4.5

Table 1: Con guration of PC500 system
ChipsetMemory440GX
100MHz/SDRAM/256MBgcc version NICegcs-2.91.66 AceNIC(1MB)

No QoSTransfer speed in MB/s

Packet loss in percentage

1200

Transfer speed in MB/s

100901000

80108086006050

66040404302000

Transfer speed in MB/s

300

600
900
1200
Message size in bytes

Assigned bandwidth = 50 Mbit/s

20

2100
1500

0300

Transfer speed in MB/s

50100

120

4080

10060

3001500

0600

900

Message size in bytes

Packet loss in percentage

Packet loss in percentage

10070

200

Assigned bandwidth = 10 Mbit /s

12800

400

PCIbus32-bit/33MHz
acenic driver version
0.81200No packet loss in any case

Assigned bandwidth = 100 Mbit/s

Packet loss in percentage

1008080606040204040201000204060

Message size in bytes

800100No packet loss after 60-byte size

202000300600

900
Message size in bytes

Figure 1: Packet loss on UDP/IP multicast transfer

1200

01500

No packet loss after 200-byte size

3.1 Measurement
On the measurement of QoS, we measured and evaluated Packet loss for UDP/IP multicast
transfer, TCP/IP transfer speed with/without QoS, CPU usage of QoS on TCP/IP transferand packet distribution on TCP/IP transfer with/without QoS & regular trigger. We used
netperf and ttcp utilities as the measurement tools. The Linux network driver for Alteon
AceNIC was used. We enabled the Nagle algorithm for the measurement. The TCP bu er size
was 500k bytes.

3.2 Setup

The con guration of the PCs is shown in Table 1. Four PCs of PC500 were used with a Gigabit
Ethernet switch for the evaluation of QoS. The switch was a 3Com Super StackII Switch 9300,
which has 12 Gigabit Ethernet ports.

3.3 Packet loss on UDP/IP multicast transfer

Figure 1 shows an analysis of packet loss on UDP/IP multicast transfer. We made a special toolin this measurement. The dark line in the Figure shows the transfer speed and the light line
shows the percentage of packet loss. Without QoS, lots of the packet loss were observed. Thisdepends on the message size and the percentage of packet loss was over 60 % when the message
size was 10 bytes. On the other hand, no packet loss was observed when an expected bandwidth
was assigned with 10 Mbit/s. When the bandwidth was assigned with 50 Mbit/s, packet loss
occurred at small message sizes. When the assigned bandwidth was 100 Mbit/s, the message
size that packet loss occurred became larger. We should manage the assigned bandwidth when
we eliminate packet loss on UDP/IP transfer. It depends on the message size. We also observedthat there was no packet loss at the sender while the receiver lost the packet.

3.4 Speed and CPU usage on TCP/IP transfer
Transfer speed in Mbit /sec.

110700600

CPU Usage (%)

100500sender cpuusage(No QoS)
sender cpu
usage(QoS)

90400

No QoS
QoS

30080200701000101001000

Message size in bytes

100006010100

100010000

Message size in bytes

Figure 2: Speed and CPU usage on TCP/IP transfer
Figure 2 shows transfer speed and CPU usage. The assigned bandwidth was 1000 Mbit/s
when QoS is applied. This means the transfer is not limited by the assigned bandwidth. TheQoS algorithm uses more CPU time, compared with No QoS. The overhead of QoS algorithm orthe eciency of transfer depends on the message size. The transfer with QoS is very ecient atsmall message and large message. That is, QoS overhead is small. The CPU usage between thetransfer with QoS and that without QoS was small at small message, but not at large message.
10 % more CPU was used at large message on the transfer with QoS.

3.5 Packet distribution on TCP/IP transfer

00.0250.05

31021031021010

1

100.025

Time (sec)QoS-10Mbps

0.05

Number of packets

310

Number of packets

10

Number of packets

Number of packets

A pseudo trigger was provided for generating a regular trigger signal. The assigned bandwidth
was 10 Mbit/s. The message size was 10 bytes. The tcpdump utility was used for sampling
the data. We measured the packet distribution in 4 cases, the transfer with QoS, the transferwithout QoS, the transfer with QoS in way to generate regular trigger and the transfer without
QoS in the way. Figure 3 shows the packet distributions on TCP/IP transfer in those cases.
The x axis is a time interval between sending packets, in seconds. The y axis is the numberof packets. In rst case, which is on the left side of the Figure, QoS managed to send packets
at 10 msec, which is the Linux OS scheduling time. In the second case, packets were mostly
sent in several 10 usec. In the third and fourth case, packets were mostly sent around 1 msec,
which is the regular trigger rate, 1 kHz. The di erence between the third and the fourth was
not observed.

00.025Time (sec)
NoQoS

0.05

103

10200.025

Time (sec)QoS-10Mbps-VTRG

0.05
Time (sec)

NoQoS-VTRG

Figure 3: Packet distribution on TCP/IP transfer

4 Conclusion
We measured and evaluated Packet loss on UDP/IP multicast transfer, speed and CPU usage
on TCP/IP transfer, and packet distribution on TCP/IP transfer with/without QoS & regulartrigger. QoS could eliminate packet loss on UDP/IP multicast transfer. This shows the feasibility to solve the scalability of Atlas TDAQ event builder. QoS was eciently performed onTCP/IP transfer. CPU usage of QoS on the transfer was small in comparison with that of No
QoS on the transfer. The QoS makes the packet ow at in unit of the scheduling time.

5 Acknowledgments
The authors wish to thank Prof. Takahiko Kondo at KEK for his support and encouragement.
The authors also like to thank Dr. Hanspeter Beck and Dr. Reiner Hauser for checking ourpaper and their encouragement.

References
[1]
[2]
[3]

Y.Hasegawa, Y.Nagasaka, Y.Yasu, DAQ/EF-1 Event Builder system on Linux/Gigabit
Ethernet, ATL-DAQ2000-008, March 2000
Y.Yasu et al, Quality of Service on Gigabit Ethernet for Event Builder, the 3rd International Data Acquisition Workshop on Networked Data Acquisition Systems, Lyon, France,
October 20, 2000
S.Floyd and V.Jacobson, Link-sharing and Resource Management Models for Packet Networks, IEEE/ACM Transactions on Networking, Vol.3 No.4, August 1995