Quality of Service on Gigabit Ethernet for Event Builder
Y.Yasu1, Y.Nagasaka2,Yoji.Hasegawa3, A.Manabe1,M.Nomachi4, H.Fujii1, Y.Watase1
1

High Energy Accelerator Research Organization (KEK), Japan
2
Nagasaki Institute of Applied Science (NIAS), Japan
3, Shinshu University, Japan
4
Osaka University, Japan

Abstract
Congestion avoidance techniques for event builders
using switching network technologies is a major issue.
Traffic management of the data flow is an essential point
to avoid congestion. Global traffic control is one of themechanisms to manage the traffic. It uses global
information of the switching network. While, traffic
shaping technique is another mechanism not to use them.
Quality of Service (QoS) is one of the solutions to
avoid this congestion, because it provides the bandwidth
allocation in the network. A Gigabit Ethernet technology,
which is a major in the high performance network,
however, has no functionality of the QoS while ATM has
the functionality of the QoS on a hardware level. Recently,
IP-based QoS is intensively studied in computer science.
On Gigabit Ethernet with the QoS, it will be expected to
avoid the congestion of the event builder traffic.
We used Alteon network interface card, which canhandle large Maximum Transmission Unit (MTU) called
jumbo frame. Jumbo frame improves the transfer speed.
We also used Linux-based PCs for measuring theperformance. We chose several parameters such as device
driver parameters, MTU, chipset for PC, bus width of PCI(64/32-bit) and so on, related to the performance.
We found high performance chipset for PC and 64-bit
PCI bus improved the transfer speed in TCP/IP up to990Mbit/s with jumbo frame. We had the results that theoverhead of the IP-based QoS is very small and the
bandwidth allocation worked.

I. INTRODUCTION
In an event builder, event fragments from all sources
are concentrated coherently into one destination via aswitching network. However, commercial-available
switching networks are designed for random traffic like
Tele-communication data. They may not be able tohandle coherent traffic like event builder. Congestion
avoidance is indispensable for switch type event builder.
A way to avoid the congestion is to establish a global
traffic control. A circuit switch was applied to an event
builder at Fermilab [1] and KEK [2]. The data traffic is
controlled by a global traffic signal. Another way is todesign data flow by allocating the bandwidth for each node
in a switching network. The RD31 project at CERN
established a way to shape traffic on ATM network[3,4].
An other way to solve the problem is over-provisioning of
the switching network, but we do not mention it.

Gigabit Ethernet is one of the technologies, which enables
a high-speed transfer for the event builder. It is prefer-able touse industrial standard equipment for the event builder if
possible, in order to reduce the cost of development and easemaintenance. Gigabit Ethernet is fully compatible withexisting Ethernet installation, which is not only aninternational standard but also de facto standard. TCP/IP as
candidate software for the data path also has the sameadvantages.
Thus, we investigated the feasibility of the QoS on Gigabit
Ethernet, which is a way to design data flow by allocating
bandwidth.

A. Requirements and Gigabit Ethernet with QoS
Latency of packet routing as low as possible is better for
event builder, but the latency is not critical. Thus, high levelprotocol such as TCP/IP with QoS may be used. On the otherhand, Gigabit Ethernet technology with jumbo frame wasinvestigated [5,13] because the throughput is critical. It
showed that jumbo frame doubled the transfer speed. It is
expected that use of the jumbo frame works well for the event
builder.
Recent Gigabit Ethernet companies guarantee“ wirespeed” at each port while the possibility of the congestion atburst data flow still remains within a switch. It is not clear toguaranteeῌthe speed on a coherent traffic. It is expected thatGigabit Ethernet with QoS shapes the traffic in a way to avoidcongestion.
The next section describes congestion control andbandwidth allocation in Linux. 3 section describes theperformance measurement of Gigabit Ethernet with QoS on
PC/Linux.

II. CONGESTION CONTROL AND BANDWIDTHALLOCATIONIn TCP/IP, delayed ACK is a traffic control method toreduce unnecessary packets, by acknowledging multiple
packets with a single ACK. The Nagle algorithm packs shortmessages to avoid congestion. Traffic control can be done bycontrolling the TCP sliding window size. However, it is notclear that those algorithms are enough to avoid the congestion
on event builder.
Researches and developments of QoS are recently done forInternet application, in environments LAN and WAN. InternetEngineering Task Force (IETF) is working on terminology
and architecture of QoS. It defined Resource ReSerVation
Protocol (RSVP) [7] for multimedia application such as Voiceover IP and Video on demand. The essential point oftraffic management is packet scheduling, known as
queueing method. PC-based scheduling mechanism isstudied in computer science [8,9]. Packet queueing
disciplines are studied in techniques related to ATM incomputer science. They are Class-Based Queueing (CBQ),
Token Bucket Filter (TBF) and so on.

A. Terminology of QoS
We introduce elementary technologies of QoS first.
There are admission control, packet classification, packet
scheduling and traffic shaping. The admission control is a
way to control reserving resources in a session such as
RSVP. A setup protocol makes signaling on the path.
Figure 1 shows Class-Based Queueing discipline. The
packet classification is to classify incoming packets into
groups called class.

queueing discipline for Linux. It defines a queueing
discipline, CBQ as root with bandwidth 1000Mbit/s.
When a packet comes into the queueing discipline, thepacket will be sent to a classifier called u32, whichclassifies the packet. The packet will come into a classcalled CBQ, which also assigns 1000Mbit/s as thebandwidth. After the packet passes through the class, aqueueing discipline called TBF will limit the transfer rate.
The transfer rate will be limited to 100Mbit/s according to
the rate of the token even if packet arrives over the rate.

Figure 4. Queueing Disciplines for LinuxIII. PERFORMANCE EVALUATIONA. Measurement

Figure 1: Class-Based Queueing

The packet scheduler is to arrange the scheduling foroutgoing packets. There are many ways according to thequeueing method and the buffer management. Figure 2
shows Token Bucket Filter as an example of a packetscheduling algorithm. The outgoing packet will be sent
according to the size of the token buffer and the rate. The
traffic shaper is a technique to make the burst data flat.

On the measurement of transfer speed, we evaluated thetransfer speed on three frequencies of CPU, three types ofchipset, and on 64/32-bit PCI. We also measured copy
speed of memory on the PCs.
On the measurement of QoS, we evaluated 1x1, 1x3
and 3x1 systems with QoS. We used TBF in CBQ for
allocating transfer rate.
Before the measurements, we tuned network driver andTCP buffer size. We evaluated TCP buffer size. The valueof 528288 was best. We had evaluated Nagle algorithm[13] and then adopted Nagle algorithm for themeasurement.
We used the netperf utility [14] as the measurement
tool to evaluate performance at point-to-point link.

B. Setup

Figure 2. Token Bucket Filter

B. Traffic Management in LinuxFor Linux, kernel 2.2.x and later support many kindsof QoS [10]. The developer toolkit also supplies somecommands for manipulating the parameters in the kernel[11]. The typical command is called tc. Figure 3 shows atraffic control in TCP/IP for Linux. The traffic control isdone only at the output queueing.

The configuration of the PCs is shown in Table 1. We
used three types of PCs. A pair of PC733 and a pair ofPC800 were used for the evaluation of transfer speed. Four
PCs of PC500 were used with a Gigabit Ethernet switch
for the evaluation of QoS and transfer speed. The switch
we used was Alteon AceSW180 [6], which has 9 ports ofGigabit Ethernet with 8Gbit/s backplane. Linux driver for
AceNIC was used [12].
Table 1. Configuration of PC500, PC733 and PC800
SystemPC500

PC733

PC800

CPU type

PentiumIII

PentiumIII

PentiumIII

CPU Freq.

500MHz733MHz800MHz

Cache

512KB

256KB

256KB

Chipset440GXSuperWorks
440BX
ServerSet III LE

Bus speed

100MHz133MHz

100MHzFigure 3. Traffic control in TCP/IP

Memory type

SDRAM

SDRAMSDRAM

The Linux kernel supports CBQ queueing discipline
and TBF queueing discipline. Figure 4 is an example ofMemory size

256MB

128MB

256MB

PCI(Bus width) 32-bit

64/32-bit

32-bitTransfer speed in Mbit/s

OS version

2.2.14-12

2.2.14-12

2.2.14-5

1000

gcc version

egcs-2.91.66

egcs-2.91.66

egcs-2.91.66

900

NIC

AceNIC(1MB) AceNIC(1MB)AceNIC(1MB)800

Driver version

v0.47

v0.47

700

v0.47

600

tx=400:rx=120
tx=996:rx=1000
tx=2000:rx=2000
tx=40:rx=12500
400

C. Copy speed of memory

300

We measured the copy speed of memory on the PCs
by using a Linux system call, memcpy. The speed
depends on CPU speed and performance of chipset. Highperformance memory copy is crucial to obtain highbandwidth using TCP/IP.
500
450400350

PentiumIII733MHz/SuperWorks
PentiumIII500MHz/440GX
PentiumIII800MHz/440BX

300
250200150
1005001100
00.01

1100

10000

Message size in KB

Figure 6. Transfer speed between PC500s in normal frame

Copy speed in MB/s

0.120010100

Data size in MB

Figure 6 shows the transfer speed on PC500s in normalframe. At the values of 996/1000 and 2000/2000, thetransfer speed was better in comparison with that at defaultvalues. We chose the value of 996/1000, which isrecommended by the developer.
Figure 7 shows the transfer speed in jumbo frame,
which means MTU equals 9000. At values of 200/300,
the transfer speed was better in comparison with that atdefault values.
Transfer speed in Mbit/s
1000
900

Figure 5. Performance of memory copy

800

PC733 was the fastest because it has a highperformance chipset.

700
600

D. Tuning Network Driver

tx=20:rx=30
tx=200:rx=300
tx=400:rx=600
tx=2:rx=3500
400

The parameters in the driver code were tuned to get thebest performance. We have modified and tested theparameters with regard to coalescing. The coalescing
affects the times of interrupts to CPU from the NIC. We
varied the following two parameters tx_coal on thetransmitting side and rx_coal on the receiving side. Whenthe value of the parameter is small, the interrupt timeincreases. We tuned those parameters for normal frame
and jumbo frame in Table 2.
Table 2. tx_coal and rx_coal for normal and jumbo frames
tx_coalrx_coalNormal (best/default)
996/400
1000/120
Jumbo(best/default)
200/20300/30We varied the values of tx_coal and rx_coal parameters.
In normal frame, which means MTU equals 1500, the pairof values for tx_coal/rx_coal, 40/12, 400/120, 996/1000
and 2000/2000 were chosen for the evaluation.

300
200
10000.01

110010000

Message size in KB

Figure 7. Transfer speed between PC500s in Jumbo frame

Thus, we chose 996/1000 in tx_coal/rx_coal fornormal frame and 200/300 in tx_coal/rx_coal for jumboframe. We confirmed that the feature was similar on
PC733s and PC800s.
Figure 8 shows the transfer speed on PC500s, PC733s
and PC800s in case of MTU1500. The speed did notdepend on CPU speed.

Transfer speed in Mbit/s

F. QoS performance on 1x1 system

1000
900
800
700
600
Pentium500MHz
500

Pentium733MHz
Pentium800MHz

400

The QoS was adopted at source node. The rate wasassigned by TBF. With the QoS, the measured rate
showed that the bandwidth allocation worked well.
Furthermore, the CPU overhead of QoS was very smalland nearly 1%.

Measured rate in Mbit/s
600300200No-QoS

500

100

40000.01

1100

10000

Ideal rate

Message size in KB

300

Measured rate

Figure 8. Transfer speed in normal frame on various PCs

However, the transfer speed with jumbo frame changed
the feature. PC800 with highest performance CPU andworst chipset had poor performance while PC733 withhighest performance chipset was best. Figure 9 shows theresult.

200

10000Transfer speed in Mbit/s

100

200

300

400

500

600

Assigned rate in Mbit/s

1000

900

Figure 11. QoS performance on 1x1 system

800

G. QoS performance on 1x3 system

700
600
Pentium500MHz
Pentium733MHz500

Pentium800MHz
400
300
200
10000.01

1100

The rate was also assigned by TBF. Namely, TBFlimits the transfer to each destination node at source node.
With the QoS, the measured rate showed that thebandwidth allocation worked at small rate, but did notwork well at full rate. Furthermore, the transfer speed
without QoS at each destination node exceeds 200Mbit/s
and the speed to each destination was nearly identified.

Measured rate in Mbit/s

10000

Message size in KB

250

Figure 9. Transfer speed in jumbo frame on various PCs

E. 64-bit PCI vs. 32-bit PCI
We evaluated transfer speed on 64-bit PCI. Figure 10shows the transfer speed on PC733s in cases of 64-bit PCIand 32-bit PCI. At MTU1500, the transfer speed with 32bit PCI was similar to that with 64-bit PCI. However, thetransfer speed with 64-bit PCI was better to that with 32bit PCI at MTU9000. The speed reached to 990Mbit/s.

200

No-QoS

150

Ideal rate
Measured rate on node1
Measured rate on node2
Measured rate on node3

10050

Transfer speed in Mbit/s
100009000800

50100

150200250

Assigned rate in Mbit/s

70060032-bit PCI(mtu1500)
32-bit PCI(mtu9000)
64-bit PCI(mtu9000)

400
300
200
10000.01

Figure 12. QoS performance on 1x3 system

64-bit PCI(mtu1500)

500

110010000

Message size in KB

Figure 10. 64-bit PCI vs. 32-bit PCI for transfer speed

H. QoS performance on 3x1 system
The rate was also assigned by TBF. Namely, TBF
limits the transfer to each destination node at source nodes.
With the QoS, the measured rate showed that thebandwidth allocation worked well.

Measured rate in Mbit/s

250

200

No-QoS

150

Ideal rate
Measured rate on node1Measured rate on node2
Measured rate on node3

100500050100150200250

Assigned rate in Mbit/s
Figure 13. QoS performance on 3x1 system

IV. CONCLUSION
The traffic management of event data flow is necessary
for the event builder. We investigated another solution of
the traffic management using IP-based QoS on PC/Linux
instead of ATM-based traffic management.
On the evaluation of basic performance with Gigabit
Ethernet on PC/Linux, we tuned the network driver andthen measured the transfer speed with various types of
CPU and chipset, and 64-bit PCI. The result showed thatthe transfer speed did not depend on CPU speed in normal
frame. It also showed that high performance chipset, 64-bit
PCI and jumbo frame were effective and necessary to getbest performance.
On the evaluation of the QoS, transfer rate assignment
was possible by using tc command, but it did not makeclear that the QoS on Gigabit Ethernet was effective for
event builder.

V. ACKNOWLEDGMENTS
The authors wish to thank Prof. Takahiko Kondo at
KEK for his support and encouragement, Dr. Hiroyuki
Sato at KEK for his help, and people of Netone Systems
Corporation and Alteon Web Systems in Japan for their
help.

V. REFERENCES[1] Ed. Basotti et al., A Proposed Scalable Parallel Open
Architecture Data Acquisition System for Low toHigh Rate Experiments, Test Beam and All SSC
Detectors, IEEE Trans. NS, Vol.37, No.3 (1990)[2] Y.Nagasaka et al., Performance Analysis of a Switchtype Event Builder with Global Traffic Control
System, IEEE Trans. NS, Vol.43, No.1(1996)[3] D.Calvet et al., Evaluation of a Congestion
Avoidance Scheme and Implementation on ATM
Network based Event Builders, Proc. Second
International Data Acquisition Workshop on
Networked Data Acquisition Systems, Osaka, Japan,13-15, November 1996, World Scientific Publishing
1997, pp.96-107.

[4] D.Calvet et al., Operation and Performance of an
ATM based Demonstrator for the Sequential Optionof the ATLAS Trigger, IEEE Trans. NS, Vol.45,No.4(1998)[5] Y.Yasu at al., Evaluation of Gigabit Ethernet with
Java/HORB, Contributed to the International
Conference on Computing in High Energy Physics,
CHEP98, Chicago, August 31 - September 4, 1998
[6] Home
pageofAlteon
WebSystems,
http://www.alteonwebsystems.com/[7] R.Braden, L.Zhang, S.Berson, S.Herzog andS.Jamin, Resource ReSerVation Protocol (RSVP) –Version1 Functional Specification, RFC2205.
[8] S.Floyd and V.Jacobson, Link-sharing and Resource
Management Models
for PacketNetworks,
IEEE/ACM Transactions on Networking, Vol.3
No.4, August 1995
[9] K.Cho, A Framework for Alternate Queueing:
Towards Traffic Management by PC-UNIX BasedRouters, Proceedings of USENIX 1998 Annual
Technical Conference, New Orleans LA, June 1998
[10] Alexey Kuznetsov, Implementation of queuing
disciplines in Linux kernel. They are available inLinux kernel 2.2.x. http://www.linuxhq.com/[11] Alexey Kuznetsov, tc command is available.
ftp://ftp.inr.ac.ru/ip-routing/[12] Jes Sorensen, Linux device driver for AceNIC,
http://home.cern.ch/~jes/gige/acenic.html[13] Yoji Hasegawa, Yasushi Nagasaka, Yoshiji Yasu,
DAQ/EF-1 Event Builder system on Linux/Gigabit
Ethernet, ATL-DAQ2000-008, March 2000
[14] Netperf Home page,
http://www.netperf.org/netperf/NetperfPage.html