Quality of Service on Linux forJPS
Okinawa the Atlas TDAQ Event Building NetworkATLAS
TDAQY.Yasu(1), A.Manabe(1), Y.Nagasaka(2), H.Hasegawa(3),M.Shimojima (4), M.Nomachi(5), H.Fujii(1) and Y.Watase(1)on behalf of the Atlas Trigger/DAQ group(1)High Energy Accelerator Research Organization (KEK)
Oho 1-1, Tsukuba, Ibaraki 305 Japan(2) Department of Information and Intellectual Systems Engineering Facultyof Engineering Hiroshima Institute of Technology
2-1-1 Miyake, Saeki-ku, Hiroshima 731-5193 Japan(3)Faculty of Science, Shinshu University
Asahi 3-1-1, Matsumoto, 390-8621 Nagano Japan(4)Department of Electrical Engineering
Nagasaki Institute of Applied Science (NIAS)
536 Aba-machi, Nagasaki 851-01 Japan(5)Graduate School of Science, Osaka University
Machikaneyama 1-1, Toyanaka-city, 560-0043 Osaka Japan

September 24, 2001

Y.Yasu1

ContentsJPS
Okinawa

ATLAS
TDAQ

Why QoS is necessary
What QoS is
Performance Measurements
Packet loss on UDP/IP multicast transfer
QoS software overhead on TCP/IP transfer
Packet distribution on TCP/IP transfer
Conclusion

September 24, 2001

Y.Yasu2

JPS
Okinawa

ATLAS Event Builder System

September 24, 2001

Y.Yasu

ATLAS
TDAQ

3

Why QoS is necessary

JPS
Okinawa

ATLAS
TDAQ

> UDP/IP multicast transfer is attractive for broadcasting messages
to Event Builder Sources from scalability point of view,
but UDP/IP may lose the messages(packets).

Event Builder
Sources

Data Flow
Manager> Constant Bit Rate of ATM network can eliminate the packet loss
by managing the bandwidth.
> QoS in Linux kernel can manage the bandwidth.
Can the QoS eliminate the packet loss?
September 24, 2001

Y.Yasu4

JPS
Okinawa

What QoS is:Class-Based Queueing (CBQ)Class 1

packetclassifierATLAS
TDAQ

Set over-limit

estimator

Class 2Packet scheduler

Class 3

CBQ can classify incoming packets into multiple classes.
CBQ can share and limit the transfer rate.

September 24, 2001

Y.Yasu5

JPS
Okinawa

What QoS is :Token Bucket Filter (TBF)

ATLAS
TDAQ

rate
Tokenbuffer
Data packet

Bucket size

Data Buffer

TBF can limit the transfer rate according to the token rate

September 24, 2001

Y.Yasu6

JPS
Okinawa

What QoS is :Traffic control in Linux kernel

Upper layers(TCP,UDP,...)

Input de-multiplexing

Forwarding

ATLAS
TDAQ

Traffic ControlOutput queueing

Traffic control is done only at output queueing.
Outgoing packet is only controlled.

September 24, 2001

Y.Yasu7

JPS
Okinawa

What QoS is :Queueing discipline for LinuxClassifier
u32:
Specified IP
Address

ATLAS
TDAQ

Qdisc (TBF)
bandwidth:100Mbit/s

Class(CBQ) 1:1 bandwidth:1000Mbit/s

Root Qdisc(CBQ) 1: bandwidth :1000Mbit/s

U32 can specify IP address to manage the bandwidth.
TBF limits the bandwidth to 100Mbit/s on 1000Mbit/s networkat this example.
September 24, 2001

Y.Yasu8

JPS
Okinawa

Performance measurement

ATLASTDAQ

Packet loss measurement
Measurement of QoS software overhead
Packet distribution measurement

September 24, 2001

Y.Yasu9Testbed

JPS
Okinawa

ATLAS
TDAQ

3Com Super StackII Switch 9300
12 ports

PC500

PC500

PC500

PC500

Configuration of PC500 system
CPU:
ChipsetMemory:PCIbus:
NIC:

September 24, 2001

PentiumIII/500MHz
440GX
100MHz/SDRAM/256MB
32-bit/33MHz
AceNIC(1MB)Linux kernel version : 2.4.5
gcc version:
egcs-2.91-66
acenic driver version : 0.8

Y.Yasu10JPS
Okinawa

Packet loss on
UDP/IP multicast transfer

ATLAS
TDAQ

3Com Super StackII Switch 9300
12 ports

PC500

PC500

PC500

PC500

SenderReceiver Receiver Receiver

Sender sends message with 10 bytes to receivers in best effort.

September 24, 2001

Y.Yasu11JPS
Okinawa

Packet loss on
UDP/IP multicast transfer (Cont.)
Transfer speed in Mbit/s

No QoSATLAS
TDAQPacket loss in percentage

1200

10090

1000

80
70

800

60
50
40600
400

30
20200
00

September 24, 2001

100300
600
900
1200
1500
Message size in bytes packet loss in any size
Y.Yasu12JPS
Okinawa

Packet loss on
UDP/IP multicast transfer(Cont.)

Transfer speed in Mbit/sAssigned bandwidth = 10 Mbit/s

ATLAS
TDAQ

Packet loss in percentage

12100

1080

860
6404220001500

0300

600900

Message size in bytes
September 24, 2001

Y.Yasu1200

No packet loss in any size
13JPS
Okinawa

Packet loss on
UDP/IP multicast transfer(Cont.)

Transfer speed in Mbit/s

Assigned bandwidth = 50 Mbit/s

ATLAS
TDAQ

Packet loss in percentage

50100

4080

3060

2040

102000100

0204060

Message size in bytes
September 24, 2001

Y.Yasu80

packet loss in 60-byte size
14JPS
Okinawa

Packet loss on
UDP/IP multicast transfer(Cont.)

Transfer speed in Mbit/s

Assigned bandwidth = 100 Mbit/s

ATLAS
TDAQ

Packet loss in percentage

120

100

100808060604040202000
September 24, 2001

0300
600
900
1200
1500
Message size in bytes No packet loss out of 200-byte size
Y.Yasu15QoS Software Overhead

JPS
Okinawa

ATLAS
TDAQ

3Com Super StackII Switch 9300
12 ports

PC500

PC500

SenderReceiver

PC500

PC500

Assigned bandwidth is 1000Mbit/s.
Sender sends message with 10 bytes to a receiver in best effort.
This bandwidth does not limit the transfer, but it consumes CPU time.
September 24, 2001

Y.Yasu16JPS
Okinawa

QoS Software Overhead: TCP/IP transfer speed

ATLAS
TDAQ

Transfer speed in Mbit/sec.

700
600
500
400

No QoS
QoS

300
200
100010100

1000

Message size in bytes

September 24, 2001

Y.Yasu

10000

Peak speed was same.
17JPS
Okinawa

QoS Software Overhead: the CPU Usage

ATLAS
TDAQ

CPU Usage (%)

110
10090

sender cpu
usage(No QoS)
sender cpu
usage(QoS)

80
7060
10100

1000

10000

Message size in bytes

September 24, 2001

Y.YasuIn small size of message,QoS was 10 % worse.
In large size of message,
the percentage increases.
18

TCP/IP Packet Distribution

JPS
Okinawa

ATLAS
TDAQ

3Com Super StackII Switch 9300
12 ports

PC500

PC500

SenderReceiver

PC500

PC500

Assigned bandwidth is 10Mbit/s.
Sender sends message with 10 bytes to a receiver.
tcpdump utility is used for capturing packets

September 24, 2001

Y.Yasu19

JPS
Okinawa

TCP/IP Packet Distribution(Cont.)
QoS

ATLAS
TDAQ

No QoS

Horizontal axis : sending packet time intervalWith QoS, some packets were sent at 10 msec scheduling time.
Without QoS, packets are mostly sent in several 10 usec.
September 24, 2001

Y.Yasu20

JPS
Okinawa

TCP/IP Packet Distribution(Cont.)
QoS with 1kHz trigger rate

ATLAS
TDAQ

No QoS with 1kHz trigger rate

With 1kHz trigger, packets are mostly sent around 1msec.
There are no difference between QoS and NoQoS.
September 24, 2001

Y.Yasu21

JPS
Okinawa

Conclusion

ATLAS
TDAQ

We measured with/without QoS in Linux kernel:
1) Packet loss on UDP/IP multicast transfer
2) QoS software overhead on TCP/IP transfer
3) Packet distribution on TCP/IP transferWe concluded:
1) QoS could eliminate packet loss on UDP/IP multicast transfer.
This shows the feasibility to solve the scalability issueof ATLAS TDAQ Event Builder.
2) QoS was efficiently performed on TCP/IP transfer.
CPU usage of QoS on the transfer was small.
3) With QoS, some packets were sent at 10 msec scheduling time.

September 24, 2001Y.Yasu22