Quality of Service(品質保証機能)を用いたネットワーク技術のアトラスイベントビルダへの適用

JPS @立教大学

ATLAS
TDAQ

Y.Yasu(1), Y.Nagasaka(2), H.Hasegawa(3), M.Shimojima (4),A.Manabe(1), M.Nomachi(5), H.Fujii(1) and Y.Watase(1)on behalf of the Atlas Trigger/DAQ group(1)High Energy Accelerator Research Organization (KEK)
Oho 1-1, Tsukuba, Ibaraki 305 Japan(2) Department of Information and Intellectual Systems Engineering Facultyof Engineering Hiroshima Institute of Technology
2-1-1 Miyake, Saeki-ku, Hiroshima 731-5193 Japan(3)Faculty of Science, Shinshu University
Asahi 3-1-1, Matsumoto, 390-8621 Nagano Japan(4)Department of Electrical Engineering
Nagasaki Institute of Applied Science (NIAS)
536 Aba-machi, Nagasaki 851-01 Japan(5)Graduate School of Science, Osaka University
Machikaneyama 1-1, Toyanaka-city, 560-0043 Osaka Japan

September 13, 2002

Y.YasuContents

JPS @立教大学

アトラスイベントビルダとQoS
Basic performance measurements
Conclusion and Plan

September 13, 2002

Y.YasuATLAS
TDAQWhat is QoS (Quality of Service)

JPS @立教大学

ATLAS
TDAQ

Functionality of QoS
Admission control
decides whether input QoS request is acceptable or not.

Provisioning
sets up network configuration.

Classification
classifies input packets according to the QoS.

Queue management and Schedulingprocesses packets at output port.

September 13, 2002

Y.YasuWhat is QoS:Class-Based Queueing (CBQ)JPS @立教大学

Class 1

packetclassifier

Set over-limit

estimator

Class 2

Packet scheduler

Class 3

CBQ can classify incoming packets into multiple classes
CBQ can share and limit the transfer rate

September 13, 2002

Y.YasuATLAS
TDAQWhat is QoS :Token Bucket Filter (TBF)JPS @立教大学

rate
Tokenbuffer
Data packet

Bucket size

Data Buffer

Token buffer is fulfilled with tokens at start time.

September 13, 2002

Y.YasuATLAS
TDAQCongestion control with QoS

JPS @立教大学

Constant Bit Rate of ATM network could control the congestion on
event builder by managing the bandwidth.
However, Ethernet does not have the functionality.

Linux QoS can manage the bandwidth.
Can Linux QoS with Ethernet control the congestion?

September 13, 2002

Y.YasuATLAS
TDAQConfiguration of Testbed

JPS @立教大学

ATLAS
TDAQ

Gigabit Ethernet Switch

UDP/Multicast
SenderPseudo Trigger
Message size : 1kB

ReceiverMeasurement by tcpdump utility
PC500

PC500

PC500

PC500

SenderReceiver Receiver Receiver

Configuration of PC500 system
CPU:
ChipsetMemory:PCIbus:
NIC:

September 13, 2002

PentiumIII/500MHz
440GX
100MHz/SDRAM/256MB
32-bit/33MHz
AceNIC(1MB)Linux kernel version : 2.4.5
gcc version:
egcs-2.91-66
acenic driver version : 0.8

Y.YasuMessage rate (100Hz) without QoS

JPS @立教大学

ATLAS
TDAQ

Time interval of message in msec.

Periodic Trigger : 100Hz, Message size : 1kB, without QoS

2220
18
16
100Hz-noQoS
14

First message
Last message

12
Data rate :800kbps(100kB)
Without QoS

10Time ---->

80200

400

600

Message number
September 13, 2002

Y.Yasu8001000

Message rate (100Hz) with QoS

JPS @立教大学

Time interval of message in msec.

22ATLAS
TDAQ

Periodic Trigger : 100Hz, Message size : 1kB, Assigned rate : 500kbps

20QoS begins scheduling (bandwidth:500kbps)

1816100Hz-500kbps
1412Messages were scheduled in10msec and 20msec.

Token is still enough here.

Data rate :800kbps(100kB)
Without QoS

1080200

400

600

Message number
September 13, 2002

Y.Yasu8001000

Message rate (1kHz) with QoS

JPS @立教大学

Periodic Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps

12Time interval of message in msec.

ATLASTDAQ

108Average transfer rate was 5Mbps.

61kHz-5Mbps

4Messages were sent
immediately.
This is due to Linux schedulingtime. THIS IS PROBLEM!

200

200

400

600

Message number
September 13, 2002

Y.Yasu800

1000

JPS @立教大学

Important Linux Scheduling time

Linux scheduling time is usually 10msec.
It is defined by a parameter HZ in the kernel.
Namely, HZ is 100.
毎秒１００回のタイマー割り込みによるプロセススケジューリング
This value is not convenient for the traffic shaping
because the average trigger rate of Event Builder
input is over 1kHz.
Let’s make the value 1500(667usec).
September 13, 2002

Y.YasuATLAS
TDAQMessage rate (1kHz) with QoS

JPS @立教大学

ATLAS
TDAQ

Time interval of message in msec.

Periodic Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps

1210
86

1kHz-5Mbps

Messages were sent around 1.6msec.

42Data rate :5Mbps(Assigned rate :5Mbps)

00200

400

600

Message number
September 13, 2002

Y.Yasu8001000

QoS measurement with random trigger

JPS @立教大学

Periodic trigger Æ OK
However, actual trigger is random.
Can QoS make random traffic regular?
Namely, does it simulate Constant Bit Rateas ATM?
Let’s make random trigger.

September 13, 2002

Y.YasuATLAS
TDAQMessage rate (1kHz) with QoS

JPS @立教大学

ATLAS
TDAQ

Time interval of message in msec.

Gaussian Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps

1210

QoS makes random traffic regular!
Message rate is constant.

864

Data rate :5Mbps(Assigned rate :5Mbps)

200

September 13, 2002

500

1000
Message number
Y.Yasu15002000JPS @立教大学: Rikkyou

Linux kernel overhead

ATLAS
TDAQ

• 使ったベンチマークプログラムについてmemory copy性能を指標とした。
• ノーマルなスケジューリングの場合と毎回１５００回のそれを比較しての場合が約1％程度の性能劣化があった。
つまり、タイマー割り込みによるオーバーヘッドは約１％程度だった。
Size of copy

HZ=100

HZ=1500

Overhead(%)

4MB

179.32MB/s

177.8MB/s

0.8476478MB

179.34MB/s

177.08MB/s

1.26017616MB

179.34MB/s

177.68MB/s

0.925616

32MB

179.3MB/s

177.67MB/s

0.909091

September 13, 2002

Y.YasuJPS @立教大学

Conclusion and Plan

ATLAS
TDAQ

Conclusion• Linux QoS with Ethernet can make 1kHz random
traffic regular. This feature makes the feasibility ofcongestion control on Atlas Event Builder data path.

Plan• We are now applying QoS to real Atlas event builder
software framework on test-beds at Japanese sites
and CERN for the Technical Design Report in nextyear.
September 13, 2002

Y.Yasu