DAQ2000, Lyon

ATLAS-Japan DAQ Group

Quality of Service on Gigabit Ethernet
for Event Builder
Y.Yasu(1), Y.Nagasaka(2), H.Hasegawa(3), A.Manabe(1),M.Nomachi(4), H.Fujii(1), Y.Watase(1)(1)High Energy Accelerator Research Organization (KEK)
Oho 1-1, Tsukuba, Ibaraki 305 Japan(2)Department of Electrical Engineering
Nagasaki Institute of Applied Science (NIAS)
536 Aba-machi, Nagasaki 851-01 Nagasaki Japan(3)Faculty of Science
Shinshu University
Asahi 3-1-1, Matsumoto, 390-8621 Nagano Japan(4)Graduate School of Science
Osaka University
Machikaneyama 1-1, Toyanaka-city, 560-0043 Osaka Japan
October 20, 2000

Y.Yasu1

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Contents
Gigabit Ethernet and Quality of Service(QoS)
for Event Builder
Basic performance of Gigabit Ethernet
QoS in LinuxQoS performance on Gigabit Ethernet

October 20, 2000

Y.Yasu2

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Gigabit Ethernet and Quality of Service
for Event Builder
Congestion Avoidance
by Global Traffic Control(Fermilab, KEK)
Special Hardware with Barrel shifter etc.
by traffic shaping for each node(RD31/ CERN)
ATM including QoS(Constant Bit Rate)

How can we use Gigabit Ethernet with TCP/IP ?

October 20, 2000

Y.Yasu3

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Basic performance of Gigabit Ethernet
Hardware configuration and setup
Performance measurement of memory copy
Tuning network driver
Performance measurement of Gigabit Ethernet

October 20, 2000

Y.Yasu4

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Hardware configuration
SystemPC500

PC733

PC800

CPU type

PentiumIII

PentiumIII

PentiumIII

CPU Freq.

500MHz733MHz800MHz

Cache

512KB

256KB

256KB

Chipset440GXSuperWorks
ServerSet III LE

440BX

Bus speed

100MHz133MHz

100MHzMemory type

SDRAM

SDRAM

SDRAM

Memory size

256128256PCI(bus width)

32-bit64-bit/32-bit

32-bit

OS version

2.2.14-12

2.2.14-12

2.2.14-5

gcc version

egcs-2.91.66

egcs-2.91.66

egcs-2.91.66

NIC

AceNIC(1MB)AceNIC(1MB)AceNIC(1MB)

Network Driverversion

v0.47

v0.47

v0.47

October 20, 2000

Y.Yasu5DAQ2000, Lyon

ATLAS-Japan DAQ Group

Performance measurement of memory copy
Memory-copy speed in MB/s

500
450
400
350
PentiumIII733MHz/SuperWorks
PentiumIII-500MHz/440GX

300
250

PentiumIII-800MHz/440BX

200
150
10050

by memcpy system call

00.1

110100

Memory size in MBOctober 20, 2000

Y.Yasu6

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Hardware setup
Alteon Switch 180

PC500

PC500

PC500

PC500

For measurement of QoS performanceand measurement of basic performance

October 20, 2000

Y.Yasu

PC733

PC800

PC733

PC800

For measurement ofbasic performance

7

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Tuning network driver
Normal frame

Jumbo frame

Transfer speed in Mbit/s

1000
900
800
700
600

tx=400:rx=120
tx=996:rx=1000
tx=2000:rx=2000
tx=40:rx=12500
400
300
200
100
00.01

1100

Transfer speed in Mbit/s

1000

700
600

tx=20:rx=30tx=200:rx=300
tx=400:rx=600
tx=2:rx=3500
400
300
200
100

1100

10000

Message size in KB

Message size in KB

October 20, 2000

800

00.01

10000

Tuned parameters of Network driver :/proc/sys/net/core/rmem_max 4000000/proc/sys/net/core/wmem_max 4000000
TCP buffer size : 524288 bytes
Nagle algorithm is enable.

900

Normal frame : tx_coal=996 : rx_coal=1000
Jumbo frame : tx_coal=200 : rx_coal=300
Other parameters : default
Used systems : PC500s
Measuring tool : NetperfY.Yasu

8

DAQ2000, Lyon

ATLAS-Japan DAQ GroupPerformance measurement of Gigabit Ethernet

Transfer speed in normal frame
Transfer speed in jumbo frame
Bus width dependency(64-bit PCI vs. 32-bit PCI)

October 20, 2000

Y.Yasu9DAQ2000, Lyon

ATLAS-Japan DAQ Group

Transfer speed in normal frame
1000

Transfer speed in Mbit/s

900
800
700
600
Pentium500MHz
500

Pentium733MHz
Pentium800MHz

400
300
200
100
00.01

0.1

1

10100

1000

Message size in KB

Transfer speed does not depend on CPU speed in normal frame
October 20, 2000

Y.Yasu10

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Transfer speed in jumbo frame
Transfer speed in Mbit/s

1000
900
800
700
600

Pentium500MHz
Pentium733MHz

500

Pentium800MHz
400
300
200
10000.01

110010000

Message size in KB

Transfer speed depends on Chipset performance in jumbo frame.
October 20, 2000

Y.Yasu11DAQ2000, Lyon

ATLAS-Japan DAQ GroupBus width dependency(64-bit PCI vs. 32-bit PCI)
Transfer speed in Mbit/s

1000
900
800
700
600

32-bit PCI(mtu1500)

500

64-bit PCI(mtu1500)
32-bit PCI(mtu9000)

400

64-bit PCI(mtu9000)

300
200
100
00.01

On PC733s
110010000

Message size in KB

Bus width does not affect transfer speed in normal frame.
64-bit PCI improves transfer speed in jumbo frame, up to 990Mbit/s.
October 20, 2000

Y.Yasu12

DAQ2000, Lyon

ATLAS-Japan DAQ GroupQoS in LinuxClass-Based Queueing
Token Bucket Filter
Traffic control in TCP/IP
Queueing discipline for LinuxQoS performance on Gigabit Ethernet
1x1 system
3x1 system
1x3 system
October 20, 2000

Y.Yasu13

DAQ2000, Lyon

ATLAS-Japan DAQ GroupClass-Based Queueing (CBQ)Class 1

packetclassifier

Set overlimit

estimator

Class 2

Packet scheduler

Class 3

CBQ can classify incoming packets into multiple classes.
CBQ can share and limit the transfer rate.
October 20, 2000

Y.Yasu14

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Token Bucket Filter (TBF)
rate
Tokenbuffer
Data packet

Bucket size

Data Buffer

TBF can limit the transfer rate according to the token rate.

October 20, 2000

Y.Yasu15

DAQ2000, Lyon

ATLAS-Japan DAQ GroupQoS in LinuxQoS in Linux kernel 2.2.x and Iproute2 for Linux
supplies the following functions to control IP packets;
For IP packets of inbound and outboundClassifying
Prioritizing
SharingLimiting
QoS in Linux kernel 2.2.x : in kernel standard distribution
Iproute2 : ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz
Iproute2 supports a typical command called tc.
Can we use the QoS such as Class-Based Queueing andToken Bucket Filter for shaping traffic on Event Builder?
October 20, 2000

Y.Yasu16

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Traffic control in TCP/IP
Upper layers(TCP,UDP,...)

Input de-multiplexing

Forwarding

Traffic Control
Output queueing

Traffic control is done only at output queueing.

October 20, 2000

Y.Yasu17

DAQ2000, Lyon

ATLAS-Japan DAQ Group

Queueing discipline for LinuxClassifier
u32:
Specified IP
Address

Qdisc (TBF)
Rate:100Mbit
ClassCBQ 1:1

QdiscCBQ 1:

rate:1000Mbit
bandwidth:1000Mbit

U32 is a classifier.
TBF limits the transfer rate to 100Mbit/s at this example.
The configuration was used at 1x1 system for the evaluation.

October 20, 2000

Y.Yasu18

DAQ2000, Lyon

ATLAS-Japan DAQ GroupQoS performance on 1x1 system
Measured rate in Mbit/s

600

No QoS

500

400
Ideal rate

300

Measured rate

200

100

On PC500s
00100200

300

400500600

Assigned rate in Mbit/s

Bandwidth assignment worked well.
The overhead of QoS was very small and nearly 1 %.
October 20, 2000

Y.Yasu19

DAQ2000, Lyon

ATLAS-Japan DAQ GroupQoS performance on 3x1 system

Measured rate in Mbit/s

250200

No QoS

150

100

Ideal rate
Measured rate on node1
Measured rate on node2
Measured rate on node3

50

On PC500s
0050

100

150

200

250

Assigned rate in Mbit/s

Bandwidth assignment worked well
October 20, 2000

Y.Yasu20

DAQ2000, Lyon

ATLAS-Japan DAQ GroupQoS performance on 1x3 system
Measured rate in Mbit/s

250200

No QoS

150

Ideal rate
Measured rate on node1
Measured rate on node2
Measured rate on node3

10050On PC500s
0050100150200250

Assigned rate in Mbit/s

Bandwidth assignment worked well at small rate, but did not work at full rate.
Packet sharing performance without QoS was better than that with QoS.
October 20, 2000

Y.Yasu21

DAQ2000, Lyon

ATLAS-Japan DAQ GroupConclusion* Traffic management of event data flow is necessary for event builder.
* IP-based QoS with Gigabit Ethernet on PC/Linux was investigated.
* The transfer speed did not depend on CPU speed in normal frame.
* High performance chipset and 64-bit PCI improved
the transfer speed in jumbo frame, up to 990Mbit/s.
* Assignment of transfer rate was possible by using Linux/TC command,
but it is not clear that the QoS on Gigabit Ethernet is effective for
event builder. Thus, more investigation is necessary.

October 20, 2000

Y.Yasu22