Reassignment of network addresses
at SPring-8 control system
T. Sugimoto, M. Ishii, T. Ohata, T. Sakamoto, and R. Tanaka
Japan Synchrotron Radiation Research Institute / SPring-8

The 7th International Workshop on Accelerator Operations
KAERI, Daejeon, KOREA, April 12-16, 2010

Outline Introduction
 Motivation

 Procedure Results and Summary

INTRODUCTION

Location: Korea and Japan

KOREA
JAPAN

We jointly hosted FIFA World Cup 2002.

650km far from the KAERIKAERISPring-8

650km

Bird’s-eye view of the SPring-8 site

SPring-8 is the largest synchrotron
radiation facility in the world.
Total site area is about 1.4x106 m2(c) RIKEN/JASRI

Bird’s-eye view of the SPring-8 site
SCSS EUV-SASE Accelerator
Injection Accelerators
1 beamline(Linac and BoosterSynchrotron)“NewSUBARU”
1-1.5 GeV Storage Ring
9 beamlines~ 500mXFEL Accelerator Facility(under construction)

52 beamlines
8 GeV Storage Ring

SPring-8 accelerator complex:(c) RIKEN/JASRI
SR / Linac / Booster Synchrotron / NewSUBARU / SCSS / XFEL.

Why we need to reassgin network addresses

MOTIVATION

First days of SPring-8 control system
Original design of the network architecture based on ,TCP/IP technology established as standard
protocol/architecture.
 Layer 2 topology without any router Throughput and availability of routers were low at that time.
 Only 350 network devices were installed, then L2 was very

simple and easy to use.



We took advantage of Ethernet as a distributed control
field bus.
 Fast packet forwarding with Ethernet switches Scalability using cascaded connection

SPring-8 control system, 10 years later Control system became complicated.

(~2008) The number of network devices increased.
 More than 1200 devices were connected insingle broadcast domain. (network segment)

A lot of problems arose due to the networkarchitecture.
Two examples are shown in next slides:

Problems on SPring-8 control system (1) IP address exhaustion
 We used the subnet mask of 21 bit.
 Address capacity is 2000.

 ~100 devices were newly installed every year: Virtuallization (Virtual machines) Increase of network-connected devices

 After few years, no more devices would beinstalled.

Max. 2000

 It is difficult to integrate another accelerator (XFEL)

control system.

We must expand address capacity.

Problems on SPring-8 control system (2) Broadcast domain was too large.
 Broadcast traffic was typ. 30 packet/sec.
 In the burst case, broadcast traffic was raised

to > 100 packet/sec.
 Broadcast packets are found to be harmful

for certain network-connected embeddeddevices.
 less processing capability, buffer overflow
 Motor control unit[1] Digital multimeterWe must shrink broadcast domain.
[1] T. Sugimoto et. al: Proceedings of PCaPAC 2008, THX03 (2008)

1. We must expand address capacity.
Most simple solution which keeps L2 topology
is enlargement of broadcast domain, but ...

2. We must shrink broadcast domain.
 Two requirements are conflicted.

Problems are caused by L2 topology!
Make a plan to change the networkarchitecture of SPring-8 control system.

Plan to change architecture
Previous Architecture

New Architecture

Logical Topology

Layer 2(No router)

Layer 3(Using L3 switch)

Address Capacity~2000~65000(for each accelerator complex)

Broadcast domain(Subnet mask)~ 2000.
(21 bit mask)~ 500(23 bit mask)

IP Address(Class-B private)172.24.8-15.z

172.x.y.zx=20 for SP8
x=16 for XFEL, and so on.

VLAN ID

1100x + yName Resolution

NIS

DNS

Domain(none)

4 (sp8.cntl.local, xfel.cntl.local, etc.)

Details: T. Sugimoto et. al: Proceedings of ICALEPCS2009, WED006 (2009)

Network segment: Router v.s. L3 switch
Segmented network needs IP forwarding (L3) network instruments.
RouterL3 Switch

Historically used
Low throughputSoftware based

Multifunction

Available in end of 1990sHigh throughput
Hardware based

Limited function

We don’t need most part of functions!
High reliability
Availavility: intermediate

10 sec failover by VRRPHigh reliability
High availavility< 1 sec failover by stacking

Cisco Catalyst 6504E VSS: to be installed in 2010

Plan to change architecture
Previous Architecture

New Architecture

Logical Topology

Layer 2(No router)

Layer 3(Using L3 switch)

Address Capacity~2000~65000(for each accelerator complex)

Broadcast domain(Subnet mask)~ 2000.
(21 bit mask)~ 500(23 bit mask)

IP Address(Class-B private)172.24.8-15.z

172.x.y.zx=20 for SP8
x=16 for XFEL, and so on.

VLAN ID1

100x + yName ResolutionNIS

DNSDomain(none)

4 (sp8.cntl.local, xfel.cntl.local, etc.)

Details: T. Sugimoto et. al: Proceedings of ICALEPCS2009, WED006 (2009)

Why we choose 23 bit mask.
Max. 2000

After 2007, problem on embedded device appeared.

We allocate “~500” (/23) IP address space to single
broadcast domain.

Plan to change architecture
Previous Architecture

New Architecture

Logical Topology

Layer 2(No router)

Layer 3(Using L3 switch)

Address Capacity~2000~65000(for each accelerator complex)

Broadcast domain(Subnet mask)~ 2000.
(21 bit mask)~ 500(23 bit mask)

IP Address(Class-B private)172.24.8-15.z

172.x.y.zx=20 for SP8
x=16 for XFEL, and so on.

VLAN ID

1100x + yName ResolutionNISDNSDomain(none)

4 (sp8.cntl.local, xfel.cntl.local, etc.)

To accomplish changing architecture,
it is necessary to reassign IP addresses.
Details: T. Sugimoto et. al: Proceedings of ICALEPCS2009, WED006 (2009)

New Network Architecture Archive stable network operation
 Reduce broadcast domain, because wemust guard devices, which are vulnerable toheavy traffic.
Storage Ring
 We changed single segment architecture.
D-zone Network is segmented, divided bynumber C-zone
Booster
Synchrotoronof devices.
A-zone We can append newNewSUBARU
segment easily.

B-zoneLinac
Control Room(c) RIKEN/JASRI

How to change network architecture

PROCEDURE

Procedure Boundary condition
 Address reassignment must be finished by theend of summer shutdown period (~ 1 month).
 New network configuration must work no later
than 2 days after the end of the scheduledperiod.
 Estimated project period
 Test and exercise: 1 month Setup and reconfiguration: 4 days
 Validation: 1 monthShould be done by control division staff .
 We cooperated with > 10 persons.

Test and exercise (July, 2009) Test first, and exercise address

reassignment procedure.
 We built a test bench simulating the actualcontrol system.
 We held rehearsal 4 times:
 Examine shell scripts, which change IP

addresses of > 100 devices at a time.
 Find out and fix problems.
 Estimate required time.

Feedback from exercise
 From the exercise experience, we refined

work procedure as follows:
1.
2.
3.
4.
5.
6.

Name resolution
File server
DatabaseVMEs
Network
Other instruments

Estimated required time was 2 days.

Confirmation and confidence to achieve
successful reassignment – then do it.

AugustJuly

Execution (August 3—6, 2009)

Reassign IP addresses of independently operated
devices, in advance.
31 Accelerator isAll
shutdown
system shutdown (because of power outage)
We check system validity to distinguish trouble
1 Planned power outage
caused by either power outage or network2architecture change.
3 Given period of time
Reassign IP addresses of devices
4(DNS, File server, Database, VME, Network)
30 Accelerator is in operation

567

Restart control system, on schedule.
Start validation of control system (~ 1 month)Validation (August 7 – 31, 2009) Almost all control devices were OK.
 A few problems remain We could not configure default gateway ofVERY OLD devices.
 Multi-channel analyzer purchased in 1990s

 L2-based maintenance terminal did notwork.
To resolve such problems, we configure
port-based VLANs for these devices.

SPring-8 control system was ready to runat the end of August.

RESULTS AND SUMMARY

Results
 Network architecture of SPring-8 control

system is completely changed from L2 toL3 topology.
 We measure address exhaution.
 IP address capacity is upgraded from 2,000 to65,000.

 We also resolve broadcast domain problem.
 Broadcast traffic is reduced from 30 pps to < 1pps, and no burst has been observed.
 Now, motor control unit is operated with notrouble.

Summary Network architecture of SPring-8 is changed to L3.
 Motivation: resolve problems caused by L2 topology
 IP address exhaution
 Too large broadcast domain

 Man power
 It took < 4 days with > 10 persons. (~ 50 man-days)
 We changed > 1000 devices distributed in large SPring-8 site.
 Shell scripts are useful to change many devices at a time.

 Other benefit Asset management
 We assigned responsible person to each device, which was notassgined to.
 We removed ~ 200 devices, which were not used any longer.

 And now, SPring-8 is in operation very stable.

Thank you for your attention!

さくら – Cherry blossoms near the SPring-8, April 2010